Figure 1. A wide range of products were developed in the class. Shown above is the schematic drawing of a latch, cinch, and power lock system for an automobile door.
Download Postscript file for printing (936 K)
Abstract
In this study, the grades assigned to 10 projects at quarterly intervals were plotted along side the number of distinct noun phrases in the project reports. It was found that the grades were strongly associated (gamma > 0.7) with the number of distinct noun phrases while they were weakly associated with other variables like readability and number of words. These initial results open up a new set of ways for assessing design work and as a consequence, improving the performance of students doing design tasks.
Introduction
In the last decade, the number of Engineering Classes that require students to design and build hardware solutions has increased. Typically, this occurs in response to open-ended design problems. One important consequence of this change is that the more traditional tools for measuring student performance such as written examinations and multiple choice questions are no longer as pertinent. For example, the absence of a single right answer at any point in time tends to result in widely varied student responses to assignments. In meeting this challenge, instructors are adopting a variety of newer assessment tools to measure academic performance. These include video interaction analysis [1], peer evaluation [2], methodology assessment [3] and computer activity log analysis [4, 5]. Thus as the instructional situation moves from one in which there is one right answer and method to one in which there are several alternative answers and methods, there is a need for assessment tools that present different viewpoints and are valid in the face of uncertainty.
This paper presents one such viewpoint. It focuses
on noun phrases used by students to report their work. Such noun
phrases, as the commentary on technical language usage below demonstrates,
are not correct in the grammatical sense but as most engineering
instructors and practitioners will realize, they are an important
formative aspect in the discipline. The commentary is presented
here as a precursor to the rest of the paper.
" In the newer technologies - notably in engineering - the {nomenclature} conventions are not systematic or clear; the {engineers} themselves are either unaware of the lack of clarity and system, or do not choose to make the effort to repair it. Therefore anyone who undertakes to read technical documents must make his way through agglomerations like these:.... This situation will stay with us until the {engineers} establish some firm conventions and hold to them as chemists and mathematicians hold to theirs."
- the highest previously available intrinsic coercive force
- single side band transmission
- high frequency stability
- high-energy particle accelerator
- internal transducer excitation supply
the segmented multiple ablative chamber concept - combustion chamber crossover manifold coolant passages
Conner, J.E., A Grammar of Standard English [6].
As correctly pointed out by the grammarian, noun
phrases of these kinds break several grammatical rules. Viewed
differently though, they point to the fact that technical language
is very inventive and more attention needs to be paid to this
aspect of design problem solving. This paper investigates the
changes in noun phrases used by students to report their work
over a period of seven months and explores how these changes may
be used for assessment purposes.
Overview
The work summary reported here is part of a larger
effort to improve the mechanical engineering design process by
using a university based design class as a test-bed. For a more
detailed report the reader is referred to reference [7]. I will
begin by describing the context of this study, then the method
for collecting and analyzing the noun phrases and finally present
and discuss the results.
Materials and Method
The raw data for the study comes from a set of three
reports, named requirements documents, submitted by students in
a project based mechatronics systems design class named ME210.
In this class, students working in three-person teams develop
a list of industry-specified product requirements into a fully
functional hardware prototype within a period of seven months.
The class spans three academic quarters and the product development
process which begins with a re-examination of the client's requirements
is very similar to the product development process guidelines
used in industry [8].
Projects
The projects analyzed were taken from the 1992-93
academic year and dealt with a wide range of products including
a catheter for gene therapy in the human body, a control mechanism
for maintaining the focus of an infrared optical system, and a
power locking device for an automobile door (Figure 1).
Figure 1. A wide range of products were developed in the class. Shown above is the schematic drawing of a latch, cinch, and power lock system for an automobile door.
A complete list of the projects and the class grades
for the winter and spring quarters is shown in Table 1. Observe
that the grades varied between A+ and B.
Requirements Document
While there was a general guideline for writing
the document, there were noticeable variations in the format used
by each project team. For example, one project team described
the design requirements in a series of paragraphs, while another
used a table format, and still another used a combination of both.
Also, some teams used numbered lists and others used bullets.
Some teams listed clauses and others listed complete sentences.
It was important to understand these differences so that all
the peculiarities could be accommodated.
Computer Programs
There were four computer
programs used in the studies; XPOST, a parts-of-speech tagger,
EXTRACTOR, a suite of Microsoft Excel macros written specifically
for this work, TEXAS, an indexing and key-word-in-context search
program, and CORRECT GRAMMAR, a commercial grammar checking software.
Method
The tasks involved in analyzing noun phrases found
in the requirements documents can be grouped under three general
categories: preprocessing, main processing, and post processing.
Figure 2 shows the major tasks and the primary output of each
of the three categories.
Figure 2. The aim of the study is to better understand the evolution of noun phrases during a design project and to compare relevant properties of this evolution with the project performance. The method consists of three phases, each having a primary output which is shown in bold
Preprocessing: The aim of preprocessing is to establish the baseline documents. The process includes removing the graphics, inspecting individual sentences and correcting any obvious mistakes like the omission of a full stop, breaking up paragraphs into individual sentences, translating tables to texts, and saving the document as a text file. These text files constitute the baseline documents.
Main Processing: The aim of this process is to obtain in form of a list, the nouns and noun phrases in the document. This is accomplished by using a parts-of-speech tagger and a suite of custom built macros named extractor for reference purposes.
The parts-of-speech tagger takes as its input, a line such as:
The inner hub holds the steel friction disks
and causes them to rotate when road input is transmitted
through the connecting link to the rotating inner shaft.
and produces a tagged output of two lines. The lower
line consists of parts-of-speech tags corresponding to words in
the upper line. The first letter of each tag occupies the same
position in line as the word to which it refers:
The inner hub holds the steel friction disks
and causes them to rotate when road input is transmitted
through the connecting link to the rotating inner shaft.
at jj nn vbz/2 at nn nn nns
cc vbz/2 ppo/2 to/2 vb wrb nn nn bez vbn
in at vbg nn/2 in/2 at vbg jj nn
For a detailed explanation of the parts-of-speech tagger, the reader should see reference [9]. The input to the tagger was a baseline document and the output was a tagged baseline document. This output was then exported to an excel spreadsheet for further processing using the extractor macros.
In an excel spreadsheet all the lines from the tagger are placed in the first column. Since the position of the first letter of each tag coincides with the position of the tagged word, it was easy to locate words corresponding to a particular tag or sequence of tags. For example, when extracting nouns, the extractor macro copies to the adjacent columns all the words that are tagged with "nn" or variations thereof. Based on a study of noun phrases in a sample of reports, a set of corresponding tag sequences was developed and this can be found in reference [7]. This set was used to extract noun phrases from all the reports
Post Processing: Post processing was aimed at producing
one and/or more of the following representations: a list of un-inspected
terms, a list of inspected terms, a list of terms grouped into
distinct categories, a hierarchical tree-like representation of
the terms, and a matrix representation of the cross-references
between terms. This paper focuses only on the first of the representations.
This is the same list as the output of the extractor except that
duplicated terms have been removed. From this list one can count
the number of distinct noun phases in the document. In addition,
using the TEXAS program, a concordance of noun phrases was produced,
and this allowed one to study the variation in noun modifiers.
The interest in an un-inspected list is for reasons of efficiency.
Inspection and other downstream operations take up additional
time and are less automated.
Results
Variation in number of distinct noun phrases
During the fall quarter, the average number of distinct
noun phrases per document was 542. The figure rose to 827 in
winter and then to 1066 in the spring quarter. The maximum number
of noun phrases in a document was 1552 and the minimum was 287.
Table 2 shows the number of distinct noun phrases for each project
for all three quarters.
Table 2. The number of distinct noun phrases generally increased from the fall to the spring
Variation in noun modifiers
While the numbers are useful for indicating the
general trend, the changes in the noun modifiers provides the
qualitative significance of the numbers. Table 3 gives an overview
of the change in a sample of noun phrases for the cinch project
between the fall quarter and the winter quarter. Observe that
the changes reflect a growing level of detail in the project.
Comparison with Project Grade
To test the hypothesis of this work, it was necessary to know the relation between the number of distinct noun phrases and the class grade. In Figure 3, the project grades have been superimposed on a graph of the quarterly variation of the number of noun phrases.
To calculate the level of association between the
two variables, a gamma table was constructed for the data points.
Gamma is a measure of association between two ordinal variables
and indicates for a pair of cases the degree to which their ordinal
ranking on one variable will correspond (positively or negatively)
to their ordinal ranking on the other The results showed that
gamma is equal to 1.0 for the winter quarter and 0.71 for the
spring quarter. Hence, project grade is positively associated
with the number of distinct noun phrases.
Table 3. A closer look at the change in the number of noun phrases from the fall quarter to the winter quarter in a sample project shows that several terms were repeated, a few terms were dropped, while many new terms were added, for example "acceleration of the window", "contour of the window", "upwards window force" and "window closing force".
Error Sources
Comparison with inspected noun phrases
In general, the numbers were lower for the inspected
noun phrases but the trend remained the same for both lists.
Incorrect Tagging
In previous tests, the Xerox tagger resulted in
an accuracy of 96% when used on the brown corpus [10]. In this
work there were not too many tagging mistakes hence the same accuracy
figure is assumed.
Synonyms
It was difficult to automatically determine the
synonyms of noun phrases. For single nouns, it would be much easier,
however several nouns have more than one synonyms and so this
would remain a difficult area.
Figure 3. When the winter and spring quarter grades were superimposed on the plot of noun phrase score versus academic quarter, it was found that the reports with low number of noun phrases corresponded to those projects with low grades and vice versa, those with a high number of noun phrases had a higher grade.
Discussion and Summary
The methodology demonstrated that accurate analysis
of the distinct noun phrases in engineering design documents can
be carried out in a very short time. Furthermore, statistical
results presented in Table 2 show that the average number of distinct
noun phrases in these documents is very high and increases from
one quarter to the next. Finally, the gamma association between
the number of distinct noun phrases and the class grade was positive
and varied from 1 (a perfect rank order predictor) in the winter
quarter to 0.71 in the spring quarter. Each of the results will
be considered separately.
Efficiency
The efficiency of analyzing noun phrases was important
from a practical standpoint since faster processing time enabled
more reports to be analyzed.
Association
There are two ways to look at the positive association
between the number of noun phrases and the grade. First, it reflects
an emphasis on documentation and the substantiation of evidence
in the grading practice. The teams that provide more detailed
descriptions of their requirements and their designs get better
grades. This fact is corroborated by the results of the Lincoln
Arc Welding competition where the top awards were given to projects
with high number of noun phrases, Figure 4.
Figure 4. At the end of the spring quarter, the teams participate in a number of national design competitions. The most regular of these being the Lincoln Arc Welding competition. The figure shows the awards for each project beside the number of distinct noun phrases for the spring quarter. Observe the position of the top awards namely the best-in-program and the silver relative to the bronze awards. This external assessment is an important complement to the internal assessment by the instructor
Second, the fact that there was a high positive association in the winter quarter and in the spring quarter is indicative of a consistency in grading practices. Low noun phrase scores were assigned the letter grade B or B+ and high noun phrase scores were assigned the letter grade A or A+. It was not possible to make this sort of differentiation based on the readability of the documents for which the gamma association with the class grade was low (0.36 for the winter quarter and 0.31 for the spring quarter). The gamma association between the grade and the number of words in the document were higher than that of the readability but lower than that of the noun phrases (0.93 in winter and 0.49 in spring). Hence the number of noun phrases appears to be an important measure for ranking projects. These are in reality all that can be inferred from the data. They leave unanswered many questions such as those dealing with the quality of the hardware, the circumstances of the project team, the differences in the project requirements and the truth of statements asserted in the documents. These types of questions will require a human evaluator.
Quantity
The third main result concerns the large number of noun phrases. There appears to be four primary reasons why the number of noun phrases generated in design is high.
First, design involves the search for alternatives.
In ME210, there is an emphasis on the generation of several alternative
solutions. Depending on the number of components and the pertinent
features of each alternative, the number of distinct noun phrases
will be large.
Second, the artifact interacts with different physical
environments during its development. The ME210 process is geared
towards the delivery of well justified hardware prototypes. This
process includes re-examination of the requirements proposed by
the client, identification of existing hardware and patents, selection
of materials, and the manufacturing and testing of components.
Each of these activities generates new noun phrases. For example
a re-examination of the requirements often involves a description
of the operational environment for the desired artifact. In the
manufacturing phase, there is often concern about the material
and the manufacturing processes. The testing phase brings into
the picture the test equipment and instruments. From the activity
point of view therefore the progress of an idea from concept to
hardware involves an increase in the number of noun phrases associated
with it.
Third is the fact that engineering design is a
goal-driven activity. This results in important shifts in the
conceptual focus. The following paragraph shows how the connection
between goal-attainment and focus-shifts relates to changes in
the modifiers of individual nouns and hence an increase in the
number of noun phrases.
"In order to reduce the cost of central shaft,
the material of was changed to steel, and its diameter was increased
to keep the same load bearing capability."
The paragraph describes the relationship between
cost reduction (goal), behavior (material properties), structure
(geometrical properties), and performance (goal) and from it the
following list of noun phrases can be extracted:
. cost of central shaft
. material of central shaft
. diameter of central shaft
. load bearing capability of central shaft
Fourth is the fact that engineering design is a method-based activity. This results in important shifts in the medium of expression. In a recent ethnographic study of mechanical engineering design classroom [11], the researchers enumerated six types of assignments that were used to aid both the process of doing design and the process of learning to do design. These assignment types are shown in Figure 5 and provide a way of classifying design methods.
Figure 5. Noun phrase analysis will reflect the different activities that takes place during design and serve as a critical feedback signal for monitoring and improving the process of doing design and learning to do design.
The key thing to observe is that with the exception
of the artifact-mediated assignments and to a lesser degree the
graphics-mediated assignments the other types of assignments will
by necessity generate and use noun phrases. This observation
is not limited to the classroom. An earlier series of case studies
of highly successful Japanese companies described knowledge creation
in them as a process of making tacit knowledge explicit.
"To convert tacit knowledge into explicit knowledge
means finding a way to express the inexpressible. Unfortunately,
one of the most powerful management tools for doing so is also
among the most frequently overlooked: the store of figurative
language and symbolism that managers can draw from to articulate
their intuitions and insights. At Japanese companies, this evocative
and sometimes extremely poetic language figures prominently in
product development" [12]
Based on interviews with managers and engineers in these companies the authors enumerated three ideal steps that were required to complete the knowledge creation process: the use of metaphors, the use of analogies, and the building of a model. Among the metaphors for example were such terms as tall-boy car, and man-maximum machine-minimum concept. Both of these were important ideas in the early stages of developing the Honda City, the company's distinctive urban car.
Therefore, the wide range of methods used in design to express the inexpressible and to transform vague ideas to physical reality, whether in the classroom or in industry appear to be primary generators of noun phrases.
In summary, the search for alternatives, the change
in physical environments, the necessary shift in focus due to
the interdependency of design goals, and the wide range of media-in-use
results in a high number of noun phrases which will change with
time. The methodology described in this paper demonstrates that
these changes can be captured and that they provide an important
signal of the design process which associates positively with
design performance. Figure 5 is a simple feedback model that illustrates
how the analysis of noun phrases can be used to better help students
do design and learn to do design.
References
