Download Postscript file for printing (1.9 MB)
Evaluating Product-Based-Learning Education
Larry Leifer
Center for Design Research
Department of Mechanical Engineering
Stanford University
560 Panama Street, Stanford, CA 94305-2232, USA
leifer@cdr.stanford.edu
http://cdr.stanford.edu
It is hypothesized that Product-Based-Learning (PBL) methodology and technology
referenced in this paper are evolving rapidly in a manner that will make
widespread PBL adoption and assessment financially feasible for graduate and
undergraduate engineering education. It is asserted that student-created case
material, a natural by-product of PBL curricula, will support learning re-use
and external validation of curriculum reform.
Engineering Education Reform has adopted constructivist learning theory without
fully adopting the education process implications. It is the thesis of this
paper that both reform and assessment objectives are served by moving
vigorously towards product-based-learning core-curricula. The author prefers
"product" to "project" or "problem" based learning because the word focuses our
attention on delivering something of value beyond an "academic exercise". The
objectives of PBL curricula may be stated briefly as follows [1]:
- familiarize students with problems inherent in their future profession;
- assure content and process knowledge relevant to these problems;
- assure competence in applying this knowledge;
- develop problem formulation and solving skills for these problems;
- develop implementation (how to) skills;
- develop the capacity to lead and facilitate collaborative problem solving;
- develop the skills to manage emotional aspects of leadership;
- develop and demonstrate proficiency in self-directed learning skills.
The achievement of PBL objectives rest, in large part, on constructivist
learning theory.
Students have been observed to learn in four different ways,
Figure-1. Kolb [2] proposed that repeated cycles of experiences moving
through these learning modalities improves understanding. The cycle starts
with perception and processing, two dichotomies which are considered orthogonal
to each other [2][3]. Beginning with a need for learning stimulated by an
immediate experience, a learner should cycle through questions created by the
learning environment which lead to a complete cycle of experiences and
understanding in depth. In concert, theory and practice underscore the
importance of experience and the need for regular feedback on the intention and
meaning of that experience as a learning opportunity.
Those who seek to support distant learning, asynchronous team work and
interdisciplinary collaboration must be vigilant about preserving the critical
features of an experiential learning framework while exploring the power of
emerging technology to bring these methods to more people, in more places and
at lower cost than has previously been possible.
Figure 1 Kolb's [1] model of experiential learning. Students learn in
four different ways. Kolb proposed that a cycle of experiences improves
understanding and builds bridges between theory and practice. This is a
qualitatively satisfying view of project-based-learning as well as learning in
general. It remains a challenge to objectively, even quantitatively,
demonstrate performance improvements when this iterative experience pattern is
experienced.
Educational theorists (including Jean Piaget) have developed a model of
learning by which students develop knowledge structures based on previous
experience. A theory of how to teach science known as the "scientific learning
cycle" is a direct outgrowth of Piaget's ideas and constructivism [4]. Our
experience in engineering design education supports the critical role of
experience (versus information or theory) as the learning medium. It is also
the coin by which designers, engineers, scientists and most professionals are
valued. One may summarize the lessons of constructivist research as declaring
that, quite simply, learning is best done by creating something, a product,
that embodies our knowing. This is the central tenant of
product-based-learning.
Vygotsky argues that knowledge is social before it is personal, which suggests
that it must be interactively and socially constructed. This is usually
observed through language usage though it can also be visualization. In this
model, the learning must be external and shared before it can be internalized
and made personal. Much of the current work on group-based learning derives
from his thinking [5]. Once again, this model reflects our experience that
engineering design is a social activity [6]. I have observed that design team
failure is almost always due to failed team dynamics. However, learning
failure is usually blamed on the individual. It is increasing clear that doing
design is a learning activity and I must speculate that more attention should
be given to the influence of team-dynamics on individual learning. Learning
is promoted in PBL situations where there is an open-ended opportunity to be
creative about one's learning.
Our curriculum technology focus is on Mechatronic systems design-development
(smart products). Mechatronics is frequently defined as the integration of
real-time software, electronics, and mechanical elements for imbedded systems.
In our definition, we include human-computer-interaction and materials
selection. Typical products in this domain include: disk drives, camcorders,
flexible automation-systems, process-controllers, avionics, engine controls,
appliances and smart weapons. Mechatronics is a particularly good medium for
introducing PBL because of its deep dependence on interdisciplinary
collaboration.
Implementation of this curriculum builds, in part, on recently developed
internet tools and services for distributed product-development teams (virtual
design teams). Such teams are common in the mechatronics field. Using the
World-Wide-Web (WWW) as an informal, work-in-progress document archive and
retrieval environment we electronically instrument design-team activity and
knowledge sharing for project management and learning assessment purposes.
Examples from our work in the ARPA-MADE program include:
Share: a MadeFast experiment (URL http://www.madefast.org/), 6
universities, 6 corporations, built a virtual company, delivered a product in 6
months and documented it all on the WWW [7];
ME210: a graduate curriculum in Cross-Functional-Team Mechatronic
Systems Design at Stanford (URL http://me210.stanford.edu), 14 companies, 45
students (1/3 remote) built and documented 14 Mechatronic prototypes [8];
Synthesis Mechatronics Curriculum: a multi-university (8)
co-development project focused on undergraduate mechatronics (NSF Synthesis
Coalition web (URL http://www.synthesis.org).
Corporate clients of the Stanford Center for Professional Development (SCPD)
are increasingly adamant about the need to give their employees a continuing,
life-long, education opportunity without losing them to full time study on
campus. Encouraged by this demand, Stanford's graduate level
"Cross-Functional-Team Mechatronic Systems Design" course was offered for the
first time in 1994-1995 to SCPD students across the country. The course is
intensely hands-on. Distributed design teams (typically 3 members) work on
different industry sponsored projects (typically 14 per year, the 1994-1995
academic year included Ford, GM, FMC, Lockheed, Pfizer, 3M, Raychem, NASA-JPL,
Redwood MicroSystems, Quantum, Toshiba, Seiko and HP Medical Products). The
deliverable, after 9 months at a 40% effort level, is a functional product
prototype and detailed WWW documentation of the development process and
product. These teams won 11 of 12 awards in the Lincoln Foundation Graduate
Design Competition in 1995.
A class information system based on the World-Wide-Web was created to meet the
interactive distance education challenge. ME210 utilizes broadcast video and
the Internet. The focus is on capturing and re-using informal and formal
design knowledge in support of "design for re-design". Broadcast video is
needed for high-bandwidth, real-time transmission of lectures and design
reviews originating on campus. The internet is used for low-bandwidth data
transmission, e-mail and knowledge capture. Electronic mail is used
extensively for communication between the teaching staff, students, coaches and
industry liaisons. Communication is automatically archived and organized by a
Hypermail utility on the 210-web. The class syllabus, 5 years of past student
reports, and each project's electronic notebook are part of the web. The
210-web functions as a perpetual class memory. It facilitates informal
knowledge sharing within the class and gives subsequent generations our legacy
knowledge.
In addition to distance education benefits, the dynamic nature of the 210-web
fundamentally changes the model of interaction between student teams, teaching
staff, coaches and industry liaisons. Before the 210-Web, product development
team progress was only observable at formal meetings and quarter
critical-reviews. With the 210-Web, work-in-progress is available for review
by all members of the 210 community, any time, any place. This option augments
and strengthens traditional briefings. It facilitates feedback to the design
team form the teaching team, coaches and corporate partners. Importantly,
teams can share their "lessons-learned" in real-time. All of this gives our
corporate partners a basis for judging the validity of our curriculum and the
value of their educational investment.
The course and the web become a model for "instrumented" learning. It is the
author's experience with this course that led to formulation of the following
PBL assessment-instrumentation model.
Product-based-learning integrates five key pedagogic themes, each central to
assessment: 1) externally sponsored projects motivate student learning; 2)
theory and practice are synthesized in hands-on development; 3) real-world
projects demand multi-disciplinary experience; 4) project management requires
problem formulation, teamwork, negotiation, oral communication, and effective
written documentation; 5), naturally occurring bi-products of project work
(proposals, presentations, lab-notes, products and reports) directly support
formative, summative and validative assessment.
PBL pedagogy themes map closely to the activity and issues of real product
development. Accordingly, the framework for our approach to learning
assessment is derived from observational methodology in the design research
community, especially the work of Tang [9] and Minneman [10], Figure-2. These
findings tell us what to instrument, when to instrument and to focus our
attention on mapping inputs to outputs, theory to practice. Engineers must
develop special critical thinking skills to map abstracts to hardware and back
again. They must reason about the product and the process that will bring it
into being. This is an extremely complex spatial, temporal and cognitive
space. It is a space best explored by cross-functional teams.
Figure-2 Video interaction analysis (VIA) of teams of designers doing
product development and teams of students learning engineering have revealed a
remarkably similar set of features. Foremost amongst these are the
realizations that: (1) design is active learning; (2) active, constructivist
learning is design like; (3) learning and design are social activities; (4)
design and learning are negotiated; (5) management of alternatives and
constructive use of ambiguity are critical skills in both endeavors; and (6),
design and learning are both based on re-use of prior knowledge, i.e., they
depend on re-design and re-learning.
The half life of fundamentals in any field is much longer than the half life of
today's technology. This focuses our teaching investment on learning and
fundamentals. However a large body of research in physics learning has shown
that students have difficulty connecting abstract fundamental concepts to their
understanding derived from experience [11],[12]. Brereton [13] explores how
concepts are used in practice and considers how teaching and learning can be
improved to help students integrate and leverage conceptual knowledge in the
engineering workplace. Given PBL objectives and these learning realities we
have adopted an automatic-control metaphor for identifying the role and
interactions between various assessment methods and their point-of-insertion in
the education enterprise (Figure-4).
Figure-3 Assessment is itself calibrated and validated (authenticated)
in Project-Based-Learning. Project outcomes are compared with industry
performance standards across courses and campuses. Results obtained from
assessment activity along these three feedback paths support triangulation
between findings across methods and assessors.
The guiding metaphor for our assessment and evaluation activity is one of
instrumentation. We use this term in the sense of observing both independent
and dependent variables in an automation environment [14]. The metaphor is
visualized in Figure-3. In each assessment situation we identify the locus for
instrument insertion, the variables of interest and observation method of
choice. The nature of the activity and information structure in the process
both influence these choices. Two examples are given in Figure-4ab for video
interaction analysis (one of the most powerful, if labor intensive methods) and
in-class minute essays (one of the most rewarding, low effort methods).
Guidelines for learning instrumentation include:
- observe teaching & learning;
- observe process & content;
- actively control the education system with feedback;
- formative learning feedback keeps the class on target
- summative teaching feedback keeps the curriculum on target
- external validative feedback keeps the university on target.
- who listens to the feedback defines which loop it is in.
Figure-4ab The locus of activity and the placement of evaluation
instruments is superimposed on the instrumentation model. On the left,
video-interaction-analysis is shown to be effective in many aspects of the
teaching-learning environment, but perhaps least attractive in the classroom
itself. Three alternatives for formative classroom assessment are shown on the
right, with minute papers and student journals being amongst the most
frequently used. Concept maps can be particularly effective for isolating
student misconceptions about fundaments.
Bridges and Hallinger [1], present an interesting dilemma for the assessment of
PBL curricula. They point out that we have striven to transfer the learning
activity locus-of-control from the instructor to the learner. This includes
the moment to moment structure of learning, including the questions to be
answered, and formulation of the problems to be solved, . Now they argue, is
it not appropriate to transfer the locus of control for judgment from the
instructor to the student. Figure-5 presents this issue in the form of a
matrix that clearly reveals the combinations and permutations for structure and
judgment. Type-5 situations are the norm today. Type-1 situations should be
the norm tomorrow. And, increasing, outsider driven structure and judgment
(Types 9) will be required to validate the education enterprise.
Figure-5:
Building on the work of Bridges and Hallinger [1], PBL assessment can be
structured by the instructor, the student or an outsider. Learning, whether
formative or summative may be judged by the instructor, student or outsider.
But validative assessment can only be structured and judged by outsiders The
author sees no reason rule out any combination of alternatives and finds value
in the prospect of using the full range of options in a pedagogically informed
manner with the role of the outsider focused on validation.
Project-Based-Learning is both old and new. It is familiar as the classical
conveyor of skills through apprenticeship. It is new as re-interpreted to be
the formal process of linking and integrating theory and practice. Through
emerging communication technology it is almost revolutionary in given power
back to the learner to structure and assess one's own learning performance at
all levels of professional development.
We sincerely thank all students and faculty who have graciously taken part in
our ethnographic video interaction analysis studies. In addition we thank all
collaborating members of the NSF Synthesis Coalition and NSF for supporting
this work (University of California at Berkeley (current headquarters), Cornell
University (past headquarters), Stanford University, Southern University,
Tuskegee University, Hampton University, Iowa State University and the
University of California State Polytechnical College at San Louis Obispo. The
Share-Madefast project is supported, in part, by the ARPA (the Advanced
Research Projects Agency).
- Bridges, E.M., and Hallinger, P., "Problem Based Learning: in leadership
development", ERIC Clearinghouse on Education Management, Eugene, Oregon,
1995
- Kolb, David A. "Experiential Learning" Englewood Cliffs, N.J.
Prentice-Hall, 1984
- Harb, J., Durrant, S., Terry, R., "Use of the Kolb Learning Cycle and the
4MAT System in Engineering Education", ASEE Journal of Engineering Education ,
82, 3, 70-77, 1993
- Lawson, A. E., Abraham, M.R., Renner, J.W., "A Theory of Instruction: Using
the Learning Cycle to Teach Science Concepts and Thinking Skills", Monograph 1,
National Association for Research in Science Teaching, Cincinnati, OH, 1989
- Moll, L., "Vygotsky and Education", Editor, Cambridge University Press,
1990
- Leifer, L., Koseff, J., and Lenshow, R., "PCL White paper: report from the
International Workshop on Project Centered Learning", 7-11 August, 1995,
Stanford, CA
- Toye, G., Cutkosky, M., Leifer, L., Tenenbaum, M., and Glicksman, J.,
"SHARE: A methodology and environment for collaborative product development,"
International Journal of Intelligent and Cooperative Information Systems, Vol.
3, No 2, pp.129-153, 1994
- Hong, J., and Leifer, L., "Using the WWW to Support Project-Team
Formation", Proceedings of the FIE'95, 25th Annual Frontiers in Education
Conference on Engineering Education for the 21st Century, Atlanta, Georgia,
November 1 - 5, 1995
- Tang, J. C. and Leifer, L. J. , "An Observational Methodology for Studying
Group Design Activity", Research in Engineering Design, 1991, Vol. 2, No. 4.,
pp. 209-219., reprinted in Waldron, M., Reading in Design Research Methodology,
1992
- Minneman , S., and Leifer, L., "Group Engineering Design Practice: the
social construction of a technical reality", Proceedings of the International
Conference on Engineering Design (ICED'93), The Hague, Netherlands, August
17-19, 1993
- Kuhn, Thomas, "The Structure of Scientific Revolutions", University of
Chicago Press, Chicago, IL 1962
- diSessa, A. A., "Toward an Epistemology of Physics," Cognition &
Instruction, 10 (2 & 3), 1993
- Brereton, M.F., S.D. Sheppard, L.J. Leifer, "How Students Connect
Engineering Fundamentals to Hardware Design," in Proceedings of the 10th
International Conference on Engineering Design. Held: Prague, August 22-24,
1995. WDK 23 Vol 1 Publ by Heurista, Zurich 1995. p 336-342.
- Atman, C., Leifer, L., Olds, B., and Miller, R., "Innovative Assessment
Opportunities", National Technical University satellite broadcast (video tape
available, 14 November, Fort Collins, CO, 1995