Research and Tools for Analyzing BME Design Team Projects

1
Research and Tools for Analyzing BME Design Team
Projects

• Mary Besterfield-Sacre
• Larry Shuman
• University of Pittsburgh

• Phil Weilerstein
• Angela Shartrand
• National Collegiate Inventors Innovators
  Alliance

October 7th 2009
2
Motivation

• To improve design education, engineering
  educators need to better understand how student
  teams navigate the process from concept to
  prototype.

3
Our Research Involves

• Bio-engineering capstone level engineering
• Team based approach
• NSF BES-RAPD
• The BME-IDEA Competition Assessing Innovative
  Design in Biomedical Engineering Education
• Student Reflection Tools

4
BMEidea Competition - Objectives

• Focus on innovative design entrepreneurship
• Promote and support experiential educational
  opportunities in product design, innovation, and
  entrepreneurship
• Bring academic and industrial biomedical
  engineering communities together
• Connect students/academic community to real world
  translation and commercialization opportunities

5
BMEidea Competition - Requirements

• Documentation of final design
• Prototype (photo, video)
• Proof that design is functional and will solve
  problem
• Assessment of patentability
• Proposed regulatory pathway
• Market analysis (with estimated costs)
• Business plan (strategy for commercialization and
  opportunity statement)

6
BMEidea Competition judging criteria

• Technical and economic feasibility
• Clinical utility impact
• Regulatory strategy
• Market potential
• Novelty and patentability
• Potential for commercialization

7
BMEidea Competition outcomes

• Experience working in multidisciplinary teams
• Ability to articulate ideas in a market and
  clinical context
• Better understanding of FDA requirements,
  regulatory strategy
• Translation Commercialization outcomes
• Licensing
• New ventures

8
Student Reflection Tools - Overview

• Real time assessment (Individually)
• Web-based reflection (twice per week)
• For the entire project
• At the end wrote an essay about their design
  experiences, influences, and contributions
• Retrospective analysis - Process maps
• Team-based reflection
• At the completion of the design
• Focus activities that students engage in

9
THE STUDY
10
Actually two studies in one

• In-depth in-process analysis
• Our students
• Three cohorts
• Pitt 07-08
• Pitt 08-09
• Rose-Hulman 08-09
• 26 teams

• Broad retrospective assessment
• Cream of the crop
• BMEIdea
• 07-08
• 08-09

11
In-Depth StudyBioengineering Capstone Design

• Independent Variables
• Design process refection
• Twice per week
• Stage of the design process activities
• Summarize how the team is progressing
• Explain any ah-ha experiences during the past
  week
• The stories
• Design influences essay and Background
  questionnaire
• Team Developer 08-09 only
• Team Process Map

• Dependent Product Variables
• Final design, prototype report
• Graded via rubric/scale
• Technical Performance Standards
• Working Prototype
• Documentation
• Innovation
• Overall Impact the Market

12
Broad retrospective assessmentBMEIdea
participants and Capstone

• Independent Variables
• Team Process Map
• Pitt bio-engineering teams 18 teams
• Rose-Hulman 8 teams
• BME-Idea Competition 2 competitions
• NOW 75 maps!
• Interviews with the top winners

• Dependent Product Variables
• Graded via rubric/scale
• The BMEIdea Ratings

13
ACTIVITIES WHAT STUDENTS SAY THEY ENGAGE IN
14
Design Product DevelopmentActivities

• Technical
• Strategic
• Competitor
• Human
• Societal
• Financial
• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

15
Design Product DevelopmentActivities
Stage 1 Opportunity Identification

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

16
Design Product DevelopmentActivities
Stage 2 Design and Development

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

17
Design Product DevelopmentActivities
Stage 3 Testing and Preproduction

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

18
Design Product DevelopmentActivities
Stage 4 Introduction and Production

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

19
Design Product DevelopmentActivities

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

Stage 5 Life Cycle Management
20
Design Product DevelopmentActivities

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

Stage 5 Life Cycle Management
21
Design Product DevelopmentActivities

• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

Stage On-going
22
Design Product DevelopmentActivities

• Technical
• Strategic
• Competitor
• Human
• Societal
• Financial
• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

23
Design Product DevelopmentActivities
Ratings by nine biomedical/bio-engineering design
experts from both industry and academia on the
most important elements

• Technical
• Strategic
• Competitor
• Human
• Societal
• Financial
• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.

24
ASSESSMENT TOOLSIN-PROCESS
25
Web-based reflective journaling to capture
engineering design activities

• Evaluate how engineering students
• Navigate the process of design from initial
  conception to product prototype, and
• Determine the extent that resultant innovative
  designs are a function of the process used
• Motivated by the work of Gorman
• Two term bio-engineering senior capstone course
• Prototype of medical device is the artifact
• Gorman, M.E., et. al. (2004). Collaborative
  Research into the Societal Dimensions of
  Nanotechnology A Model and Case Study. IEEE
  Technology and Society Magazine, 23, 4, 55-62.
• Spickard-Prettyman, S., et. al. (2005) Using a
  Vertically Integrated Team Design Project to
  Promote Learning and an Engineering Community of
  Practice, Proceedings of the ASEE Annual
  Conference.

26
Web Based Reflective Journaling
27
Reflective Journaling
28
A Sample of One Teams Activities Over Two Weeks
29
Team - TNH
30
Team - TNH
31
Team - TNH
32
Team - TNH
33
Team - JJEM
34
Team - JJEM
35
Team - JJEM
36
ASSESSMENT TOOLSRETROSPECTIVE
37
Process mapping to capture engineering design
activities

• Teams reflect upon and explain their experiences
  from idea conception to submission
• Teams document the relationships and importance
  of the various elements they employed
• To facilitate the process map
• Use a technique similar to concept maps
• Use an exhaustive list of elements that span the
  stages of design and product development
• Besterfield-Sacre, M., J. Gerchak, M. Lyons, L.J.
  Shuman, and H. Wolfe, Scoring Concept Maps
  Development of an Integrated Rubric for Assessing
  Engineering Education, Journal of Engineering
  Education, 93(2), April 2004, pp. 105 116.
• Golish, B., M. Besterfield-Sacre, L. Shuman,
  Comparing the Innovation Processes in Academic
  and Corporate Settings, Journal of Product
  Innovation Management, 25 (1), 2008, pp. 47-62.
• .

38
Team Process Map Triangulation - Reflective
Journals
39
SOME ANALYSES TO DATE
40
IN DEPTH STUDYQUALITATIVE AND QUANTITATIVE
APPROACHEXAMPLE WITH ONE TEAM
41
The TeamsStory
42
(No Transcript)
43
(No Transcript)
44
Dyms Model

• Summarized activities into model
• Activity could be in one or more stage

45
Team Dynamics Dyms Model Over Time
46
ExampleAssociations BetweenStagesData mining
technique
47
Associationsand Scores
48
Associationsand Scores
49
What are we starting to see

• With the mixed method approach
• Teams that do poorly have high associations at
  the beginning stages
• They really dont progress through the entire
  process
• Teams that do well have high associations in the
  later stages or are consistent throughout the
  process
• This seems particularly true with innovation and
  technical performance

50
SOME MORE ANALYSES TO DATE
51
PROCESS MAPS
52
A few hypotheses that we are testingNot All
Data is Analyzed

• Ho1 Do teams utilize similar activities when
  developing their designs or is the process
  specific to the particular design?
• How do these compare with experts
• Ho2 Do teams utilize similar paths when creating
  their designs and is this indicative of design
  instruction at their particular institution?
• Ho3 Do certain activities and/or process paths
  relate to the overall quality of the design?

53
Ho1 Do teams utilize similar activities

• Phase 1 - 32 of the elements were utilized on
  at least the majority of maps
• Majority defined
• 8 of the 9 Capstone
• 21 of the 27 BMEIdea

54
Ho1 Do teams utilize similar activities

• Phase 2 - 21 of the elements were utilized
• Phase 3 13 of the elements were utilized

55
Ho1 Do teams utilize similar activities

• Phase 4 no consistency

• Phase On-Going 33 elements were utilized

56
Ho1 Do teams utilize similar activities
Elements appearing in all maps
57
Ho1 Do teams utilize similar activities
Elements appearing in all maps
58
Ho1 Do teams utilize similar activities
The Important Elements
59
Important to Experts

• Customer Needs Analysis/Feedback
• Brainstorming
• Analysis, Evaluation and Reporting of Test Data
• Alpha Testing
• Prototype Testing
• Design Reviews
• Design/Prototype Reviews
• Product Design Selection

• Reliability Testing, Test to Failure, Limit
  Testing
• Test Method Definition
• Customer Feedback Evaluation
• Refine Tests and Methods
• Documentation
• Modeling and Simulation to Study Design
• Optimization of Design

60
Ho1 Do teams utilize similar activities
The Important Elements
dd
dd
dd
dd
61
Ho1 Do teams utilize similar activities

• Bottom line, yes
• Trails off as teams progress through the phases
• Prototype Development and Prototype Testing are
  consistent across the two groups
• Some consistency with experts
• But
• True of all design or a function of the
  instructor/class?
• Comparison with two institutions (2008-09)
• Comparison with two years of BME-Idea participants

62
Ho2 Do teams utilize similar paths

• Algorithm to track the from-to paths
• Capstone (8 maps)

63
Ho2 Do teams utilize similar paths

• BME Idea (13 maps)

64
Ho2 Do teams utilize similar paths

• Repeating paths between elements are not shared
  by the majority of the teams
• The product design/development process is unique
• There are a large number of activities that can
  be conducted during the process
• Path consistency among the various teams is
  arduous
• Investigating differences between novice and
  expert design teams

65
Ho3 Do certain activities relate to the overall
quality of the design?
66
Ho3 Do certain activities relate to the overall
quality of the design?
Low scoring maps were below the average number of
elements observed
67
Ho3 Do certain activities relate to the overall
quality of the design?
Lower scoring teams documented substantially
fewer societal elements
68
Next Steps

• Complete analyses
• Intention to make both tools readily available to
  schools.
• Several schools in the BMEIdea competition have
  requested to use process maps
• Accreditation provides motivation to better
  measure the design process beyond review of the
  final artifact

69
With Thanks to Our Sponsors

• Supported under NSF BES RAPD collaborative grant
  award 0602484
• With additional support from