Empirical Studies of Design Ideation:

1
Empirical Studies of Design Ideation Alignment
of Design Experiments with Lab Experiments
JamiJ. Shah Noe Vargas-Hernandez Mechanical and
Aerospace Engineering Arizona State University,
Tempe, AZ
Steve M. Smith David R. Gerkens Department of
Psychology Texas AM University, College Station,
TX
Muqi Wulan Department of Industrial and
Manufacturing Systems Engineering Beijing
University of Aeronautics and Astronautics,
Beijing, China
NSF Grant Number DMI-0115447
ASME 2003 International Conference on Design
Theory and Methodology September 5, 2003, Chicago
Il
2
INTRODUCTION
MOTIVATION
There are many methods for design synthesis How
useful are these? Which ones are better? There
isnt much empirical data on specific
effectiveness of Idea Generation (IG) methods
3
INTRODUCTION
OBJECTIVE
If our experiments at different levels have the
same results, we can establish a connection and
we can run more of the simpler Lab
Experiments More Experiments means more empirical
data on specific IG methods Our alignment
approach is still WIP and is part of a bigger
project Our ultimate objective is to develop a
theoretical model of design ideation Such a
model would help us better understand IG methods
4
INTRODUCTION
DESIGN IDEA GENERATION METHODS
Figure 1. Classification of Idea Generation
Methods (Shah et al., 2000)
5
PRELIMINARY RESEARCH
PAST EXPERIMENTS
For several years we have been doing experimental
studies to define the effectiveness of ideation

• We used two distinct approaches to conducting
  experiments
• Direct Method IG methods are studied as a whole
  
• Indirect Method Ideation Components are studied

• And two distinct approaches to assessing the
  effectiveness
• Process Assess the ideation process (e.g.
  protocol studies)
• Outcome Assess the ideas produced (e.g.
  sketches)

6
PRELIMINARY RESEARCH
IDEATION COMPONENTS
Ideation Components are mechanisms that are
believed to intrinsically promote IG or to help
designers overcome mental blocks.

• Examples of Ideation Components
• Provocative Stimuli
• Deferred Judgment
• Flexible Representation
• Frame of Reference Shifting
• Incubation
• Example Exposure

• Examples of Blocks
• Being Judgmental
• Emphasis on Quality
• Lack of Motivation
• Having a tight grip on problem specs.
• Rigid Problem Representation
• Design Fixation
• Imposing Fictitious Constraints

These are common Known in Engineering Design
Research and acknowledged by Cognitive Psychology
7
PRELIMINARY RESEARCH
OUTCOME EFFECTIVENESS
Four measures were defined in our previous
projects (Shah, Kulkarni, and Vargas-Hernandez,
2000).
Table 1. Effectiveness Measures for Idea
Generation Outcome.
8
RESEARCH APPROACH
FUNDAMENTAL ISSUES
How can we compare results from experiments at
different levels?

• LAB EXPERIMENTS
• (Done AM Psychologists)
• Focus on atomic cognitive processes
• Little similarity between the condition of these
  experiments and design concept generation in the
  real world.

• DESIGN EXPERIMENTS
• (Done by ASU Engineers)
• Simulate real world better
• Incorporate more and complex variables
• Require prohibitive number of experiments
• Unable to explain the performance of methods
  under different conditions.

How can we align these two?
9
RESEARCH APPROACH
RESEARCH APPROACH
How can we compare results from experiments at
different levels?
Figure 2. Research Strategy
10
RESEARCH APPROACH
IDEATION COMPONENTS
The number of components identified is more than
a dozen. Because of limited resources and the
prohibitive number of experiments required to
study all possible interactions, only the most
relevant were selected.
Table 2. Selected Components
11
RESEARCH APPROACH
DESIGN OF EXPERIMENTS (DOE)
Two levels were considered for each of the
selected components. Although more levels could
be defined, it is recommended to run experiments
initially with few levels.
Full Factorial Experiments A Frame of Reference
Shifting B Incubation C Example Exposure

• Simple Comparative Experiments
• Provocative Stimuli
• Suspend Judgment
• Flexible Representation

12
RESULTS
DESIGN EXPERIMENTS DONE BY ASU ENGINEERS
Experiment Variables Subjects Undergraduate
Engineering students. Comparable
expertise/knowledge between subjects is assumed.
Task One design problem was used for all
experiments. The objective was to design a device
for throwing a ping-pong ball the farthest
distance. A list of allowed materials was given
this to improve the quality of sketches. Idea
Recording Subjects were asked to generate ideas
individually using sketches Nuisance
Variables Similar environmental settings procured
for each run (classroom, noise, light, etc.)
13
RESULTS
DESIGN EXPERIMENTS DONE BY ASU ENGINEERS
Figure 3. Sample Sketches from Design Experiments
14
RESULTS
LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS

• Experiment Variables
• Subjects
• Undergraduate Psychology students. Comparable
  expertise/knowledge between subjects is assumed.
• Task
• Listing members of large taxonomic categories,
  sense impression categories, and ad-hoc
  categories.
• Divergent thinking, unusual uses of common
  objects.
• Idea Recording
• Subjects were asked to generate ideas
  individually using text for member listing tasks
  and sketches for divergent thinking tasks
• Nuisance Variables
• Similar environmental settings procured for each
  run (classroom, noise, light, etc.)

15
RESULTS
LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS
Figure 4. Sample Sketches from Lab Experiments
16
RESULTS
IDEATION COMPONENT INCUBATION
Table 5. Lab Experiments Done by TAMU
Psychologists Mean Ideation Effectiveness Scores
Table 6. Design Experiments Done by ASU
Engineers Mean Ideation Effectiveness Scores
17
RESULTS
IDEATION COMPONENT INCUBATION
Table 7. Two Sample t-test
18
RESULTS
IDEATION COMPONENT INCUBATION
Table 8. Correlation Between Lab and Design
Experiments
19
CONCLUSIONS
IDEATION COMPONENT INCUBATION

• Based on the results from (TAMU Psychologists)
  Lab and (ASU Engineers) Design Experiments,
  Incubation increases the effectiveness of ideas
  generated. Results correlate at both levels and
  show a satisfactory confidence level.

• Incubations positive impact on Design Ideation
  is substantiated by concrete Engineering evidence
  (from ASU Design Experiment results) and has a
  theoretical basis (from TAMU Lab Experiment
  results).

20
CONCLUSIONS
OVERALL

• More experiments needed to prove connection
• The alignment procedure provides a framework for
  comparison between both levels. Results for
  Incubation exemplify how the alignment works.

Connection Proven
Experiments on other components have been
completed at ASU and TAMU.
21
FURTHER CONCLUSIONS

• According to our results we found Frame of
  Reference Shifting (FORS), Incubation (I), and
  Example Exposure (EE) to have similar main
  effects.
• Interaction effects weren't that clear, probably
  because some components are much alike (specially
  FORS and EE) and hence aren't independent.
• This generates a question Maybe these Ideation
  Components have the same effect on ideation ?

• Two or more Ideation Components sharing the same
  effect could belong to the same higher level
  Ideation Principle

22
FUTURE WORK
IDEATION PRINCIPLES
Table 9. Comparison of Ideation Principles with
Cognitive Components
23
FUTURE WORK
IDEATION PRINCIPLES

• Refine Ideation Principles and its
  Implementations (Ideation Components)
• Run more exercises, collect and analyze more data
  to prove/disprove our theory about Ideation
  Principles
• We still continue experimenting on Ideation
  Components, but with a better understanding of
  Ideation Principles, experiments can be better
  targeted (e.g. less redundant) and more efficient.

24
REFERENCES

• Altshuller, G., 1984, Creativity as an Exact
  Science, Gordon and Breach, New York.
• Dennehy, E.B., Bulow, P., Wong, F., Smith, S.M.,
  and Aronoff, J.B. (April, 1992). A test of
  cognitive fixation in brainstorming groups. Paper
  presented at the meeting of the Eastern Psych.
  Association, Boston, MA.
• Dodds, R.A., and Smith, S.M., 1999, Fixation. In
  M.A. Runco Pritzker (Eds.) Encyclopedia of
  Creativity, San Diego, CA Academic Press.
• Ericsson, K., and Simon, H., 1984- Protocol
  Analysis - verbal reports as data, MIT Press.
• Finke, R.A., Ward, T.B., and Smith, S.M., 1992,
  Creative Cognition Theory, Research, and
  Applications, Cambridge, MA MIT Press.
• Hale, C., Analysis of the Engineer Design
  Process in an Industrial Context, Grant Hill
  Pubs, Cambridge, 1987.
• Jansson, D. G., and Smith, S. M., 1991, Design
  Fixation, Design Studies, Vol. 12, pp. 3-11.
• Koestler, A., 1964, The art of Creation,
  Hutchinson and Co., London.
• Langley, P., and Jones, R., 1988, Computational
  model of scientific insight, in R. J. Sternberg,
  ed., The nature of creativity contemporary
  psychological perspectives, Cambridge University
  Press, NY.
• McKoy, F., 2000, Experimental Evaluation of
  Engineering Design Representations for Idea
  Generation, MS Thesis, Arizona State University,
  Tempe, AZ.
• Mckoy, F., Vargas-Hernandez, N., Summers, J. D.,
  Shah, J., 2001, Experimental Evaluation of
  Engineering Design Representation on
  Effectiveness of Idea Generation, Proceedings,
  ASME Design Theory and Methodology Conference,
  Pittsburgh, PA.
• Koestler, A., 1964, The art of Creation,
  Hutchinson and Co., London.
• Martindale, C., 1995, Creativity and
  Connectionism, in S. M. Smith et al., eds., The
  creative cognition approach, MIT Press,
  Cambridge, MA.
• Schön, D., 1991,Teaching and learning as a
  design transaction, in Research in Design
  Thinking Delft Press, 1991.

25
REFERENCES

• Schwartz, B.L. and Smith, S.M., 1997, The
  retrieval of related information influences
  tip-of-the-tongue states. Journal of Memory
  Language, 36, 68-86.
• Shah, J., 1998, Experimental Investigation of
  Collaborative Techniques for Progressive Idea
  Generation, Proceedings, ASME Design Theory and
  Methodology Conference, Atlanta, GA.
• Shah, J., Kulkarni, S., Vargas-Hernandez, N.,
  2000, Guidelines for Experimental Evaluation of
  Idea Generation Methods in Conceptual Design,
  Journal of Mechanical Design, vol. 122, no. 4,
  pp. 377-384.
• Shah, J. J., Vargas-Hernandez N., Summers, J. D.,
  Kulkarni, S., 2001, Evaluation of Collaborative
  Sketching as an Idea Generation Technique for
  Engineering Design, Journal of Creative
  Behavior, 353, pp.1-31.
• Smith, D. K., Paradice, D. B., and Smith, S. M.
  (2000). Prepare your mind for creativity.
  Communications of the Association for Computing
  Machinery , 43, 110-116.
• Smith, S. M., 1995, Creative Cognition
  Demystifying Creativity, in C. N. Hedley et al.,
  eds., Thinking and literacy the mind at work,
  Lawrence Erlbaum Associates, Hillsdale, NJ.
• Smith, S. M. and Blankenship, S. E., 1991,
  Incubation and the persistence of fixation in
  problem solving, American Journal of Psychology,
  104, 61-87.
• Smith, S. M., Carr, J. A., and Tindell, D. R.,
  1993, April, Fixation and incubation in word
  fragment completion. Paper presented at the
  meeting of the Midwestern Psychological
  Association, Chicago, IL.
• Smith, S. M., and Vela, E., 1991. Incubated
  reminiscence effects. Memory Cognition, 19 (2),
  168-176.
• Smith, S. M., Sifonis, C. M., and Tindell, D. R.,
  1998, Hints do not evoke solutions via passive
  spreading activation. Proceedings of the
  Twentieth Annual Meeting of the Cognitive Science
  Society, Madison, WI.
• Vargas-Hernandez, N. and Shah, J. J., 2002,
  Inventory of Creativity Exercises 1995-2002,
  Tech. Report ASU/DAL/IG/02-1, Arizona State
  University.
• Wallas, G., 1926, The Art of Thought, Harcourt,
  New York.