NIST Gaithersburg

1
National Institute of Standards
Technology Information Technology
laboratory Statistical Engineering Division
Catapults, Funnels, Science, and Statistically
Designed Experiments
NIST Gaithersburg Green Auditorium June 5, 2008,
330-430
2
Outline
A Few Problems ...
A Structured Approach ...
1. Catapults 2. Funnels 3. World Trade Center 4.
Carbon Nanotube Contamination 5. Chemical Yield
(Benchmark) 6. NIST Scientific Apps
1. Generic Mapping / Model 2. Problem-Solving
Framework 3. Problem Classification 4. DEX
Worksheet 5. Orthogonal Fractional Designs 6.
Sample Size Considerations 7. DEX Analysis 8.
DEX, Data, Analysis WTC
3
Part 1 A Few Problems ...
1. Catapults
4
k ...
5
2. Funnels
6
k ...
7
3. World Trade Center
8
WTC Impact Core Damage Assessment Q. After the
plane impact of the WTC South Tower, there was no
recorded data as to how many of the interior 47
columns of the building were damaged. A
finite-element analysis (FEA) program was written
to simulate the impact. The plane was modeled by
1.4 million elements. What factors most affected
the performance of this FEA code? What factors
could be eliminated as unimportant?
Q. What factors affect quality of FEA code
predictions?
9
Construction 1.4 Million Elements for Entire
Plane (Labor Intensive)
Applied Res. Assoc.
Y Core Columns Damaged
10
. Experiment Design ?
Y Core Columns Damaged
Factors
X1
X2
X3
X4
X5
X6
X7
X8
X9
X10
X11
X12
X13
Y
1 2 3 4 5 6 7 8 9 10 11 . . . (50)
?
Runs
11
. Experiment Design ?
k ...
12
4. Carbon Nanotube Contamination
Y Carbon Nanotube Reduction ()
X1 Clay Concentration X2 NOM
Concentration X3 Bio
Concentration X4 CNT Concentration X5
Coagulant Type
k ...
13
5. Chemical Yield (Benchmark)
Y Chemical Yield ()
X1 Feed Rate (l/min) X2 Catalyst () X3
Agitation Rate (rpm) X4 Temperature (C) X5
Concentration ()
k ...
14
6. NIST Scientific Apps
Physics
Elect. Elect. Eng.
Material Science
Manufacturing Eng.
Build. Fire Res.
Information Tech.
Chemistry
15
NIST
Physics Plutonium Troubleshooting (SURF)
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability
Material Science MALDI TOF Spectrometry
Manufacturing Eng. Scatterfield Microscopy
Build. Fire Res. World Trade Center FEA Core
Damage
Information Tech. Abilene Network Loss Rate
Chemistry 3D Nanoscale Chem Imaging
16
NIST
Physics Plutonium Troubleshooting
(SURF) Am241/243 Peal Deconvolution Alg. Acc.
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers
17
NIST
Physics Plutonium Troubleshooting
(SURF) Americium 241/243 Peal Deconvolution Alg.
Acc. Cesium 137 Detection
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp.
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy Dental Polysac
Adhesion
Manufacturing Eng Scatterfield Microscopy Genetic
Alg. for Machine Tooling SMS Smart Machining
System
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST)
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error
18
NIST
Physics Plutonium Troubleshooting
(SURF) Americium 241/243 Peal Deconvolution Alg.
Acc. Cesium 137 Detection Efficiency of Gamma Ray
Emitters
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp. DAC
(Digital-to-Analog Converter) Calibration
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy Dental Polysac
Adhesion Bio Knee Cartilage Regeneration
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling SMS
Smart Machining System OLES Forensic Imaging of
Gun Casings
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST) Tall Building Deflection
Safety Codes
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration Accelerated
Testing of Compact Discs
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error Microarray Sensors for Toxic Gas
19
NIST
Physics Plutonium Troubleshooting
(SURF) Americium 241/243 Peal Deconvolution Alg.
Acc. Cesium 137 Detection Efficiency of Gamma Ray
Emitters Remote Detection of Hot-Box Radiation
(SURF)
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy Dental Polysac
Adhesion Bio Knee Cartilage Regeneration Ceramic
Machining Strength
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp. DAC
(Digital-to-Analog Converter) Calibration OLES
Firefighter Infrared Imaging Devices
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling SMS
Smart Machining System OLES Forensic Imaging of
Gun Casings
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST) Tall Building Deflection
Safety Codes HHS CONTAM Home Pollution
Dissemination
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration Accelerated
Testing of Compact Discs Apache/Linux Web
Processing Time
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error Microarray Sensors for Toxic Gas DHS
Bio-Agent Detection
20
NIST
Physics Plutonium Troubleshooting
(SURF) Americium 241/243 Peal Deconvolution Alg.
Acc. Cesium 137 Detection Efficiency of Gamma Ray
Emitters Remote Detection of Hot-Box Radiation
(SURF) Sonoluminescent Light Intensity (SURF)
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy Dental Polysac
Adhesion Bio Knee Cartilage Regeneration Ceramic
Machining Strength Combinatorial Chemistry Tape
Peel
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp. DAC
(Digital-to-Analog Converter) Calibration OLES
Firefighter Infrared Imaging Devices OLES Metal
Detector Acceptance Sampling
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST) Tall Building Deflection
Safety Codes HHS CONTAM Home Pollution
Dissemination Solar Sphere Testing of Polymeric
Sealants
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling SMS
Smart Machining System OLES Forensic Imaging of
Gun Casings
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration Accelerated
Testing of Compact Discs Apache/Linux Web
Processing Time FEA NanoCantilever Sensitivity
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error Microarray Sensors for Toxic Gas
DHS Bio-Agent Detection Radiocarbon C14
Albuquerque CO Pollution
21
Physics Plutonium Troubleshooting
(SURF) Americium 241/243 Peal Deconvolution Alg.
Acc. Cesium 137 Detection Efficiency of Gamma Ray
Emitters Remote Detection of Hot-Box Radiation
(SURF) Sonoluminescent Light Intensity (SURF) ASP
(Adv. Spectroscopic Portal) Monitoring PRD
(Personal Radiation Detectors) Maritime Radiation
Detectors Soil Leeching Sequential Extraction
Protocol
NIST
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Microscopy Dental Polysac
Adhesion Bio Knee Cartilage Regeneration Ceramic
Machining Strength Combinatorial Chemistry Tape
Peel
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp. DAC
(Digital-to-Analog Converter) Calibration OLES
Firefighter Infrared Imaging Devices OLES Metal
Detector Acceptance Sampling
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST) Tall Building Deflection
Safety Codes HHS CONTAM Home Pollution
Dissemination Solar Sphere Testing of Polymeric
Sealants
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling SMS
Smart Machining System NIJ/OLES Forensic Imaging
of Gun Casings
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error Microarray Sensors for Toxic Gas
Detection DHS Bio-Agent Detection Radiocarbon
C14 Albuquerque CO Pollution (Cu-AU) 3D Nanoscale
Chemical Imaging Dual Rotor Turbin Fluid Flow SO2
Permeation Tube Mass Loss KC (Key Comparison)
Fluid Flow
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration Accelerated
Testing of Compact Discs Apache/Linux Web
Processing Time FEA NanoCantilever Sensitivity
22
Chemistry Carbon Nanotube Water Pollution SRM
2396 DNA Base Biomarkers Gate Dialectrics SiO2
HRTEM Error Microarray Sensors for Toxic Gas DHS
Bio-Agent Detection Radiocarbon C14 Albuq. CO
Pollut. (Cu-AU) 3D Nanoscale Chem. Imaging Dual
Rotor Turbin Fluid Flow SO2 Permeation Tube Mass
Loss KC (Key Comparison) Fluid Flow
Physics Plutonium Troubleshooting (SURF) Am
241/243 Peal Deconv. Alg. Acc. Cesium 137
Detection Efficiency of Gamma Ray Emitters Remote
Radiation Detection (SURF) Sonoluminescent Light
Intens.(SURF) ASP (Adv. Spectrosc. Portal)
Monitor. PRD (Personal Radiation
Detectors) Maritime Radiation Detectors Soil
Leeching Seq. Extraction Prot.
NIST
Material Science MALDI TOF Spectrometry Nanocantil
ever Atomic Force Mic. Dental Polysac
Adhesion Bio Knee Cartilage Regeneration Ceramic
Machining Strength Comb. Chemistry Tape Peel
Elect. Elect. Eng. OLES Bullet Proof Vest
Reliability Eddy Current Probe IACP/OLES
Safety/Speed Devices Acceptance Samp. DAC
(Digital-to-Analog Converter) Calibration OLES
Firefighter Infrared Imaging Devices OLES Metal
Detector Acceptance Sampling
Build. Fire Res. World Trade Center FEA Core
Damage Cigarette Ignition Propensity FHWA Highway
Concrete Strength (COST) Tall Building Deflection
Safety Codes HHS CONTAM Home Pollution
Dissemination Solar Sphere Testing of Polymeric
Sealants Optimization of Hot Plate Gap
Parameters Interlab Thermal Hot Plate
Conductivity Tomographic Flow Detection in
Polymer-Bonded Concrete HUD Lead Paint Test Kit
Accuracy HUD Lead Paint Extraction Hospital
Energy Consumption Evaluating Strategies for Fire
Safety Paint Peel Strength Aerosol Spray Flow
Rates Asphalt Roofing Vertical Peel
Testing Remote Detection of Pre-Mold Moisture in
Building Mats. WTC FDS (Fire Dynamics Simulator)
Sensitivity WTC FDS Validation WTC Impact
Sensitivity WTC FEA Insulation-on-Steel Thermal
Propagation WTC Structural Sensitivity
Manufacturing Eng. Scatterfield
Microscopy Genetic Alg. for Machine Tooling SMS
Smart Machining System NIJ/OLES Forensic Imaging
of Gun Casings
Information Tech. Abilene Network Loss
Rate Motion Imagery Quality Metrics RAVE
Visualization Facility Calibration Accelerated
Testing of Compact Discs Apache/Linux Web
Processing Time FEA NanoCantilever Sensitivity
23
Part 2 A Structured Approach ...
1. Generic Mapping / Model 2. Problem-Solving
Framework 3. Problem Classification 4. DEX
Worksheet 5. Orthogonal Fractional Designs 6.
DEX Analysis
24
1. Generic Mapping / Model
Y f(X1, X2, X3, ..., Xk)
k factors natures f
unknown (k,n) k factors, n runs (k,n,l) k
factors, n runs, l levels per factor A run y
f(x1, x2, x3, ..., xk)
25
1. Generic Mapping / Model
Output Y
Input Xi
y f(x1,x2,x3,...,xk)
f (nature)
X1 x1 X2 x2 X3 x3 X4 x4 X5 x5
Y y
k 5
26
2. Stat Problem-Solving Framework
Science Problem
Science Solution
27
2. Stat Problem-Solving Framework
Science Problem
Science Solution
Question ?
28
2. Stat Problem-Solving Framework
Science Problem
Science Solution
Question ?
Answer ?
29
2. Stat Problem-Solving Framework
Expert
Science Problem
Science Solution
Question ?
Answer ?
Data
30
2. Stat Problem-Solving Framework
Expert
Science Problem
Science Solution
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
Binary
31
2. Stat Problem-Solving Framework
Expert
Science Problem
Science Solution
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
32
2. Stat Problem-Solving Framework
Expert
Science Problem
Science Solution
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
GIGO
33
What is DEX?
Definition DEX a systematic, rigorous
approach for
engineering/scientific problem-solving Goal 1
To produce Crisp Unambiguous
Valid Defensible Supportable
Repeatable With Small Goal 2
conclusions
34
What is DEX?
Definition DEX a systematic, rigorous
approach for
engineering/scientific problem-solving Goal 1
To produce Crisp Unambiguous
Valid Defensible Supportable
Repeatable With Small Sample
Size n Goal 2
conclusions
35
What is DEX?
Definition DEX a systematic, rigorous
approach for
engineering/scientific problem-solving Goal 1
To produce Crisp Unambiguous
Valid Defensible Supportable
Repeatable With Small Sample
Size n Goal 2 Insight
conclusions
36
(No Transcript)
37
2. Stat Problem-Solving Framework
5 Steps ...
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
38
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n) Specificity
Every DEX has a k and n
39
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
Material Machine Method Measuring
Device Operator/Day Environment
Time (11 weeks)
40
2. Stat Problem-Solving Framework
DEX Substeps (4) ...
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification 2. Translation 3.
Construction 4. Execution
41
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification (4 Generic Questions) 2.
Translation 3. Construction 4. Execution

42
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification (4 Generic Questions) 2.
Translation (DEX Worksheet) 3. Construction
4. Execution
43
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification (4 Generic Questions) 2.
Translation (DEX Worksheet) 3. Construction
(Orthogonal Fractional Designs) 4. Execution
44
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification (4 Generic Questions) 2.
Translation (DEX Worksheet) 3. Construction
(Orthogonal Fractional Designs) 4. Execution
(Control, Randomize, ...)
45
2. Stat Problem-Solving Framework
Analysis ...
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification 2. Translation 3.
Construction 4. Execution
1. Estimation () 2. Testing (Y/N)
46
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
1. Classification 2. Translation 3.
Construction 4. Execution
1. Estimation 2. Testing 1. Quantitative 2.
Graphical
47
2. Stat Problem-Solving Framework
Scope ...
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data
(Pre-data)
(k,n)
Scope of Conclusions ?
48
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem (Population)
Science Solution (Population)
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data (Sample)
(Pre-data)
(k,n)
1. Local Summarization?
Scope
49
2. Stat Problem-Solving Framework
Expert
5
1
Science Problem (Population)
Science Solution (Population)
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data (Sample)
(Pre-data)
(k,n)
1. Local Summarization? 2.Global Inference?
Inference? generalizability predictability

repeatability, robustness (Science!)
Scope
50
2. Stat Problem-Solving Framework
Scope (k vs n) Tradeoff ...
Expert
5
1
Science Problem (Population)
Science Solution (Population)
2
4
3
Analysis
DEX
Question ?
Answer ?
(Post-data)
Data (Sample)
(Pre-data)
(k,n)
1. Local Summarization? 2.Global Inference?
Inference? generalizability predictability

repeatability, robustness (Science!)

? k large vs. n small tradeoff (scope vs.
budget)
Scope
51
2. Stat Problem-Solving Framework
TKS ...
Expert
5
1
Science Problem (Population)
Science Solution (Population)
2
4
3
Stat(G,Q)
DEX
(Post-data)
Data (Sample)
(Pre-data)
1. Characterizing 2. Sensitivity 3. Optimizing 4.
Modeling 5. Comparing 6. Predicting 7.
Uncertainty 8. Verifying 9. Validating
1. , Distribution 2. List Ranked Factors 3.
Vector (x1,,xk) 4. f 5. Y/N 6 7. SD() 8.
Y/N, Vector (x1, ,xk) 9. Y/N, Vector (x1, ,xk)
Estimation Testing
Reality Lab Computational
1-FAT Monte Carlo Latin HC Orthogonal Resp
Surface
52
3. Problem Classification
Y f(X1, X2, X3, , Xk)
53
3. Problem Classification ( Q)
Is this Factor Significant?
Y f(X1, X2, X3, , Xk)
54
3. Problem Classification
Most Important Factors?
Is this Factor Significant?
Y f(X1, X2, X3, , Xk)
55
3. Problem Classification
Most Important Factors?
Is this Factor Significant?
Y f(X1, X2, X3, , Xk)
Good Approximating Function?
56
3. Problem Classification
Most Important Factors?
Is this Factor Significant?
Y f(X1, X2, X3, , Xk)
Best Settings of the k Factors?
Good Approximating Function?
57
3. Problem Classification
Most Important Factors? (2. Screening/Sensitivity)
Is this Factor Significant? (1. Comparative
Robust Inference)
Y f(X1, X2, X3, , Xk)
Best Settings of the k Factors? (4. Optimization)
Good Approximating Function? (3. Regression)
58
3. Problem Classification
Comparative
Screening/Sensitivity
Regression
Optimization
59
3. Problem Classification
Comparative Focus 1 primary factor Q1. Does that
factor have an effect (Y/N)? Q2. If yes, then
best setting for that that factor ?
(vector) Constraint Want conclusions to be
robust over all other
factors Designs CRD, RBD, LSqD, TPD BHH, Ch. 4
Screening/Sensitivity
Regression
Optimization
60
3. Problem Classification
Comparative Focus 1 primary factor Q1. Does that
factor have an effect (Y/N)? Q2. If yes, then
best setting for that that factor ?
(vector) Constraint Want conclusions to be
robust over all other
factors Designs CRD, RBD, LSqD,TPD BHH, Ch. 4
Screening/Sensitivity Focus all factors Q1. Most
important factors (ranked list) Q2. Best settings
(vector) Insight! Q3. Good model
(function) Designs 2kD, 2k-pD, TD BHH, Ch. 5-6
Regression
Optimization
61
3. Problem Classification
Comparative Focus 1 primary factor Q1. Does that
factor have an effect (Y/N)? Q2. If yes, then
best setting for that that factor ?
(vector) Constraint Want conclusions to be
robust over all other
factors Designs CRD, RBD, LSqD,TPD BHH, Ch. 4
Screening/Sensitivity Focus all factors Q1. Most
important factors (ranked list) Q2. Best settings
(vector) Q3. Good model (function) Designs 2kD,
2k-pD,TD BHH, Ch. 5-6
Regression Focus all factors Q1. Good model
(function) Continuous factors Designs BBD,
XOD BHH, Ch. 10-11
Optimization
62
3. Problem Classification
Comparative Focus 1 primary factor Q1. Does that
factor have an effect (Y/N)? Q2. If yes, then
best setting for that that factor ?
(vector) Constraint Want conclusions to be
robust over all other
factors Designs CRD, RBD, LSqD,TPD BHH, Ch. 4
Screening/Sensitivity Focus all factors Q1. Most
important factors (ranked list) Q2. Best settings
(vector) Q3. Good model (function) Designs 2kD,
2k-pD,TD BHH, Ch. 5-6
Regression Focus all factors Q1. Good model
(function) Continuous factors Designs
BBD,XOD BHH, Ch. 10-11
Optimization Focus all factors Q1. Best settings
(vector) Continuous factors Designs RSD, CD,
BBD BHH, Ch. 12
63
3. Problem Classification
Comparative Focus 1 primary factor Q1. Does that
factor have an effect (Y/N)? Q2. If yes, then
best setting for that that factor ?
(vector) Constraint Want conclusions to be
robust over all other
factors Designs CRD, RBD, LSqD, TPD BHH, Ch. 4
Screening/Sensitivity Focus all factors Q1. Most
important factors (ranked list) Q2. Best settings
(vector) Insight! Q3. Good model
(function) Designs 2kD, 2k-pD, TD BHH, Ch. 5-6
Most Popular
Regression Focus all factors Q1. Good model
(function) Continuous factors Designs BBD,
XOD BHH, Ch. 10-11
Optimization Focus all factors Q1. Best settings
(vector) Continuous factors Designs RSD, CD,
BBD BHH, Ch. 12
64
3. Problem Classification
Critical Well-studied optimal experiment
designs already exist. The
problem classification (along with
k and n) dictates the choice of
the specific design. Comparative/Robust CRD,
RBD, LSD, TPD Screening/Sensitivity 2kD, 2k-pD,
TD Regression BBD,
XOD Optimization RSD, CD, BBD
65
4. Getting Organized DEX Worksheet
66
(No Transcript)
67
1. Catapults
Y Total Distance (cm)
X1 Ball Type X2 Rubber Band Type
X3 Arm Cup X4 Arm Connect Position
X5 Pull Back Angle X6 Stopper Pin Position X7
Fulcrum Position
(k , n )
68
2. Funnels
Y Total Traversal Time (sec)
X1 Ball Size X2 Funnel Type
X3 Ramp Type X4 Vertical Angle X5
Horizontal Angle X6 Riser Bar Height X7 Funnel
Stability
(k , n )
69
3. World Trade Center
Y Core Damage
1. Flight Speed 2. Flight Impact Location
(Vertical) 3. Flight Impact Location
(Horizontal) 4. Engine Assignment Set 5.
Engine Strength 6. Engine Failure Strain 7.
Engine Strain Rate Effects 8. Perimeter Column
Strength 9. Perimeter Column Failure Strain
10. Perimeter Column Strain Rate Effects 11. FEA
Model Erosion Parameter 12. FEA Contact
Parameter 13. FEA Friction Coefficient
(k , n )
70
4. Carbon Nanotube Contamination
Y Carbon Nanotube Reduction ()
X1 Clay Concentration X2 NOM
Concentration X3 Bio
Concentration X4 CNT Concentration X5
Coagulant Type
(k , n )
71
5. Chemical Yield (Benchmark)
Y Chemical Yield ()
X1 Feed Rate (l/min) X2 Catalyst () X3
Agitation Rate (rpm) X4 Temperature (C) X5
Concentration ()
(k , n )
72
5. Getting Superb Estimates/Results Orthogonal
Designs
Superb 1. Unbiased 2.
Small Variation Key For a problem with a given
(k,n), not all experiment designs are
equally good
73
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Question ?
Answer ?
Data
74
Designs in Science
1. Most common design in science 1FAT 2. Worst
design in science 1FAT 3. Best
design in science Orthogonal

75
1-Factor-at-a-Time Design
1. Execute a baseline run, then 2. Execute k
successive runs--each time changing 1 factor
from baseline. 3. Thus total n 1k
1-FAT Design (k5,n6) X1 X2 X3 X4 X5
- - - - - - - - - - -
- - - - - - - - - -
- - - -
76
Orthogonal Design
1. For each and every one of the k factors
Every level within a factor occurs the same
times 2. For each and every pair of the k
factors Every pair of levels between
factors occurs the same of times Claim
Estimates from an orthogonal experiment design
are far superior (smaller bias and
smaller variability) than estimates
from (the common) 1-factor at a time
design--even for identical sample size n!
77
Benchmark Problem (k 5) 4 Designs
Orthogonal?
25Orthogonal Design (n 32) X1 X2 X3 X4 X5
X1 X2 X3 X4 X5 - - - - - - - - -
- - - - - - - - - -
- - - - - - - -
- - - - - - - - -
- - - - - - - -
- - - - -
- - - - - - - - -
- - - - - - -
- - - - - -
- - - - -
- - -
-
25-1Orthogonal Design (n 16) X1 X2 X3
X4 X5 - - - - - - - -
- - - - - -
- - - - - -
- - - - - -
- - - - - -
- - - -
- - - -

1-FAT Design (n 6) X1 X2 X3 X4 X5
- - - - - - - - - - -
- - - - - - - - - -
- - - -
25-2Orthogonal Design (n 8) X1 X2 X3 X4
X5 - - - - - - -
- - - - - -
- - - - -
- -
78
Orthogonal vs. 1-FAT Designs Chemical Reactor
(k5) Benchmark Problem
X1 Feed Rate (l/min) 10 and 15 X2
Catalyst () 1 and 2 X3
Agitation Rate (rpm) 100 and 120 X4
Temperature (C) 140 and 180 X5
Concentration () 3 and 6
Q. Most important factors ?
79
5 Factor Model (Truth)
80
Benchmark Problem (k 5) Truth
Q. Most important factors ?
81
Benchmark Problem (k 5) Truth
Q. Most important factors ?
82
Benchmark Problem (k 5) Truth Ranked List
Q. Most important factors ?
Sensitivity Problem Ranked List (Truth)
1. X2 19.5 2. X2X4
13.25 3. X4X5 -11.0 4. X4
10.75 5. X5 -6.25 6.

83
Benchmark Problem (k 5) Truth
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Q Most important factors?
Answer List
Data
1. X2 19.5 2. X2X4 13.25 3. X4X5 -11.0 4.
X4 10.75 5. X5 -6.25 6.
(k5,n32/16/8/6)
25 25-1 25-2 1-FAT
84
Benchmark Problem (k 5) 4 Designs
Designs (4) ...
25Orthogonal Design (n 32) X1 X2 X3 X4 X5
X1 X2 X3 X4 X5 - - - - - - - - -
- - - - - - - - - -
- - - - - - - -
- - - - - - - - -
- - - - - - - -
- - - - -
- - - - - - - - -
- - - - - - -
- - - - - -
- - - - -
- - -
-
25-1Orthogonal Design (n 16) X1 X2 X3
X4 X5 - - - - - - - -
- - - - - -
- - - - - -
- - - - - -
- - - - - -
- - - -
- - - -

1-FAT Design (n 6) X1 X2 X3 X4 X5
- - - - - - - - - - -
- - - - - - - - - -
- - - -
25-2Orthogonal Design (n 8) X1 X2 X3 X4
X5 - - - - - - -
- - - - - -
- - - - -
- -
85
Benchmark Problem (k 5) 4 Designs Data
Data ...
25Orthogonal Design (n 32) X1 X2 X3 X4 X5
Y X1 X2 X3 X4 X5 Y - - - - - 61 -
- - - 56 - - - - 53 - - -
63 - - - - 63 - - - 70
- - - 61 - - 65 - - -
- 53 - - - 59 - - - 56
- - 55 - - - 54 -
- 67 - - 61 -
65 - - - - 69 - - - 44 -
- - 61 - - 45 - - -
94 - - 78 - - 93
- 77 - - - 66 - -
49 - - 60 - 42 -
- 95 - 81
- 98 82
25-1Orthogonal Design (n 16) X1 X2 X3
X4 X5 Y - - - - 56 -
- - - 53 - - - - 63
- - 65 - - - - 53
- - 55 - - 67
- - 61 - - - - 69
- - 45 - - 78
- - 93 - -
49 - - 60 -
- 95 82
1-FAT Design (n 6) X1 X2 X3 X4 X5 Y
- - - - - 61 - - - - 53
- - - - 63 - - - - 53
- - - - 69 - - - - 56
25-2Orthogonal Design (n 8) X1 X2 X3 X4
X5 Y - - - 44 - - -
- 53 - - - 70 -
- 93 - - - 66 -
- 55 - - - 54
82
86
Benchmark Problem (k 5) Truth
Analysis ...
Expert
5
1
Science Problem
Science Solution
2
4
3
Analysis
DEX
Q Most important factors?
Answer List
Data
1. X2 19.5 2. X2X4 13.25 3. X4X5 -11.0 4.
X4 10.75 5. X5 -6.25 6.
(k5,n32/16/8/6)
25 25-1 25-2 1-FAT
Q Least Squares Regression Gr 10-Step (Stat
e-Handbook)
87
Benchmark Problem (k 5) for Comparing/Evaluating
Designs
Conclusions ...
(k 5, n 32/16/8/6)
32.
16.
8.
6.
88
Benchmark Problem (k 5) for Comparing/Evaluating
Designs
(k 5, n 32/16/8/6)
32.
16.
8.
6.
89
Benchmark Problem (k 5) 4 Designs
Design Geometry ...
25Orthogonal Design (n 32)
25-1 Orthogonal Design (n 16)


X5
X5
X5
X5


X2
X2
X2
_

X2
_

X3
X3
X3
X3
_
_
_
X1
_
X1
_
_
X1

X1

_

_
X4

X4
X4
X4
25-2 Orthogonal Design (n 8)
1-FAT Design (n 6)


X5
X5
X5
X5


X2
X2
X2
_
X2
_


X3
X3
X3
_
_
_
_
X1
X1
_
_
X1
X1


_
_


X4
X4
X4
X4
90
1-FAT Versus Orthogonal Designs (k 5, n 6,8)
Design Matrices ...
1-FAT Design X1 X2 X3 X4 X5 -
- - - - - - -
- - - - - -
- - - - - - -
- - - -
25-2Orthogonal Design X1 X2 X3 X4 X5
- - - - -
- - - - -
- - - -
- - - -
- -
91
1-FAT Versus Orthogonal Designs (k 5, n 6,8)
Design Geometry ...
25-2Orthogonal Design
1-FAT Design

X5
X5

X2
X2
_

X3
X3
_
_
_
X1

X1
_

X4
X4
92
1-FAT Versus Orthogonal Designs (k 7, n 8)
1-FAT Designs X1 X2 X3 X4 X5 X6 X7
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -
- - - - - - -

27-4 Orthogonal Design X1 X2 X3 X4 X5
X6 X7 - - -
- - - - -
- - - -
- - - - - -
- - -
- - - - -
- -

93
1-FAT Versus Orthogonal Designs (k 7, n 8)
1-FAT Design
27-4Orthogonal Design





X5
X5
X5
X5




X2
_
X2
_
X2
_

X2
_



X3
X3
X3
X3
_
_
_
_
_
_
_
_
X1

X1
X1

X1


_

_
_
_
X4



X4
X4
X4
X7





X5
-
X5
X5
X5
X5




-
X2
_
X2
_
-
X2
_
X2
_




X3
X3
X3
X3
_
_
_
_
_
_
-
_
_

_
_
X1
X1


X1

X1

_
_


_
_
X4
X4


X4
X4
X4
-

X6
94
1-FAT Versus Orthogonal Designs (k 3, n 4)
23-1 Orthogonal Design X1 X2 X3
- - - -
- -
1-FAT Design X1 X2 X3 - -
- - - -
- - -
Advantages of Orthogonal Designs Fair/Balanced
for any/every single factor Fair/Balanced for
any/every pair of factors Increased sensitivity
for t test Unbiased estimates of main
effects Estimate interactions (if possible)
95
1-FAT Versus Orthogonal Designs (k 3, n 4)
23-1 Orthogonal Design
1-FAT Design


X2
X2
X2

X5
X3
X3
X3
_
_
X1
X1
X1

_
_
X4
96
1-FAT Versus Orthogonal Designs (k 3, n 4)
1-FAT Design
23-1Orthogonal Design
97
Orthogonal Designs (n 8 runs)
98
Orthogonal Design Conclusions
1. Designs makes a difference in terms of the
quality of estimates and validity of
conclusions 2. Orthogonal Designs are
excellent 3. If the number of runs n is an issue
(due to budget or time constraints), then
orthogonal fractional factorial designs are
excellent 4. 2-level orthogonal fractional
factorial designs are remarkably insightful
and extremely n-efficient
99
6. Sample Size n Considerations
Affordability (time )
100
6. Sample Size n Considerations
Simplest case all factors having l 2 levels

101
3 Ways to Reduce Sample Size n
Y f(X1, X2, X3, ..., Xk) l1 l2
l3 ... lk
Default Full Factorial Design n l1 l2 l3
... lk If a Full factorial
Design is too expensive
1. Reduce the number of factors k 2. Reduce the
number of levels li 3. Fractional Factorial
Designs
102
7. DEX Analysis
Analysis is less important than DEX itself, but
(as expected) some analysis methods are better
(more insightful) than others. Our
recommendation for analyzing 2-level orthogonal
(full and fractional) designs is a 10-step
graphical procedure (e-Handbook)
103
10 Step Graphical Analysis of 2-Level Designs
(Dataplot)
104
8. DEX, Data, and Analysis Revisiting WTC
105
WTC Impact Core Damage Assessment Q. After the
plane impact of the WTC South Tower, there was no
recorded data as to how many of the interior 47
columns of the building were damaged. A
finite-element analysis (FEA) program was written
to simulate the impact. The plane was modeled by
1.4 million elements. What factors most affected
the performance of this FEA code? What factors
could be eliminated as unimportant?
Q. What factors affect quality of FEA code
predictions?
106
Factors ...
2. Sensitivity Analysis Experiment Design List
of Factors (Component Engine)
DEX g(k,n)
(k 13, n lt 50)
(Design and data based on research carried out by
contractor Applied Research Associates)
Y Core Columns Damaged
107
Design ...
Experiment Design 1-FAT
k 13, n 113 14
108
Design ...
Experiment Design 1-FAT
k 13, n 113 14
109
Design ...
Experiment Design 213-9 Orthogonal Fractional
Factorial wcp
(k 13, n 17)
Y .175
BHH, p. 410, p. 272
110
Design ...
Experiment Design 213-9 Orthogonal Fractional
Factorial wcp
(k 13, n 17)
Y .175
111
Design ...
Experiment Design 213-9 Orthogonal Fractional
Factorial wcp
(k 13, n 16)

112
Data ...
(k 13, n 17)
Y .175
113
Analysis ...
10 Step Graphical Analysis of 2-Level Designs
(Dataplot)
DEXPLOT.DP
114
 
Analysis ...
5. Data Analysis (Graphical) Most Important
Factor, Best Setting
Ordered Data Plot
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
115
 
Analysis ...
5. Data Analysis Most Important Factor, Best
Setting
Ordered Data Plot
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
116
 
Analysis ...
5. Data Analysis Estimation of Factor Effects
Main Effects Plot
Halfnormal Probability Plot of Effects
.175
Least Squares Estimates
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
117
 
Analysis ...
5. Data Analysis Estimation of Factor Effects
Main Effects Plot
Halfnormal Probability Plot of Effects
.175
Least Squares Estimates
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
118
 
Analysis ...
5. Data Analysis Estimation of Factor Effects
Main Effects Plot
Halfnormal Probability Plot of Effects
.175
Least Squares Estimates
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
119
 
Analysis ...
5. Data Analysis Best Settings
Main Effects Plot
.175
- - - - - . - - . .
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
120
 
Analysis ...
5. Data Analysis Confounding
Interaction Effects Matrix
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
121
 
Analysis ...
5. Data Analysis Confounding
Interaction Effects Matrix
1 13 78
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
122
 
Analysis ...
6. Conclusions
  X10 -.21 (-120) (Perimeter Column Strain
Rate Effects) X3 -.10 (-57) (Impact
Location Horizontal) X9 -.08 (-46)
(Perimeter Column Failure Strain) X5 .07
(40) (Engine Strength) X2 -.04 (-23)
(Impact Location Vertical)   with least
important factors being   X13 .00 (0) (FEA
Friction Coefficient) X11 .00 (0) (FEA
Erosion Parameter) X8 .00 (0) (Perimeter
Column Strength)  
Additional ARA Runs LHC gt f
FEA for plane 1.4 million elements
  "Figure" 1.4 Data from 213-9 (with center
point) orthogonal experiment design for
engine/core-column impact study
123
Conclusions
1. Approach A structured problem-solving
approach exists, with generic and relevant
questions, issues, methodologies 2. Design
Design is more important than analysis 3. (k,n)
Every design has a (k,n) (specificity) 4. Problem
Categories Scientific problems often generically
fall into 4 categories--these categories have
corresponding designs 5. Designs Conclusions
Designs makes a difference in terms of the
quality of estimates and validity of
conclusions 6. Orthogonal 1FAT designs are poor
orthogonal designs are excellent 7. Fractional
If the number of runs n is an issue then
orthogonal fractional factorial designs are
excellent 8. 2k-p 2-level orthogonal fractional
factorial designs are remarkably insightful
and extremely n-efficient
124
References
1. Experiment Design Textbook Box,
Hunter, Hunter Statistics for Experimenters,
Wiley, (esp. p. 410 for edition 1 and p.
272 for edition 2) 2. Experiment Design
Methodology NIST/SEMATEC e-Handbook of
Statistics, Chapter 5 Improve
http//www.itl.nist.gov/div898/handbook/
http//www.itl.nist.gov/div898/handbook/pri/sectio
n5/pri59.htm 3. 10-Step Graphical Analysis of
2k-p Designs Dataplot macro DEXPLOT.DP
http//www.itl.nist.gov/div898/software/dataplo
t.html
125
NIST e-Handbook of Engineering Statistics
http//www.itl.nist.gov/div898/handbook/
5. Improve 5. Advanced Topics 9. An
EDA Approach to DEX
(3000 pages 3 million page views / month)
126
NIST Dataplot (Filliben/Heckert)
DEXPLOT.DP (10-Step)
http//www.itl.nist.gov/div898/software/dataplot.h
tml/
127
James J. Filliben Ext. 2855
james.filliben_at_nist.gov