Research Methodology

1
Research Methodology

• EPH 7112
• LECTURE 7 EXPERIMENTAL DESIGN

2
Contents

• Synthesis
• Implement Solution
• Design Experiments
• Conduct Experiments
• Reduce Results

3
Scientific Method
4
Analysis
5
Hypothesis

• Specify detail and comprehensive solution
• Assert expected results
• Define factors that will be varied
• Measure against performance metrics

6
Synthesis

• Implement the solution
• And experiment
• To accomplish the goals
• And validate the hypotheses

7
Synthesis Steps

• Implement Solution
• Design Experiments
• Conduct Experiments
• Reduce Results

8
Implement Solution

• Implement solution to test hypotheses
• Methods
• Acquire
• Construct
• Combination of both

9
Implement Solution

• Construct
• Custom made to meet requirements
• Time consuming
• Expensive

10
Implement Solution

• Acquire
• Quick solution
• Cheaper
• May not meet requirements

11
Implement Solution

• Consider strongly to acquire the solution
• Even if part of entire solution
• Consultants acquired solution?

12
Implement Solution

• Example Optical Amplification in S-band
• Construct using Thulium doped fiber
• Problem Fusion Splicing is not possible
• Solution Use Angled Connectors
• Issue Not specified during order

13
Implement Solution

• Is your solution right or is it the right
  solution?
• Careful implementation
• Step-by-step
• Troubleshooting
• Example Constructing an optical amplifier
• Problem WDM is faulty

14
Design Experiments

• To design a series of experiments
• Results used to estimate how good solution to
  solve problem
• An experiment acquires data to measure the
  performance of the solution under controlled
  conditions in a laboratory

15
Design Experiments

• Always good to have a check list
• Objective
• Unit under Test
• Inducers
• Sensors
• Supervisor
• Channels
• Domain Knowledge
• Range Knowledge
• Solution

16
Experiments

• Is it really necessary?
• How about theoretical or simulation work?
• Experiment verification
• Example Find solution of two-dimensional plane
  that satisfy certain conditions

17
Experiments

• Simulation and modeling
• Verify against experimental results
• Example Modeling of Optical Amplifier
• Advantages of modeling
• Optimization
• Analyzing the physical phenomena

18
Design Experiments

• Planning
• Specification Experiment Laboratory
• Design of experiment blocks
• Design of protocols
• Acquiring and managing data

19
Experiment Laboratory

• Laboratory is where the experiment takes place
• Large room with test measurement equipments,
  units under test, chemical mechanical
  apparatus, computers

20
Laboratory

• Experiments can also take place
• In an office
• Field
• Manufacturing Plant

21
Laboratory

• Depends on the experiment
• Objective
• Sample
• Unit under Test
• Inducers
• Sensors
• Solution

22
Laboratory Safety

• Watch out for moving or revolving parts (they
  dont like necklaces and neck ties!)
• Watch out for Electro-Static Sensitive Devices
• Limit personnel into the laboratory
• Maintenance and cleaning personnel can cause
  mishaps

23
Design Experiment Block

• Process that takes place in the laboratory during
  the experiment
• Important terms
• Sample
• Factors (independent variables)
• Inducers
• Sensors

24
Sample

• Task unit consisting of objects, living plants,
  animals, humans that is the subject in the
  experiment

25
Factors

• Condition or parameter of a task whose value is
  intentionally varied to measure its impact on the
  results of the task

26
Inducers

• A device or mechanism that alters the task unit/
  subject during the experiment

27
Sensors

• Device that capture the results from the task/
  unit or subject

28
Extent

• Each factor is assigned a set of values
• Extent of the factor space is total number of
  unique combination of values that may be assigned
  to each factor
• FVa number of values for factor a
• Extent FVa x FVb x FVc

29
Treatment

• Each one of the unique combination of values that
  may be assigned to every factor is called a
  treatment
• One instance in the entire factor space

30
Case Study

• Factors
• Typeface 2 types (Serif and Sans Serif)
• Noise Level 12 levels
• Character 36
• Extent 36 x 12 x 2 864
• Each combination treatment

31
Block Design

• If sample is a single object or device, then all
  the possible treatment must be assigned to it
  during the experiment
• Example Characterization of a Thulium doped
  Fiber Amplifier for different pump powers and
  wavelengths

32
Block Design

• If sample is more than one, then the treatments
  may be distributed in some way among the sample
• Important terms
• Experiment trial
• Experiment block

33
Experiment Trial

• Complete set of treatments applied to a sample
  during the experiment (sample is more than one)
• Example The combination of typeface serif,
  character ltA, B, Cgt and noise level lt130, 140,
  150gt

34
Experiment Block

• Set of experiment trials that provides a cover of
  the factor space that is appropriate and adequate
  for achieving the task objective

35
Block Design

• What is the appropriate set of experiment trials
• that provides an appropriate cover of the factor
  space
• for the experiment?

36
Block Design

• Three basic strategies
• Enumerated block design
• Systematic block design
• Randomized block design

37
Enumerated Block Design

• Assigns every possible treatment to every sample
• Obvious strategy if sample 1
• If sample gt 1, this is not practical
• Because total number of trials extent of factor
  space x number of sample
• Too large !!

38
Systematic Block Design

• Uses a deterministic algorithm to assign
  treatments to different sample in a systematic
  way
• Eventually covers the entire factor space
• Problem Unintentional resonance between sample
  and treatment can be sparked

39
Systematic Block Design

• Example
• A marketing survey is carried out to every 100th
  telephone number
• The chances a treatment assigned to, say a number
  03-2698 1100 belonging to a business entity
• Is higher than say 03-2698 1024
• A bias towards response of business entity may
  occur in the survey

40
Systematic Block Design

• This block design should be avoided
• Unless this bias can be ascertained

41
Randomized Block Design

• Similar to systematic block design
• Except that the treatment assigned to the sample
  are sequenced randomly
• This can also reduce the risk of systematic bias

42
Case Study

• Enumerated block design is not practical
• Total trials 864 x 14!
• Each sample has to respond to 864 treatments!
• Fatigue
• Peak Performance

43
Case Study

• Another disadvantage
• Humans are smart
• Easily guess that factor space include 10 decimal
  digits and 26 Latin characters
• Guess from elimination process
• Bias the results

44
Case Study

• Since all the license plate inspectors had same
  recognition skills
• Not all treatment need to be assigned to every
  sample/ subject
• Divide 864 treatments equally
• Reduce time for each subject
• Can a systematic block design do it?

45
Case Study

• Systematic block design also has setbacks
• Subjects can also detect the periodicity
• Biased improved performance
• Randomized block design is solution
• Computer generated pseudo-random assignment of
  treatments

46
Case Study

• Decide how many and which sets of treatments
  would be randomly assigned to subjects
• Combined to cover enough sets for each factor
• To make up set of trials that cover entire factor
  space

47
Representation Factor

• A factor that is not intended as a basis for
  measuring performance
• However they are necessary for assigning values
  of a parameter
• Example
• Characters

48
Performance factor

• A factor that is used as a basis for measuring
  performance
• Example
• Typeface
• Noise Level

49
Case Study

• Either assign each of two typefaces to half the
  subjects
• Or assign both typefaces to all
• Former solution better to avoid confusion among
  subject and more realistic

50
Case Study

• Noise Level range 130 to 240 with increments of
  10
• How to distribute the treatment to subjects?
• Condition
• Interval must be same
• Subsets different
• Same average

51
Case Study

• Subsets suitable
• 130, 150, , 210, 230
• 140, 160, , 220, 240
• 130, 140, , , 230, 240

52
Case Study

• How many subjects?
• Access to 14 trained license plate inspectors
• 2 for OP Pilot
• Number of subjects will determine the combination
  of performance factor values

53
Case Study

• Statisticians require at least 30 responses over
  entire experiment for each typeface and noise
  level combination
• Characters can be assigned to obtain response
• Must use whole set (36 characters) or multiple of
  whole set to avoid character bias

54
Case Study

• Decided
• 6 subjects for OP Pilot
• 8 subjects for experiment trial
• Using block design in column B

55
Control Trial

• Measures the performance of one set of task in
  the absence of another to isolate the effects of
  the included components on performance

56
Control Trial

• To identify possible bias in the processes of the
  project task
• Bias is a consistent tendency to behave in an
  inconsistent way under certain conditions
• Example
• A spring loses its memory when elastic limit is
  exceeded

57
Control Trial

• Case Study
• The Listening Rat
• Disabled Power Brakes

58
Control Trial

• To establish performance baselines for comparison
• Without baseline, it is impossible to test the
  hypothesis of a solution that suggests a certain
  improvement or behavior
• Example
• To test if a new hand lotion is better than not
  using any hand lotion

59
Case Study

• Two control task was introduced
• With characters but without noise
• Without characters, only noise

60
Case Study

• The first control trial to ensure that each
  subject had sufficient experience with interface
  during practice sessions
• So that the effects of the learning curve is
  negligible during test trials
• If learning bias occurs, repeat practice session

61
Case Study

• Second control trial to measure selection time in
  the absence of any characters
• Pure guessing
• The statistics of this selection time used as
  baseline for computing the confidence that
  subject did not purely guess in the test trials

62
Protocols

• Step-by-step procedure to be followed during
  preparation and conduct of experiment
• Main purpose
• To ensure that experiment can be accurately and
  precisely repeated

63
Protocols

• Check list can help ensure uniformity in
  preparation of lab before experiment trial begins
• Everyone involved must carry out protocol
  accordingly
• Pilot trials can be used to plan and debug the
  protocol

64
Protocols

• Anyone in contact with human subjects in an
  experiment trial should not expose the objective
  of the experiment
• Protocols can be printed as flowcharts,
  pseudo-code or lists

65
Case Study

• Characterization of EDFA
• Steps include
• Measure input signal
• Measure output signal
• OSA will compute Gain, NF, PASE
• Sequence of measurement is important

66
Data Management

• Most critical and frustrating task
• Protect data
• Maintain logs of data (where it is kept, which
  file is for what)
• Record ALL experiment data

67
Data Management

• Do not preprocess the raw experiment data in any
  way before recording it
• Establish clear organizational and documentation
  conventions for data files
• Back-up !!

68
Conduct Experiments

• Time to follow your plans
• Resist temptation to improvise on the fly
• If doesnt run well, stop and revise
• Consider failed experiment as pilot trial

69
Reduce Results

• Performance values to validate the hypothesis
  cannot be drawn directly from raw results
• The raw results must be reduced, combined or
  transformed to be meaningful

70
Case Study 1

• Identify Faster Traffic on Highway
• Need to measure speed
• This is not raw results
• Reduced from compression signals, time two pulses
  occurred

71
Case Study 2

• Characterization of EDFA
• Need to measure Gain (dB) and Noise Figure (dB)
• Not raw results
• Reduced from among others Input Signal Power
  (dBm), Output Signal Power (dBm), ASE Level (dBm)

72
Reduce Results

• Data Reduction methods may change
• Performance metrics may be altered
• Important to record both raw and reduced results

73
QA