Selecting Performance Measures : Objective versus Subjective Measurement

1
Selecting Performance Measures Objective
versus Subjective Measurement

• Frederick A. Muckler Sally A. Seven

2
truth external to human judgment
science
ideal
Objectivity
Subjectivity
versus
science
sin
threatens the integrity of science
3
people are low fidelity instruments.
Webb, Campbell, Schwartz Sechrest. (1966)
21 possible sources of error attributable to
human beings
Campbell (1958)
4
external report
internal (self) report
observer
Abhorred by advocates of objective measurement
5
Selecting Measures
More advanced sciences can use predetermined
textbook measurement.
Kuhn (1961)
in HF no agreed on set of measures.
measurements
generated by investigator
6
Collecting Data
Rosenthal (1976)2 kinds of contaminants
experimenter
observation
expectation
(demand characteristics)
participant
7
Avoiding Contamination

• Friedman (1967)
• Careful selection of data collectors
• Video tape sessions
• Designs that can be evaluated post hoc for
  confounds
• Thornton Zorich (1980)
• 4. Train data collectors

8
Analyzing Data
Statistics pure mechanical objectivity

• But there is always some subjectivity
• Alpha levels
• Choice of tests
• Statistical manipulations can reflect the nature
  of the manipulations rather than the nature of
  the data.

9
Interpreting Data
The same data can be interpreted in different
ways
data
policy 1
policy 2
10
Objectivity in Practice

• Consensus of objective measurements
• External expert still preferred as objective
  over internal (subjective) self-report.

observer
object
subjective
objective
observer
11
Subjectivity in Practice
Greenflo Crano (1989)
Values (subjective)
Facts (objective)
comparewith dissimilar
comparewith similar
still problems with objectivity example sleep
studiesself-report vs. brain waves
12
Abstract Concepts
When dealing with abstract concepts, subjective
measurements can be more successful.
distractibility
Self-report (more accurate)
operationalized measurements(less accurate)
13
Differential Sensitivity
When dealing with multidimensional concepts like
workload.
subjective
objective
Difference between two measurements no
necessarily wrong. In fact, the difference may
signal interesting information.
14
Predictive Indicators
When dealing with predicting something like
stress load.

subjective(how one copes)
objective(what is happening)
Together, they make a more effective predictor.
15
How do you decide what measures to use?
Define
functions dimensions
informationneeds
specify dimensions to be measured
generate candidate measures by dimensions
16
Comparing Candidate Measures

• Relative Simplicity
• simplify for precision without throwing out
  critical information
• Adequate Validity
• are you measuring what you think you are?
• (human observers good for this)
• Sufficient Reliability
• human observers can often have high
• interrater reliability
• Appropriate Precision
• unit of measurement
• accuracy

17
Comparing Candidate Measures (continued)
5. Non-reactivity As in Heisenberg
principle observation distorts the phenomenon
being measured. 6. Generalizability Subjective
measurements can contribute to more
generalizable results, i.e. faculty publications
quantity vs. quality. 7. Data Processing
Requirements deciding how data will be
processed (not just letting the machine
decide). 8. Resource Requirements are
humans or machines cheaper to employ?
18
Comparing Candidate Measures (continued)
Multiple Criteria What are the tradeoffs?
Example simplicity vs. precision vs.
generalizability

• Select candidate measure sets based on
• information needs
• instrumentation data processing
• cost-effectiveness
• credibility

19
Summary
Human beings will always be part of the
measuring process, and they can often collect
more relevant data than some objective methods.
The trick is to learn how to combine both
subjective and objective measurements in a way
that makes sense for the specific research
question.