Measurement and Data Collection

1
Chapter 15

• Measurement and Data Collection

2
Measurement Scales

• Nominal
• Ordinal
• Interval
• Ratio

3
Reliability

• Reliability - the consistency of an instrument in
  measuring what it is designed to measure
  (similarity of results over time)
• Reliability Coefficient - a numerical indicator
  of the consistency of a measurement instrument
  (from 0.00 to 1.00)

4
Reliability Measures

• Parallel (Alternative Forms)
• Test-Retest
• Split-Halves
• Kuder-Richardson
• Cronbach Alpha
• Inter-rater
• require only one administration of instrument

5
Validity

• the extent to which an instrument measures what
  it is designed to measure (internal validity)
• Content-related evidence
• Criterion-related evidence
• concurrent
• predictive
• Construct-related evidence

6
Commercial Instruments

• eliminate need to develop own instrument
• require little/no pilot testing (if used with
  appropriate population
• have available reliability/validity statistics
• overall - reduces effort in preparing for data
  collection
• But more expensive and must be appropriate for
  the purpose of the study

7
Types of Instruments

• Achievement tests
• Attitude measures
• Aptitude tests
• Personality measurements
• Rating scales
• Observation systems

8
Holistic Scoring

• Dependent variable - qualitative in nature (e.g.
  overall quality of an essay)
• holistic - scores the response as a whole, rather
  than in parts
• focuses on global quality
• subjective
• often ordinal - places responses in one of
  several ordered categories

9
Holistic Scoring

• Summarize the qualitative aspects desired in the
  response (a model response)
• Score responses anonymously and in random order
• Have two/more equally competent scorers score
  responses independently
• Use a third scorer to resolve discrepancies in
  the scores of two scorers
• Score all responses to one item before beginning
  the next item
• Make scorers aware of irrelevant factors that
  might affect scoring
• Be aware of factors such as fatigue and boredom

10
Where to Find Test Information

• Buros Tests in Print III
• Buros Tenth Mental Measurement Yearbook
• Index to Tests Used in Educational Dissertations
  (Fabiano, 1989)
• Educational Testing Service (ETS) Test Collection
  Catalog (Oryx Press)
• Volume 1 - Achievement Tests and Measurement
  Devices
• Volume 2 - Vocational Tests and Measurement
  Devices
• Volume 3 - Tests for Special Populations
• Mirrors for Behavior (Simon Boyer, 1974)
• Test Publishers
• Measurement Journals

11
Ethical Legal Considerations

• protect participants' dignity and welfare
• prevent undue intrusion of privacy
• maintain ethical standards of coworkers/assistants
  
• inform participants of the purpose of the
  research
• inform participants of their role in the project
• inform participants of the research results
  (optional)

12
Procedures (Fowler, 1984)

• Gather written confidentiality of all having
  access to data (who can identify respondents)
• Minimize links between identifiers and answers
• Prevent access to completed instruments by
  nonproject members
• Remove identifiers before showing to nonproject
  members
• Do not show data to those able to identify
  respondents by their demographic profiles
• Do not allow general users of the data file to
  link identifiers and responses
• During analysis, present data for small
  categories of respondents very carefully
• The researcher is responsible for the storage or
  eventual destruction of instruments when the
  research project is completed