Experimental Research I

1
Experimental Research I

• Day 3

2
Business

• For Tomorrow
• One article from Day 2 reading list.
• Prepare to summarize comment
• Examine 1 Thesis proposal online
• List of 12 research articles in APA format
  related you your study (bibliography)
• Reference List APA example
• Kotora, E. J. (2005). Assessment practices in the
  choral music classroom A
• survey of Ohio high school choral music teachers
  and college choral methods professors.
  Contributions to Music Education 32(2), 65-80.
• Spaces b/w reference elements 1 b/w sentences
  in text 2.

3
Experimental Research

• Only type of research with an intervention
• A direct attempt to influence a particular
  variable
• Only method that can truly begin to untangle
  cause and effect hypotheses
• Directional Hypothesis Theory statement
  predicting the outcome directional (There will
  be a significant difference). Reflects
  researchers expectations.
• Bilingual 3rd graders taught with the Kodaly
  method will demonstrate significantly higher
  musical achievement than bilingual 3rd graders
  taught with a traditional eclectic method.

4
Null Hypothesis

• Null Hypothesis Theory statement predicting the
  outcome stated in the negative non-directional
  (There will be no significant difference) The
  statistical hypothesis that states that there are
  no differences between observed and expected
  data. Does not reflect researchers expectations
  (value free)
• There will be no significant difference in
  musical achievement of bilingual 3rd graders
  taught with the Kodaly methods and bilingual 3rd
  graders taught with a traditional eclectic
  method.
• The goal is to REJECT the Null Hypothesis based
  (on 95 Confidence level or above)
• Cannot prove the null hypothesis (a negative)
• E.g. not guilty does not innocent
• Reject/not reject vs. accept

5
Type I and Type II Error

• Type I Error is erroneously claiming statistical
  significance or rejecting the null hypothesis
  when in fact, its true (claiming success when
  experiment failed to produce results)
• Possible w. incorrect statistical test
• Type II Error is when a researcher fails to
  reject the null hypothesis when it is in fact
  false (claiming failure when successful)
• The smaller the sample size, the more difficult
  it is to detect statistical significance
• In this case, a researcher could be missing an
  important finding because of study design

6
Group Comparisons

• Experimental Group
• Receives a particular treatment specified by the
  researcher
• Control/Comparison Group
• Does not receive that particular treatment
• Sometimes difficult in educational research to
  have a strict no-treatment, control
• Example Any instruction is likely to be more
  effective than no instruction

7
Randomization

• Random assignment to groups
• Every individual has an equal chance of being in
  the experimental or control/comparison group
• Supposed to help eliminate extraneous sources of
  variance
• For example if the groups are sufficiently
  large, any differences between groups on
  extraneous variables are likely to be due to
  chance or randomly distributed among the groups
• Quasi-Experimentalnon-randomized groups
• Most ed. research
• Intact classes convenience samples
• Impacts ability to generalize to whole population

8
Variables

• Independent variable (IV)
• The experimental or treatment variable
• This variable is manipulated by the researcher
• Examples instructional approach, environmental
  condition, the introduction of a particular
  musical element
• Participant attribute
• Dependent variable (DV) Compared b/w groups
• The criterion or outcome variable
• Examples student attitudes, student achievement,
  teacher effectiveness as measure by ?
• Experiments can be expressed as The effect of
  the IV on the DV
• Extraneous Variables
• Those that are not specifically included in the
  study but never the less may effect the outcome
• Object is to control for extraneous variables
• The researcher may not know them all

9
Manipulating the IV

• Presence of the variable vs. absence of the
  variable
• Kodaly instruction (treatment group) vs. no
  Kodaly instruction (control group)
• One form of the variable vs. another
• modeling vs. verbal music instruction (vs.
  control group?)
• Varying degrees of the same variable
• 100 positive feedback, no negative feedback vs.
  50 positive feedback, no negative feedback

10
Controlling for Extraneous Variables

• Best case scenario all individuals are as
  similar as possible on all variables other than
  the independent variable
• Methods to control
• Randomization large sample
• Holding variables constant (freeze private
  lessons)
• Build variable into the design (compare private
  lessons w/ no private lessons)
• Matching pairs one to control, other to exper.
• Statistical control analysis of covariance
  (ANCOVA)

11
Design and ExperimentEffect of Colored note
heads on Music Reading

• State Hypothesis and Null Hypothesis
• Select sample and assign to group (control and
  treatment). How many in each?
• Identify independent and dependent variables. Any
  possible extraneous variables?
• Describe experiment. What will you do w/ each
  group and for how long? How will you know what
  they already know?

12
Discussion of Projects

• On task
• Practice explaining your project
• Background State the problem
• Purpose statement
• Research questions
• Methodology (research design)

13
Experimental Research Designs
14
Nomenclature/Abbreviations

• When looking at the symbols used to describe
  various experimental design approaches
• R random assignment
• O testing (pre- or post-)
• X treatment
• C control/comparison
• M matched

15
Pre-Experimental DesignsPilot Studies
Generally Weak

• One Shot Case Study (X O)
• No random assignment, No control/comparison, no
  pre-test
• One-Group Pre-test, Post-test (O X O)
• No random assignment, No control/comparison group
• Static/Intact-group Comparison X O
• No random assignment O
• Static/Intact-group Pre-test, Post-test
• No random assignment, possible pre-test effects O
  X O
• O

16
True Experimental DesignsStronger not always
possible in educ.

• Randomized Post-test Only, R X O
• Control Group R O
• Still not sure about pre-test levels
• Randomized Pre-Test, R O X O
• Post-test, Control Group R O O
• Good for checking if groups are actually similar
  at the start of the study and possible effects of
  pretest
• If you do an experiment probably this one
  except w/ non-randomized groups

17
Randomized Solomon Four-Group Design

• Solomon 4Group controls for possible
  sensitization effects due to testing or
  maturation.
• 1. R O X O
• 2. R O O (maturation or pretesting?)
• 3. R X O (effect of pretest?)
• 4. R O (control group)
• In a successful experiment, what would we expect
  for each group?
• What if the Post Test scores for group 2 were as
  high as the Post Test for group 1?
• What if the Post Test scores for group 3 were
  lower than group 1?
• What if the Post Test scores for group 4 were the
  same as groups 1 2?

18
Quasi-Experimental Design

• So called b/c there is no randomization
• Matching Only
• Participants matched in pairs to control for an
  extraneous variable rather than randomly assigned
• Equivalent Materials Design (next slide)
• Counterbalanced Design (2 slides down)
• Multiple groups receive all treatment types in
  different order
• Average post-test scores across groups are
  compared to determine effectiveness/effect of the
  treatment order
• Vulnerable to multiple-treatment interference
• Time-series Design
• Outcome measured several times before and after
  introduction of the treatment O O O O X O O O O

19
Equiv. Mat. Design TuningTuba vs. Clarinet vs.
oboe vs. drone stimulus

• Band (N 1) OX(tu)O OX(clar)O OX(ob)O
  OX(drone)O
• Pre/Post tests indiv. tuning to different
  stimuli. Success mes. w/ a tuner.
• Research questions Which tuning process leads to
  most growth?
• Most interested w/ growth w/i group per treatment
• Other questions Could introducing drone second
  be more effective? How do we know that using just
  one would be just as or more effective? next
  slide

20
Counterbalanced Design(Latin Square) Order effect

• All groups take a pretest tuning to the piano
  A440 no difference in groups after time 1,
  post test serves as next pretest
• Each treatment 2 weeks w/ post-test at the end
• 1 tuba 2 clarinet 3 oboe 4 drone

21
Quasi-Experimental Design

• Factorial Design (Manipulate 2 or more factors
  IVs at different levels)
• Allows for examination of attribute (vs.
  manipulated) variables (i.e. gender, age) and
  interaction effects b/w combinations of IVs
• Example Effect of teaching method on beat
  competency of ELL and non-ELL students
• Possible outcome showing interaction of two IVs
• Non-ELL students may do equally well w/ Kod. and
  Gordon methods, while bilingual students may do
  better w/ Kod. vs. Gordon.
• What if you had not separated these groups out?

22
Factorial Example(Two Way - 2x2)

• Beat competency improvement using Gordon or
  Kodaly among biling. and non-biling students
• IVs Language classification (biling. vs.
  non-biling.) method (Kodaly vs. Gordon)
• DV rhythm pre- post-tests
• Groups (Six 3rd gr. Sections-3 Kodaly 3 Gordon
  Bilingual Non-Bilingual in all groups.)
• Bilingual Kodaly
• Non-Bilingual Kodaly
• Bilingual Traditional
• Non-Bilingual Traditional

23
2 Way Factorial Designs (2 independent variables
often one manipulated, one attribute)
24
Internal Validity - Effectiveness of Exp.
DesignControl of Extraneous Variables Time
Bound Factors

• What happens within the experiment
• History What happens b/w pretest and posttest
  (private lessons, change in practice routine)
• Maturation is change result of treatment
  natural result of repetition and improvement over
  time?)
• Mortality Loss of participants may cause
  imbalance b/w groups

25
Internal Validity Control of Extraneous
Variables Sampling Measurement Factors

• Testing pretest affect posttest. Ceiling and
  floor effects (eliminate outliers?)
• Instrumentation changes in measurement or
  observers (judges at contest from one site to the
  next)
• Statistical regression students who score
  extremely high (ceiling) or low (floor) on
  pretest may regress to the mean on posttest
• Selection participants do not represent normal
  population (also affects external validity)
• Interactions influence of a combination of the
  above factors

26
External Validity Generalizability

• Population Validity
• Extent sample is representative of the population
  to which the researcher wishes to generalize the
  results.
• Ecological
• Study conditions and setting are representative
  of the setting in which the researcher would like
  to apply the findings (e.g. university lab
  school)
• Replication
• Results cannot be reproduced (problem w/ Mozart
  effect)
• Detailed description of the sample needed in
  study
• Important regardless of sampling method

27
Other Threats to External Validity

• Effect or interaction of testing (testing will
  not occur in natural setting
• Reactive effects of sample
• Hawthorne Effect
• Effects due simply to subjects knowledge of
  being in a study
• John Henry Effect
• Control group performs beyond usual level because
  they perceive they are in competition with the
  experimental group
• Teacher or Researcher interactions different than
  in population
• Subconsciously encouraging or discouraging a
  group

28
Review Effect of Intensive Instruction on
Elementary Students Memory for Culturally
Unfamiliar Music (2013)

• Previous researchers have found that both adults
  and children demonstrate better memory for novel
  music from their own music culture than from an
  unfamiliar music culture. It was the purpose of
  this study to determine whether this
  enculturation effect could be mediated through
  an extended intensive instructional unit in
  another cultures music. Fifth-grade students in
  four intact general music classrooms (two each at
  two elementary schools in a large U.S. city) took
  part in an 8-week curriculum exclusively
  concentrated on Turkish music. Two additional
  fifth-grade classes at the same schools served as
  controls and did not receive the Turkish
  curriculum. Prior to and following the 8-week
  unit, all classes completed a music memory test
  that included Western and Turkish music examples.
  Comparison of pretest and posttest scores
  revealed that all participants (N 110) were
  significantly more successful overall on the
  second test administration. Consistent with
  previous findings, participants were
  significantly less successful remembering items
  from the unfamiliar music culture, a result that
  was consistent across test administrations and
  between instruction and control groups. It
  appears that the effect of enculturation on music
  memory is well established early in life and
  resistant to modification even through extended
  instructional approaches.

29
Identify or State

• Independent Variable
• Dependent Variable
• Treatment Group
• Control Group
• Diagram experimental design (O X)
• Write a hypothesis null hypothesis
• Paraphrase findings
• Implications for the classroom? Did the authors
  reject or not reject the null hypothesis?

30
Sampling
31
Samples of individuals/entities

• Sample vs. population
• Some vs. All
• Examples where entire population could be
  sampled?
• Relationship between sample specificity and
  generalizability
• Representative sample
• Captures relevant and essential characteristics
  of the population
• What about a sample of teachers? What should the
  sample look like?

32
Sampling Methods

• Systematic
• Random start and sampling interval
• i.e., Randomly select pages from IHSA directory
• choose every ? Name (random number b/w 1-X)
• Convenience
• not as valuable but frequent in ed. research
  why?
• i.e., intact classes, pre-service teachers from
  one institution, conference session attendees
• Purposive
• Participants fit a particular profile (female
  band directors in small towns)
• Exclude those who do not fit profile
• Often consists of volunteers (problematic)

33
Types of Samples

• Simple Random
• Everyone has equal chance of selection
• Reduce systematic bias error created by
  sampling method
• Phone book, MENC membership list (But??)
• Stratified Random
• Similar proportions between sample and population
• Gender, race, age, instrument, etc.
• Cluster Random
• Groups rather than individuals
• i.e., classes or ensembles in CPS
• Then groups can be assigned randomly
• Two-stage random - groups then individuals
• i.e., choose classes then assign individual
  students or groups to control or treatment group

34
Sample Size

• As large as possible given reasonable expenditure
  of time and energy
• Most likely to get significant results
• More statistically powerful (more likely to find
  a significant difference b/w groups)
• Sample size relative to
• the size of population (50 Cook Co. band
  directors vs. 50 band students throughout US)
• variability within population (years of teaching,
  gender, etc.)
• sampling method (need a large enough pool from
  which to draw)
• study design (qualitative vs. quantitative)