Designing Systematic Inquiry: Research Methods and Evidence Gathering

1
Designing Systematic Inquiry Research Methods
and Evidence Gathering

• Renee A. Meyers
• UWM Department of Communication
• 2006 Summer Institute
• June 28, 2006

2
Objectives

• By the end of this session, you should
• Be familiar with, or re-familiarized with, the
  objective/interpretative quantitative/qualitative
  distinction
• Be familiar with a variety of evidence-gathering
  methods
• Have thought about how you might want to gather
  evidence for your project.

3
Initial Example

• Suppose you wanted to investigate the following
  descriptive SoTL question
• What is happening with the feedback we give to
  students (on papers, on speeches, on group
  presentations, on lab assignments, etc.)?
• How would you gather evidence to answer this
  question?
• Jot down a few ideas individually then work in
  groups to come up with some ideas.

4
Questions of EpistemologyObjective or
InterpretiveQuantitative or Qualitative?

• Objective View
• Truth is singular
• Behavior is largely determined
• Universal laws
• Quantitative methods
• Interpretive View
• Truth is largely subjective
• Behavior is conscious choice
• Contextual behavior
• Qualitative methods

5
Methods for Gathering SoTL Evidence

• More quantitative
• Questionnaire/survey
• Quasi-experiments
• Content analysis
• Secondary analysis (test data, student records)
• More qualitative
• Focus groups
• Interviews
• Talk-alouds/Protocol Analysis
• Observation
• Text and discourse analysis

6
Making the Decision Benefits/Drawbacks

• In pairs/groups, identify the main benefits and
  drawbacks to each evidence-gathering method.

7
Surveys/Questionnaires

• Benefits
• Efficient
• Less time consuming for participant
• Works for a variety of topics
• Drawbacks
• Hard to design good survey
• Subject to varying interpretations
• Possibility of biased or leading questions
• Self-report datahow accurate? Sensitive
  questions?
• Response rate

8
Example Partial Survey

• Which of the following do you think are true
  about working in class groups? Use the following
  scale to rate each statement
• not at all true very true
• 1 2 3 4 5 6 7
• _____Some group members do too much while others
  do too little
• _____Working in a group allows you to meet new
  people
• _____There are scheduling conflicts
• _____There is greater input because we work as a
  group
• _____It is hard dealing with difficult group
  members
• _____I get feedback on my ideas
• _____It is hard to get all members to agree
• _____I can learn people skills
• _____Not all group members have the same sense of
  purpose
• _____There is greater objectivity because we work
  as a group
• _____Some people are shy around others
• _____I learn from others experiences
• _____Some of the group members are not in the
  mood to work in a group
• _____I can improve my interpersonal communication
  skills
• _____There is always a fear of someone not
  showing up

9
Surveys/Questionnaires

• Overcoming drawbacks
• Use both closed and open-ended questions
• Pre-test on sample similar to those you will be
  using in actual investigation
• Use adequate number of response categories
• Be sure question are relevant
• Use incentives for completing survey

10
Quasi-Experiments

• Comparison of two sections of same course where
  one section has an intervention
• Benefits
• Increases confidence in causation
• Considered more generalizable
• Drawbacks
• Difficult to isolate causes
• Selection effectare two groups exactly the same?
• Mortality effectstudents drop out
• Non-representation sample bias (all adult
  students all freshmen, all Communication majors,
  etc.)

11
Quasi-Experiments

• Overcoming Drawbacks
• Carefully choose two groupsmatch on as many
  factors as possible
• Carefully control as many contaminating factors
  as possible
• Use random assignment when possible

12
Content Analysis

• Analysis of text or interaction data looking for
  patterns or coding into categories
• Benefits
• Provides structure for analysis of textual data
• Allows quantitative analysis of large amounts of
  textual evidence
• Drawbacks
• Category scheme development is time-consuming
• May overlook some complexities in data
• Coding is time-consuming and often difficult
• Reliability of coders is issue

13
Example Coding Scheme for Content Analysis

• Evidence in Group Quiz Discussions
• No Evidence categories
• Provide claim but no evidence (I put that too, I
  put true, I put D, nonverbal agreement, Yeah,) a
  claim is verbally or nonverbally stated, but no
  evidence is provided here at all.
• Provides neither claim nor evidence doesnt
  speak to quiz question.
• Indicate lack of own knowledge or expertise as
  source
• Evidence Provided categories
• A. Expert-Based Evidence--Use sources other
  than self
• Use/Cite text authority (It said in the book, I
  remember from the book, I dont remember reading
  that in the book, The text said that . . .,etc.)
• Use/Cite teachers authority/expertise (She said
  this in class, I asked her, etc.)
• Use/Cite class discussion as authority (We talked
  about this in class, When we did this in class,
  Remember when we did that activity in class,
  etc.)
• Use/Cite another question on the exam as evidence
  
• Use/Cite class notes as authority (I wrote it in
  my notes, I remember from my notes, etc.)
• B. Speaker-Based EvidenceUse self as source
• Use own authority (I know that is right, I
  remember that, I know this is true, I am sure I
  am right) OR
• use own sources such as an example or
  illustration or definition or clarification to
  clarify/explain the concepts/theories/wording in
  test question, etc.
• III. Othertypes of evidence that are NOT
  contained in the codes above

14
Content Analysis

• Overcoming drawbacks
• Tie coding scheme to theory and previous
  research
• Pretest coding scheme on similar data
• Train coders very thoroughly
• Check reliability part way through coding process

15
Secondary Analysis

• Data collected for some purpose other than the
  investigation at hand
• Benefits
• Less costly (usually)
• Less time consuming
• Readily available
• Drawbacks
• May not be completely relevant to your question
• May not be completely accurate
• May not know who collected the data, how they
  were collected, why they were collected

16
Secondary Analysis

• Overcoming the drawbacks
• Data modification (convert the units, redefine
  the content categories, update the data)
• Data miningferreting out the data that is most
  useful, relevant

17
Focus Groups

• Discussion moderator introduces a topic/question
  to group of participants and guides their
  discussion in non-directive fashion
• Benefits
• Richness of data
• Versatility
• Drawbacks
• Lack of generalizability
• Opportunity for misusemust be careful with
  interpretation
• Cost
• Time-consuming
• Communication issues status, conformity,
  apprehension, dominance, etc.

18
Example Focus Group Protocol

• Student Perspectives on Learning Through Research
• In your own words, how would you describe or
  define research?
• Why do people do research?
• What does research accomplish?
• What are the overall goals of researchers?
• Why do students participate in research?
• Have you been involved in any research in the
  past? If so, describe that experience(s)?
• Why did you participate in this research?
• What did you learn by participating in this
  research?
• Was this experience a valuable learning
  experience for you? Why or why not?
• If so, what did you learn by participating in
  this research?
• If not, what could have been done differently so
  that the experience would have been a more
  valuable learning experience?
• Are you currently involved in research? If so,
  describe that experience.
• Why did you participate in this research?
• What do you expect to learn by participating in
  this research?
• Do you think you will accomplish those learning
  goals? Why or why not?
• If you were to design the ideal research project
  for a student, what would it look like? Who
  would be involved? How would the project proceed
  from start to finish?
• Have you ever been involved in a research project
  that looked like this ideal?

19
Focus Groups

• Overcoming drawbacks
• Carefully choose participants
• Choose and train moderators carefully
• Pretest questions
• Set up a relaxed atmosphere
• Capture on audio- or videotape if possible

20
Interviews

• Face-to-face one-on-one question-answer sessions
  with open-ended questions
• Benefits
• Reactions with influence of peers
• More in-depth probing
• Allows for psychological depth (attitudes,
  motivations, explanations, etc)
• Drawbacks
• Time consuming
• Difficult to get participants
• Questions/Responses may be biased or selective
• Hard to get sensitive data
• Interviewee may go off-track hard to control

21
Interviews

• Overcoming drawbacks
• Pre-test questions in practice interviews
• Offer incentives for participation
• Select and train interviewers carefully
• Allow for flexibility, but have structure
  available
• Audio- or videotape if possible

22
Talk-Alouds/Protocol Analysis

• Student describes process out loud while
  performing process
• Benefits
• Allows access to process evidence
• Provides rich source of data
• Drawbacks
• Students may have difficulty describing process
• Social desirability bias possible
• Time consuming
• Difficult to get students to participate

23
Talk-Alouds/Protocol Analysis

• Overcoming drawbacks
• Pre-test process of think-aloud
• Select process that is not overly complex
• Draw up set of guiding questions to help student
  (apply only if necessary)
• Offer incentives for participation
• Audio- or videotape if possible

24
Observation

• Classroom observations of some activity
  note-taking and/or audio or videotaping
• Benefits
• Captures actual behavior or process
• Fairly easy to accomplish
• Provides rich data
• Drawbacks
• Can seem intrusive
• Hard to capture everything (note-taking) can be
  hard to observe AND take notes
• Difficult to know what to observe (sometimes)
• Quality may differ across observers

25
Observation

• Overcoming drawbacks
• Simply process for observation
• Pretest observation skills and note-taking
  skills
• Carefully select and train observers
• Seek to be as non-intrusive as possible
• In note-taking, take as extensive notes as
  possible (or write down everything you remember
  immediately afterward)

26
Text and Discourse Analysis

• Analysis of texts via interpretation or discourse
  analysis
• Benefits
• Rich data deeper understanding
• Convenient sources of data (papers, portfolios,
  essays, reflections, etc.)
• Drawbacks
• Singular interpretation is not generalizable
• Interpretation must be well supported or is seen
  as possibly inaccurate multiple interpretations
• Difficult to get published?

27
Text Analysis

• Overcoming drawbacks
• Rigorous and well-argued analysis
• Multiple sources of evidence utilized in analysis
  (papers, reflections, essays, presentations,etc)
• Check back with participants for validity

28
Making the Decision

• Based on your question, and your disciplinary
  expertise, what form(s) of evidence do you want
  to gather? How will you go about gathering that
  evidence?

29
Summary

• By now, you should
• Be familiar with, or re-familiarized with, the
  objective/interpretive quantitative/qualitative
  distinction
• Be familiar with a variety of evidence-gathering
  methods
• Have thought about how you might want to gather
  evidence for your project.
• Access http//sotl.uwm.edu for many more
  resources!