Doing Qualitative Data Analysis on Language Learning and Teaching

1
Doing Qualitative Data Analysis on Language
Learning and Teaching

• ?????????????

?? ?????????? ???????? ???
????????????? 98????????? III (10/24/2009)
2
Before we start

• One BIG TRUTH or many small truths?

• Do you believe there is one single correct
  perspective or there are multiple perspectives
  that may be all true?

3
Before we start
Doing qualitative data analysis is like solving a
jigsaw puzzle!
4
Outline
Doing Qualitative Data Analysis on Language
Learning and Teaching

• Why qualitative research?
• What are qualitative data?
• Types of qualitative data analysis
• Data preparation and management
• Data analysis process
• Categorization, display, and interpretation
• Quality of qualitative analysis

5
1. Why qualitative research?
Quantitative Research Qualitative Research
Assumptions about reality Single can be broken down into parts studied Multiple can only be studied holistically
Purpose of research To generalize, predict, and posit causal relationships To contextualize and interpret
Research questions Deductively formulated Inductively formulated
Research design Begins with a hypothesis and set methodology Evolves over time and is affected by the data gathered
Typical data A large, random sample data include numerical indices involving tests or responses to surveys A small, purposeful sample data include field notes, interviews, recorded activities, and documents
Data analysis Statistical Interpretive
(adapted from McKay, 2006, p. 7)
6
1. Why qualitative research?

• Strengths of qualitative data
• naturally occurring
• focused and bounded phenomenon embedded in a
  particular context
• rich and holistic with strong potential for
  revealing complexity
• typically collected over a sustained period (this
  allows us to go beyond snapshots of what or
  how many to how and why things happen as they
  do)
• with strong emphasis on peoples lived experience
  (their perceptions, assumptions, prejudgements,
  presuppositions)

(Miles Huberman, 1994, p.10)
Can you also point out weaknesses of
qualitative data?
7
1. Why qualitative research?

• A qualitative approach is a person-centered
  enterprise.
• Qualitative research is designed to
• explore the complexities and problems of the
  complicated social world
• understand the patterns and purpose in our
  behavior and provide insights that will enrich
  our understanding
• have personal engagement with the lived world

(Richards, 2003, pp. 8-9)
8
2. What are qualitative data?
Types of qualitative data Examples
Samples of learner language - Spoken oral presentations, public speeches, discussions in pairs or groups, role plays, online chats - Written compositions, notes, responses to questions, blogs, emails, online discussions
In-class activities and interactions - observation notes - audio/video recordings
Reports from learners/ teachers - responses to open-ended questions in questionnaires - interviews - journals - learning / teaching histories - self-revelation using think-aloud - feedback given to students
Teaching materials - textbooks, supplementary materials
9
3. Types of qualitative data analysis
Types of QDA
Ethnographic analysis
Content analysis
Linguistic analysis
Discourse analysis
Interaction analysis (can be subsumed in discourse analysis)
Focus
Context
Meaning / theme
Form (linguistic features)
Form and function in a particular context
Interaction/communication pattern and strategy use
10
3. Types of qualitative data analysis

• Your research purpose will determine what type(s)
  of analysis to be used.
• Examples
• 1) to identify and understand students errors
  in the use of the past tense
• 2) to identify and understand students problems
  in making requests
• 3) to understand how students solve problems in
  groups
• 4) to understand how students perceive the
  effectiveness of a new learning program
• 5) to find out why students keep silent to
  teachers questions
• 6) to find out why students feel anxious in
  foreign teachers classes
• 7) to compare how students write persuasive
  essays in Chinese and in English
• 8) to understand how foreign teachers view
  Taiwanese students language learning attitudes

11
4. Data preparation and management

• Data organization
• Data selection
• Data transcription

12
Example 1
13
Example 2
14
4. Data preparation and management

• Transcribing data
• Is it necessary to transcribe all the interviews
  or recordings of interactions?
• How detailed or tidy does a transcription need
  to be (e.g., linguistic forms, errors/slips of
  the tongue, gap fillers, pauses, paralinguistic
  features, non-verbal behavior, gestures, eye
  gaze, contextual information)?
• How should the data be transcribed into segments,
  such as lines, sentences, and paragraphs?
• How and when should the data be translated if the
  data are not recorded in the target language?
• What tools can be used for more efficient data
• transcription?

15
Example 1A
A students oral narrative

• Ill talk about one simple event that I went to.
  I guess that is one thing that I feel really
  surprised. That is when last time we had Moon
  festival. During the moon festival we all
  together went to have a barbecue. But I normally
  thought when I come to this event, I only think
  that I will just have something that they want me
  to do, not to think of something very interesting
  to do during the barbecue. I first find out one
  thing that just existed in my imagination before.
  That is seeing somebody who can play guitar very
  well and sing in front of us, because that image
  only exist in my imagination, because I dont
  think that I will meet someone who can just
  perform that well when I am in the university. So
  this really makes me feel that I really enjoy my
  university life .

16
Example 1B

1. uh..Ill talk about one simple ..umeventthat I
   went to
2. I guess..uhI guess that is one thing that I feel
   really surprised
3. That is whenuhlast time we hadMoon festival
4. During the moon festival we..we all together went
   to have a barbecue
5. But I...normally I thought when I come to this
   event
6. I only think that I will just have someuh
7. or something that they want me to do
8. not to think of something very interesting toto
   do
9. souhdurin.. during the barbecue
10. I first find out one thing that..that I.
11. that just existed in myin myimagination before
    
12. that isseeing somebody who canuhplay guitar
    very well
13. and sing in front of us
14. Because that image only exist...in my...in my
    imagination
15. Because I dont think that I will mis..meet
    someone who can just perform that well
16. uhwhen I am in the collin the uniuniversity
17. So..uhthisreallyuhmakes me feel that
18. I reallyuh enjoy my university life

17

1. Helen Well, we need to tell them ((sees
   John coming into the office))
2. Hi, John, how // are ? you?
3. John // Hi
   
4. Sue Hi // John ?
5. John // Hi Sue
6. Helen Wel?come to the CELLAR.
7. John Thanks.
8. Helen You had a pretty tough summer, uh?
9. John (.) Yeah ?, you heard?
10. Helen Yea?h ?
11. Sue Yeah?, I heard // of your story too.
12. John // Oh, yeah??
    Everybody knows ? heh, heh ((chuckles))
13. Helen Yeah, youve been heard of EVERYwhere
14. John Yeah ? ((smiles but feels embarrassed))
15. Sue ((laughs))
16. Helen ((sees John have a book with him)) So, you
    wanna sell back your book here?
17. John No ((chuckles)), I came to uh drop
    this book to um Eun-young
18. Helen Ok. ((The telephone rings and Helen
    answers the phone))
19. John Ill put my name on it for the first
    time.

Example 2
18

• Symbols used in the transcription
• . - indicates a stopping fall in tone at end of
  utterance
• , - indicates a continuing intonation
• - indicates an extension of the sound or
  syllable
• ? - indicates a rising intonation at tend of
  utterance
• ! - indicates an animated tone
• - indicates no interval between adjacent
  utterances
• // - indicates overlap of two utterances
• ? - indicates quieter than the surrounding talk
• - indicates a rising shift in intonation
• ? - indicates a falling shift in intonation
• (.) - indicates very slight pause
• (( )) - indicates other voice qualities or the
  analysts observation notes
• Capital initial - indicates start of utterance
• Capital letter - indicates heavy accent and
  emphasis

19
Example 3
A Group discussion on a gender issue

• Carrie ????????????,??????????,?????????,???????
  ???
• If you think this is treating women unfairly,
  asking men to do all the housework would imply
  the same. Dont you think its weird?
• May ?????,???????????
• Why is it weird? Dont you think its righting
  the wrong?
• Carrie ????????,????????,???????????
• Does that mean men will come back to dominate
  women in 10 years? And will this become a vicious
  circle?

20
Example 4
Interview data

• ??? ?????????! ???????????????????????????????
  ????????????????????????????????,?????????????????
  ??????,?????????????,?????????????????????????????
  ????????????
• Student A It was really a torture in the
  beginning! The schoolwork had already been too
  heavy to make me breathe. Then the teacher added
  this time- and effort-consuming job to us. I
  almost gave up in the first and second
  discussions. However, when I survived to the
  third discussion, I started to feel it was fun.
  Everyone was expecting to read others postings
  and getting their feedback. Frankly speaking, I
  think I did learn a lot both in English writing
  and the information searching skills on the
  Internet. I hope we can still continue this
  activity.

21
5. Data analysis process
(Seidel, 1998)

• Notice read carefully, disassemble, identify,
  and code
• Collect sort and sift
• Think search for patterns relationships, and
  make sense of data

• Doing QDA is an iterative, progressive,
  recursive, and holographic process.

22
5. Data analysis process
Data display interpretation
Figure 6.1 Aspects of analysis (Richards, 2003,
p. 271)
23
5. Data analysis Categorization

• (Merriam, 1998, p. 179)
• Devising categories is largely an intuitive
  process,
• but it is also systematic
• and informed by the studys purpose,
• the investigators orientation and knowledge,
• and the meanings made explicit by the
  participants themselves.

24
5. Data analysis Categorization

• Guidelines for category construction
• (Merriam, 1998, pp. 183-184)
• categories should reflect the purpose of the
  research
• categories should be exhaustive
• categories should be mutually exclusive
• categories should be sensitizing
• categories should be conceptually congruent

25
5. Data analysis Categorization

• Essential features of an adequate category
• (Richards, 2003, p. 276)
• Analytically useful
• When used, does it contribute anything to
  understanding?
• Conceptually coherent
• Does it make sense in terms of the conceptual
  framework within which interpretation will be
  framed?
• Empirically relevant
• Can it be mapped onto the data?
• Practically applicable
• Is it possible to specify criteria that can be
  used to assign data bits to the category?

26
5. Data analysis Categorization

• Levels of analysis (Strauss Corbin, 1998)
• Open coding breaking down the data for the
  purpose of categorizing, conceptualizing and
  comparing
• Axial coding looking for patterns and
  concentrating on organizing the data, based on
  the axis of a category. It involves relating
  categories to subcategories and making
  connections between categories.
• Selective coding identifying a central category
  or explanatory concept in terms of which other
  categories can be refined and integrated

27
Example 1
Analysis of Online Discussions
Participation
Length
Participants Discussions
Interaction patterns
Exchange structure- IRF
Frequency
Thinking skills
Feedback type questions vs. comments
Feedback function information exchange vs.
relationship building
Language
Responses to Questionnaire
Perceived learning effectiveness
Content
The relationship between the students online
interaction and their learning effectiveness
The relationship between the students online
interaction and their learning effectiveness
Focus group Interviews
28
Analysis of thinking skills
Example 2
Signal words / Question stems Cognitive operations
Memory questions who, what, where, when naming, defining, identifying, designating, yes or no responses
Convergent thinking questions why, how, in what ways? explaining, stating relationships, comparing and contrasting
Divergent thinking questions imagine, suppose, predict If ... then ... , How might ... , What are possible consequences... predicting, hypothesizing, inferring, reconstructing
Evaluative thinking questions defend, judge, justify/ What is your opinion ...? Why do you think is better? Valuing, judging, defending, justifying choices
(Blanchette, 2001 Ciardiello, 1998 Gallagher
Aschner, 1963)
29
Example 3
Analysis of Reasoning Performance
Type of fallacies Definition
Failure to follow Reasons irrelevant or contradictory to a point
Faulty evidence Facts, experiences and authoritative sources falsely used to support a point
Overgeneralization Generalizations made without giving representative or sufficient reasons/evidence
Shifted focus Reasons/evidence deviating away from a point to a less significant one
Straw person Failure attack due to a misinterpretation of the opponents views
(Ennis, 1996)
30
Analysis of Politeness Strategies
Example 4
31
Thematic Analysis of Students Views of Gender
Images
Example 5

• Mens reluctance to give up power to women
• Mens control over women as sex objects
• Mens search for independence
• Womens perceived subordination to men
• Womens emotional dependence on men
• Womens longing for love
• Womens longing for physical beauty
• Womens quest for self-liberation
• Womens conflicts between the traditional and
  modern self

32
5. Data analysis Display

• Display refers to visual format that presents
  information systematically, so the user can draw
  valid conclusions and take needed action.
• (Miles Huberman, 1994, p.91)
• Common displays of qualitative data
• Quotations
• Vignettes / narratives
• Tables / figures
• Excerpts / samples of written works/documents
• Pictures / images

33
Content/thematic analysis of interviews
Example 1

• Womens conflicts between the traditional and
  modern self
• Doris In the end of the episode, its still the
  mother who holds the childs hand after they got
  out of the car, not the father. Although womens
  status is enhanced and we can do things men do,
  we still have to do the work that we were
  previously designated to.
• ????????,???????,????????,???????????,???????????
  ,????????????????
• Nancy I think this commercial speaks for women
  in the contemporary society. On one side, you
  want to be independent, but on the other side you
  still want to be cared for. For me, I can drive
  and Im good at directions. Cars represent a
  symbol of independence to me. But at the same
  time, I still want to be a little girl taken good
  care of by men.
• ??????????????????????????????,?????????,????,???
  ???,??????????????????????????????????,????????,??
  ??????????

34
Example 2
Ethnographic analysis of three classes
Class A Class B Class C
Purpose of Use To facilitate students writing process and evaluate their writing performance at the initial stage To facilitate students writing process To grade students essays and replace the human grader
Functions Emphasized Both the grading system and the writing resources/ editing features The writing resources and editing features The grading system
Requirements Grading Policy Students were asked to write multiple versions on assigned topics at home. They were not allowed to turn in their essays to the teacher until they gained a score of 4 given by the program. Students were asked to write drafts on assigned topics in class and develop them into better organized essays at home. The scores given by the program were not important. Students were asked to write multiple versions on assigned topics at home. The scores they obtained from the program accounted for 40 of their final grades.
Duration of Use 4 months 1 month 4 months
Post-grading Feedback Consultation The teacher gave individual written feedback and held in-class discussions on students essays. The teacher gave individual written feedback and provided after-class consultation with students when needed. The teacher gave no feedback to students essays and little consultation in class.
35
Ethnographic analysis of a students email
practice
Example 3
To Whom When For What Purposes
Lawyer Feb. 1999 Jan. 2001 - to request advice for Green Card application - to request action or assistance - to check progress of the application process - to make an appointment - to respond to the lawyers questions
Administrative Personnel Administrative Personnel Administrative Personnel
a) in his work place Nov. 1999 Feb. 2000 - to request information/assistance regarding his employment visa application
b) in different universities Oct. 2000 May 2001 - to request information/assistance regarding his Ph.D. study application
Professors Nov. 2000 Dec. 2001 - to make an appointment - to request recommendations - to request permission - to request simple information - to show his academic interests - to give thanks
Friends (mostly Taiwanese who did not live close to him) May 1999 Dec. 2001 - to send personal care and greetings - to share life experiences - to give thanks - to request information/assistance - to give information - to express disagreement
36
Example 4
Discourse analysis of a students email requests
Request Types Linguistic Forms of Request Acts Examples
Requesting advice (a) Please Imperative (b) Want Statement (c) Interrogative - (If you have time/If you know the answer), Please let me know/tell me. - I need your suggestion/advice. - What should I do? - Do I need to contact ?
Requesting action (a) Please Imperative (b) Query Preparatory - Please review this file for me. - Can/Could you send xxx to me?
Checking work progress (requesting a result or a confirmation) (a) Just Want Statement (b) Interrogative - Just want to know/ask if you ? - Have you sent the form to ?
Making an appointment (requesting a meeting) (a) Want Statement (b) Query Preparatory - I want/hope to make an appointment with you . - Can we find sometime next week to talk about ?
37
Example 5
Table 4.1 Structures of the students oral
narratives
Structure Elements Level Groups Structure Elements Level Groups Abstract Orientation Orientation Orientation Action Action Action Evaluation Result Coda
Structure Elements Level Groups Structure Elements Level Groups Abstract Complete Complete Incomplete Complete Complete Incomplete Evaluation Result Coda
Structure Elements Level Groups Structure Elements Level Groups Abstract With evaluative devices Without evaluative devices Incomplete With evaluative devices Without evaluative devices Incomplete Evaluation Result Coda
High (N3) O12 Y 1 1 0 8 3 1 6 Y Y
High (N3) O14 N 3 2 0 9 4 2 13 Y Y
High (N3) O15 Y 0 2 0 8 1 0 4 Y Y
Middle (N3) O8 Y 0 4 0 3 8 1 5 Y Y
Middle (N3) O10 Y 4 4 0 9 2 0 5 Y N
Middle (N3) O11 Y 2 1 0 2 7 0 2 Y N
Low (N9) O1 N 0 0 0 4 8 9 0 N N
Low (N9) O2 Y 6 3 0 2 2 0 8 Y N
Low (N9) O3 Y 1 3 2 4 2 6 2 Y N
Low (N9) O4 Y 1 1 0 3 1 0 9 N Y
Low (N9) O5 N 6 3 0 0 0 0 10 N N
Low (N9) O6 N 5 1 5 0 0 0 13 N N
Low (N9) O7 Y 4 9 3 3 1 2 8 Y N
Low (N9) O9 Y 5 3 0 1 0 0 1 N N
Low (N9) O13 Y 1 8 0 1 0 2 7 Y N
38
Discourse analysis of online debate
Example 6
Partial agreement

• Dear Hedy
• According to your statement, imitation is
  important when learning an L2. I agree with your
  point of view, and your examples are quite true.
  However, i have a few questions to the examples
  you mentioned above. We learn L1 by imitating
  from the videos or tapes that we heard, or from
  the teachers and parents...etc, but when we learn
  an L2, sometimes you cannot "acquire" it that
  easily. What if the teachers or parents don't
  speak the L2 at all? What if the language is not
  spoken by many people and there are no programs
  or tapes or books to learn at all? We speak L1
  everyday, we speak it everywhere, there is only a
  little time to actually imitate an L2, and even
  if there is a plenty of time, there might be
  errors when we are practicing it. I would like to
  know what's your point of view on this problem.

Indirect disagreement
Counter- argument
Mitigated request for response
39
Content analysis of students attitudes toward
online discussions
Example 7
Improving writing fluency

Figure 4.2 A students perceived improvement in
writing
40
5. Data analysis Interpretation

• Data interpretation making sense of data
• To explain
• how things work how people do things
• why things are working or not working
• how things can be made to work better
• To relate findings to theories or develop new
  theories
• To compare the present study with existing
  empirical studies and make new contributions

41
Example
Figure 3.2 Analytical Framework of Email
Practices
42
Lets review the data analysis process
(Richards, 2003, p. 271)
43
6. Quality of qualitative analysis

• A qualitative study can be judged in three
  aspects
• Process What data collection and analysis
  methods are used to ensure the validity and
  credibility of the findings?
• Product How adequately are the data represented
  and interpreted? What contribution does the study
  make?
• Position What does the researcher bring to the
  study in terms of his/her orientations,
  standpoints, perspectives, qualifications, and
  experiences?
• (Richards, 2003, p. 293)

44
6. Quality of qualitative analysis

• Descriptive validity
• Interpretive validity
• Theoretical validity

• Strategies to enhance validity
• Triangulation
• Constant comparison
• Member checks
• Long-term or repeated observations
• Peer examination
• Clarification of the researchers biases
• Rich, thick description

• Credibility (internal validity)
• Transferability (external validity)
• Dependability (reliability)

(Richards, 2003, pp. 285-287)
45
Final Reminders

• the human element of qualitative inquiry is
  both its strength and weakness
• its strength is fully using human insight and
  experience,
• its weakness is being so heavily dependent on
  the researchers skill, training, intellect,
  discipline, and creativity.
• The researcher is the instrument of qualitative
  inquiry, so the quality of the research depends
  heavily on the qualities of that human being.
• (Patton, 2002, p. 14)

46
Final Reminders

• But research is inherently imperfect, and we
  would support the line of argument that multiple
  perspectives and methods increase the likelihood
  of reaching good explanations.
• (Jaworski Couland 1999, p. 37)
• we need to ensure that there is sufficient
  evidence and sufficient kinds of evidence. the
  link between the evidence we present and the
  interpretations we derive from that evidence must
  be robust.
• (Richards, 2003, p. 283)

47
References

• Jaworski, A. Coupland, N. (1999). Introduction
  Perspectives on discourse analysis. In A.
  Jaworski N. Coupland (Eds.), The discourse
  reader (pp. 1-44). New York Routledge.
• McKay, S. L. (2006). Researching second language
  classrooms. Mahwah, NJ Lawrence Erlbaum.
• Merriam, S. B. (1998). Qualitative research and
  case study applications in education. San
  Francisco, CA Jossey-Bass.
• Miles, M. B., Huberman, A. M. (1994).
  Qualitative Data Analysis (2nd ed.). Thousand
  Oaks, CA Sage.
• Patton, M. Q. (2002). Qualitative evaluation and
  research methods (3rd ed.). Thousand Oaks, CA
  Sage.
• Richards, K. (2003). Qualitative inquiry in
  TESOL. New York Palgrave MacMillan.
• Seidel, J (1998). Qualitative Data Analysis. The
  Ethnography v5 Manual, Appendix E. Available
  online at http//www.qualisresearch.com/
• Strauss, A., Corbin, J. (1998) Basics of
  Qualitative Research (2nd ed.). Newbury Park, CA
  Sage.

48
Questions Comments

• Your questions and comments are welcome.

If knowledge is worth having, it is worth
sharing. Deborah Cameron
??? Dr. Chi-Fen Emily Chen?????????? ????????
Website http//www2.nkfust.edu.tw/emchen/Home
Email emchen_at_ccms.nkfust.edu.tw