Title: Crosscultural surveys and translation
1Cross-cultural surveys and translation
- ESRC Question Bank Conference
- SURVEY MEASUREMENT ASSESSING THE RELIABILITY AND
VALIDITY OF QUESTIONNAIRE ITEMS - April 10
- Royal Statistical Society
- Janet A Harkness
gesis-ZUMA
2Outline
- Framework
- Progress in survey translation production and
assessment - Persistent problems consequences
- Towards solutions
3I. Framework
- Survey translation uses
- Expectations
- Current practice
41. Survey translations uses
- Within country research
- Groups may partially share larger context
- Shared contexts, degree of interaction, immigrant
acculturation affect translations required - Across country research
- Different contexts and languages
52. Expectations for survey translations
- Assumption is a good translation
- asks the same question
- maintains semantic pragmatic meaning
- maintains measurement properties
- retains source design features
- satisfies multiple other requirements (askable
and answerable, burden, saliency, etc)
63. Common current practice
- Depending on discipline
- Do not assess translation process quality
- Assess translation quality through back
translation - Pretest translated questions
- Assess translated instrument quality on basis of
statistical analysis (dif, IRT)
7II. Progress in survey translation production and
assessment
8Progress in survey translation production and
assessment
- Guidelines and know-how
- Tools
- Basic research
- Procedures, strategies, outcomes
9Progress in survey translation production and
assessment
- Guidelines and know-how
- Tools
- Basic research
- Procedures and strategies
101. Guidelines and Know-How
111. Guidelines and Know-How
- ESS, USA Census Bureau, International Test
Commission, QoL research and publications - Comprehensive sets of guidelines CSDI Workshop
Guidelines Initiative (lead groups at ISR,
Michigan, UNL and gesis-ZUMA)
122. Tools
132. Tools
- Process documents
- Note-taking templates
- Queries, decisions, rationales
- Harmonization templates
- Decisions, rationales
- Version records
- Decisions, rationales
142. Tools
- Technological options
- Memory files -- repeated questions, instructions,
answer scales - Translation software support
- Parallel presentation of source and target
versions - Importing modifications to existing text
- Potential do-it-all tools
- Combine questionnaire production-documentation
with translation version production-documentation
153. Basic research
163. Basic research
- Survey translation research on
- Assessment procedures outcomes
- Translation procedures outcomes
- Answer scale translation
- Oral translation and interpreting
- Tool options
- Impact of source questions
173. Basic research
- Survey translation research on
- Assessment procedures outcomes
- Translation procedures, strategies outcomes
- Answer scale translation
- Oral translation and interpreting
- Tool options
- Impact of source questions
18Procedures, strategies, outcomes
- Team translation efforts
- Interdisciplinary expertise
- Translators, reviewers, adjudicators, consultants
- Iterative process
19Team Translation
- TRAPD model --an iterative cycle
- Translation
- Review
- Adjudication
- Pretesting and refinement
- Documentation underpins all stages
- (cf. frameworks in ESS, SHARE, and US Bureau of
Census, Westat, WMHI)
20(No Transcript)
21Review, adjudicate and document
Pretest and document, reiterate if necessary
SOURCE
22Basic Team Players
- Translators selected, competent, briefed
- Reviewers selected, competent, briefed
- Adjudicator (takes final decision) selected as
possible on basis of skills as well as seniority.
- May need to work with Consultant.
23Other Team Players
- Translators Reviewers Adjudicator
- --------------
- (Co-ordinator)
- (Substantive experts)
- (External assessors)
- Copy-editors
- (Programmers)
- Back-ups (illness, vacation, leave)
- (Oral translation and interpreting extra)
24Basic Procedures
- Translators translate
- Review session reviewers discuss and review
- each question
- Adjudicator decides/signs off
- consults with senior reviewer
- if sensible/possible also joins in review
session(s).
25Scenes from a review session
26The team
- Senior reviewer, co-adjudicator
- Translator 1
- Translator 2
- Project coordinator, survey researcher
- Pre-tester, survey researcher
- Survey researcher, lay translation talent
27Clip 1 The meeting begins
- Four in room discussing the weather
- Paul and Margrit enter
- Greetings
- Seating
- Framing the session
- Paul Ive got a parking space till 5.06 pm.
- Janet Yes, we need to finish by five.
28(No Transcript)
29Clip 2 reaching decisions
- Team has been trying at length to find a phrase
that includes atheist views on religion. - Translations proposed so far imply people do have
religious views, rather than just a view about
religion (so the clip begins)
30Clip 2 events
- Discussion ongoing
- Janet halts discussion (hand motion) and
identifies the continuing problem - Peter makes a new proposal
- Team consider it and accept
- Adjustments to other text discussed, approval
re-confirmed - The core group takes notes
31.
32.
33(No Transcript)
34What we learn from teams
- Language challenges
- Source question issues
- Strategies
- Task knowledge available and needed
35Quality improvements considerable, but
36III. Persistent problems
37III. Persistent problems
- Views on translation and what it can do
- Established common practices
- Nature of questionnaires
- Researchers and good questions
381. Views on translation what it can do
- Anyone can translate
- Focus on words
- Neglect/avoidance of adaptation
39i) Anyone can translate
- My secretary speaks Czech
- My son studies French
- Harry spent a year in Turkey
- Language ability is not a guarantee for
translation ability
40ii) Focus on words
- Have you felt blue or down recently?
41ii) Focus on words
- Have you felt blue or down recently?
- NOT a matter of colour or idiom
- Temporary state
- Degree of depressed state
- Disclosure
- Symptom relevance
42iii) Neglect/avoidance of Adaptation
43Adaptation
- Deliberate modification of a question or
questionnaire to meet new requirements - Frequently but not necessarily associated with
translation - Undertaken in source questions and/or translated
questions - Various forms (cf. Harkness 2008)
44Adaptation
- Do you have difficulty walking several blocks?
- Do you have difficulty walking 100 yards?
- Do you have difficulty walking 100 metres?
- Do you have difficulty walking 200 metres?
(Sweden)
45Adaptation and Design are related
- Measurement properties should remain
- Intended latent construct should remain
- Burden or difficulty should not change
- Relationship to other questionnaire elements
should not change - Adaptation can be anticipated in source
instruments
46Adaptation and Translation are related
- Translation involves adaptation
- (Adaptation need not involve translation)
- In the context of translation, some general types
of adaptation can be identified - Blends/entanglement of different types not unusual
472. Established practices as persistent problems
482. Established practices as persistent problems
- Back translation
- False economy
- Horse-and-cart structures in survey
implementation - Horse and Cart are essentially different and
distinct - Perceived interdependence minimal
493. Nature of questionnaires as persistent problem
503. Nature of questionnaires
- Complicated text type
- Leads a double life
- Covert measurement properties
- Surveyspeak and scalespeak
513. Nature of questionnaires
- Complicated text type
- Leads a double life
- Covert measurement properties
- Surveyspeak and scalespeak
52Surveyspeak
- Dentist When did you last visit a dentist?
- Patient About two months
- Dentist follow-up Are you sure?
- Uh-huh.
- I see.
53Surveyspeak (2)
- Interviewer When did you last visit a dentist?
- Respondent About two months ago.
Survey follow-upWould you say that you are very
certain / somewhat certain / neither certain nor
uncertain / somewhat uncertain or very uncertain
about the date you just gave? (or attempt to get
date)
53
54Scalespeak in an importance scale
- very important
- somewhat important
- neither important nor unimportant
- somewhat unimportant
- very unimportant
This is somewhat unimportant
This is very unimportant
This is somewhat important
554. Good questions (and researchers) as persistent
problems
564. Good questions (and researchers) as persistent
problems
- What makes a question good?
- Q measures what it should
- Q understood as intended
- Q salient for respondents
- Q answerable and answered
- Demonstrated quality through testing and use
57Good questions "fit" their context
- Validity and reliability are facilitated by
common ground, shared speech community usage and
social norms - Intended meaning of question and answer options
perceived meaning
58- A visual example of a good instrument
59Chinese diagnostic doll
Patient remains clothed, doll is naked
- Cultural norms on disclosure observed to enable
response
Cultural norms on sensitive topics observed
60- The diagnostic doll reflects and accommodates the
cultural embedding of the instrument (doll), the
researcher (doctor) and the respondent (patient)
61When "good" questions go travelling...
- In different contexts, good questions may be poor
cultural fits - change in "meaning"
- different conceptual coverage
- socially difficult to ask or to answer
- lose or gain saliency
62When "good" questions go travelling...
- In different contexts, good questions may be poor
cultural fits
Translation may then be an inappropriate means
to "ask the same question"
63Consequences what can go wrong .and why
64TRANSLATION
SourceCommunication about my illness at home is
poor
- Translation (Spanish)
- We do not talk much about my illness at
- home
Why? Remedy?
65PERCEPTION
SourceDo you have difficulty sitting for 2 hours?
- Translation (French)
- Do you have difficulty standing for 2 hours?
Why? Remedy?
66CULTURE or INTENDED MEANING
SourceWould you take part in a demonstration?
- Translation (German)
- Would you take part in a demonstration that
- blocks the traffic?
Why? Remedy?
67GLITCH
SourceHave you ever felt like hitting someone
- Translation (Turkish)
- Have you ever felt like shooting someone
Why? Remedy?
68Translation process highlights design issues
- Do you provide financial support for grown-up
children or grandchildren? - How many hours TV do you watch on an average
weekday? - Please give me the initials of your first and
last name - Do you prefer OTC or prescription medicines?
69Unavoidable design changes
- English
- neither satisfied nor dissatisfied
isiZulu and Hebrew neither nor ? not ...
not dissatisfied ? not satisfied
70Unavoidable design changes
- English
- neither satisfied nor dissatisfied
isiZulu and Hebrew "not satisfied and not not
satisfied" cf. Henningsson et al, 1998, Harkness
et 2005
71IV Towards solutionsconcluding remarks
72IV Towards solutions
- See the source as the source
- Source of information
- Source of challenges
- Source needs to be appraised and tested for
suitability for new contexts and languages
73IV Towards solutions
- See the target as the target
- Determine aims for target language questions
- Appraise source question goals and means for
target realization - Engage in deep processing
- Target may need degrees of freedom
74IV Towards solutions more progress
- Progress will be a process
- Base any change on evidence
- Research and documentation
- Create critical mass
- Collaboration in initiatives
- Record and share lessons learned
75Thank you
76Related Literature
Harkness, J. (2008) Comparative Survey
Research Goals and Challenges. Foundation
chapter in Dillman, D., Hox., J. and de Leeuw,
E. (eds.) International Handbook of Survey
Methodology, Hyattsville, VA Erlbaum. Harkness,
J., Schoebi, N., Joye, D., Mohler, P., Faass, T.
and Behr, D. (2007) Oral Translation in
Telephone Surveys. In J.M. Lepkowski, C.
Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J.
Lavrakas, M.W. Link and R.L. Sangster. Advances
in Telephone Survey Methodology, John Wiley and
Sons, Hoboken, New Jersey. Harkness, J. Pennell,
B.-E., Schoua-Glusberg, A. (2004) Survey
Questionnaire Translation and Assessment. In
Presser, Stanley, Rothgeb, Jennifer, Couper,
Michael, Lessler, Judith, Martin, Elizabeth, and
Singer, Eleanor (Eds.) Questionnaire Development
Evaluation and Testing Methods, Wiley Series in
Survey Methodology. New Jersey John Wiley Sons
Inc.
77Harkness, J. (2003) Questionnaire Translation.
In Harkness, Janet A., Van de Vijver, Fons J.R.,
Mohler, Peter Ph.(Hrsg.) Cross-Cultural Survey
Methods. Wiley Series in Survey Methodology. New
Jersey John Wiley and Sons Inc Harkness, J.
Van de Vijver, F. J. R. Johnson, T. P. (2003)
Questionnaire Design in Comparative Research. In
Harkness, Janet A., Van de Vijver, Fons J.R.,
Mohler, Peter Ph.(Hrsg.) Cross-Cultural Survey
Methods. Wiley Series in Survey Methodology. New
Jersey John Wiley and Sons Inc Harkness, J.
Schoua-Glusberg, A. (1998) Questionnaires in
Translation. In Harkness, J. (Hrsg.)
Cross-Cultural Survey Equivalence.
ZUMA-Nachrichten Spezial Band 3. Mannheim
Zentrum für Umfragen, Methoden und Analysen 1998,
S. 87-128