Crosscultural surveys and translation - PowerPoint PPT Presentation

1 / 77
About This Presentation
Title:

Crosscultural surveys and translation

Description:

SURVEY MEASUREMENT: ASSESSING THE RELIABILITY AND VALIDITY OF ... Translators: selected, competent, briefed. Reviewers: selected, competent, briefed ... – PowerPoint PPT presentation

Number of Views:189
Avg rating:3.0/5.0
Slides: 78
Provided by: janetha
Category:

less

Transcript and Presenter's Notes

Title: Crosscultural surveys and translation


1
Cross-cultural surveys and translation
  • ESRC Question Bank Conference
  • SURVEY MEASUREMENT ASSESSING THE RELIABILITY AND
    VALIDITY OF QUESTIONNAIRE ITEMS
  • April 10
  • Royal Statistical Society
  • Janet A Harkness

gesis-ZUMA
2
Outline
  • Framework
  • Progress in survey translation production and
    assessment
  • Persistent problems consequences
  • Towards solutions

3
I. Framework
  • Survey translation uses
  • Expectations
  • Current practice

4
1. Survey translations uses
  • Within country research
  • Groups may partially share larger context
  • Shared contexts, degree of interaction, immigrant
    acculturation affect translations required
  • Across country research
  • Different contexts and languages

5
2. Expectations for survey translations
  • Assumption is a good translation
  • asks the same question
  • maintains semantic pragmatic meaning
  • maintains measurement properties
  • retains source design features
  • satisfies multiple other requirements (askable
    and answerable, burden, saliency, etc)

6
3. Common current practice
  • Depending on discipline
  • Do not assess translation process quality
  • Assess translation quality through back
    translation
  • Pretest translated questions
  • Assess translated instrument quality on basis of
    statistical analysis (dif, IRT)

7
II. Progress in survey translation production and
assessment
8
Progress in survey translation production and
assessment
  • Guidelines and know-how
  • Tools
  • Basic research
  • Procedures, strategies, outcomes

9
Progress in survey translation production and
assessment
  • Guidelines and know-how
  • Tools
  • Basic research
  • Procedures and strategies

10
1. Guidelines and Know-How
11
1. Guidelines and Know-How
  • ESS, USA Census Bureau, International Test
    Commission, QoL research and publications
  • Comprehensive sets of guidelines CSDI Workshop
    Guidelines Initiative (lead groups at ISR,
    Michigan, UNL and gesis-ZUMA)

12
2. Tools
13
2. Tools
  • Process documents
  • Note-taking templates
  • Queries, decisions, rationales
  • Harmonization templates
  • Decisions, rationales
  • Version records
  • Decisions, rationales

14
2. Tools
  • Technological options
  • Memory files -- repeated questions, instructions,
    answer scales
  • Translation software support
  • Parallel presentation of source and target
    versions
  • Importing modifications to existing text
  • Potential do-it-all tools
  • Combine questionnaire production-documentation
    with translation version production-documentation

15
3. Basic research
16
3. Basic research
  • Survey translation research on
  • Assessment procedures outcomes
  • Translation procedures outcomes
  • Answer scale translation
  • Oral translation and interpreting
  • Tool options
  • Impact of source questions

17
3. Basic research
  • Survey translation research on
  • Assessment procedures outcomes
  • Translation procedures, strategies outcomes
  • Answer scale translation
  • Oral translation and interpreting
  • Tool options
  • Impact of source questions

18
Procedures, strategies, outcomes
  • Team translation efforts
  • Interdisciplinary expertise
  • Translators, reviewers, adjudicators, consultants
  • Iterative process

19
Team Translation
  • TRAPD model --an iterative cycle
  • Translation
  • Review
  • Adjudication
  • Pretesting and refinement
  • Documentation underpins all stages
  • (cf. frameworks in ESS, SHARE, and US Bureau of
    Census, Westat, WMHI)

20
(No Transcript)
21
Review, adjudicate and document
Pretest and document, reiterate if necessary
SOURCE
22
Basic Team Players
  • Translators selected, competent, briefed
  • Reviewers selected, competent, briefed
  • Adjudicator (takes final decision) selected as
    possible on basis of skills as well as seniority.
  • May need to work with Consultant.

23
Other Team Players
  • Translators Reviewers Adjudicator
  • --------------
  • (Co-ordinator)
  • (Substantive experts)
  • (External assessors)
  • Copy-editors
  • (Programmers)
  • Back-ups (illness, vacation, leave)
  • (Oral translation and interpreting extra)

24
Basic Procedures
  • Translators translate
  • Review session reviewers discuss and review
  • each question
  • Adjudicator decides/signs off
  • consults with senior reviewer
  • if sensible/possible also joins in review
    session(s).

25
Scenes from a review session
26
The team
  • Senior reviewer, co-adjudicator
  • Translator 1
  • Translator 2
  • Project coordinator, survey researcher
  • Pre-tester, survey researcher
  • Survey researcher, lay translation talent

27
Clip 1 The meeting begins
  • Four in room discussing the weather
  • Paul and Margrit enter
  • Greetings
  • Seating
  • Framing the session
  • Paul Ive got a parking space till 5.06 pm.
  • Janet Yes, we need to finish by five.

28
(No Transcript)
29
Clip 2 reaching decisions
  • Team has been trying at length to find a phrase
    that includes atheist views on religion.
  • Translations proposed so far imply people do have
    religious views, rather than just a view about
    religion (so the clip begins)

30
Clip 2 events
  • Discussion ongoing
  • Janet halts discussion (hand motion) and
    identifies the continuing problem
  • Peter makes a new proposal
  • Team consider it and accept
  • Adjustments to other text discussed, approval
    re-confirmed
  • The core group takes notes

31
.
32
.
33
(No Transcript)
34
What we learn from teams
  • Language challenges
  • Source question issues
  • Strategies
  • Task knowledge available and needed

35
Quality improvements considerable, but
36
III. Persistent problems
37
III. Persistent problems
  • Views on translation and what it can do
  • Established common practices
  • Nature of questionnaires
  • Researchers and good questions

38
1. Views on translation what it can do
  • Anyone can translate
  • Focus on words
  • Neglect/avoidance of adaptation

39
i) Anyone can translate
  • My secretary speaks Czech
  • My son studies French
  • Harry spent a year in Turkey
  • Language ability is not a guarantee for
    translation ability

40
ii) Focus on words
  • Have you felt blue or down recently?

41
ii) Focus on words
  • Have you felt blue or down recently?
  • NOT a matter of colour or idiom
  • Temporary state
  • Degree of depressed state
  • Disclosure
  • Symptom relevance

42
iii) Neglect/avoidance of Adaptation
43
Adaptation
  • Deliberate modification of a question or
    questionnaire to meet new requirements
  • Frequently but not necessarily associated with
    translation
  • Undertaken in source questions and/or translated
    questions
  • Various forms (cf. Harkness 2008)

44
Adaptation
  • Do you have difficulty walking several blocks?
  • Do you have difficulty walking 100 yards?
  • Do you have difficulty walking 100 metres?
  • Do you have difficulty walking 200 metres?
    (Sweden)

45
Adaptation and Design are related
  • Measurement properties should remain
  • Intended latent construct should remain
  • Burden or difficulty should not change
  • Relationship to other questionnaire elements
    should not change
  • Adaptation can be anticipated in source
    instruments

46
Adaptation and Translation are related
  • Translation involves adaptation
  • (Adaptation need not involve translation)
  • In the context of translation, some general types
    of adaptation can be identified
  • Blends/entanglement of different types not unusual

47
2. Established practices as persistent problems
48
2. Established practices as persistent problems
  • Back translation
  • False economy
  • Horse-and-cart structures in survey
    implementation
  • Horse and Cart are essentially different and
    distinct
  • Perceived interdependence minimal

49
3. Nature of questionnaires as persistent problem
50
3. Nature of questionnaires
  • Complicated text type
  • Leads a double life
  • Covert measurement properties
  • Surveyspeak and scalespeak

51
3. Nature of questionnaires
  • Complicated text type
  • Leads a double life
  • Covert measurement properties
  • Surveyspeak and scalespeak

52
Surveyspeak
  • Dentist When did you last visit a dentist?
  • Patient About two months
  • Dentist follow-up Are you sure?
  • Uh-huh.
  • I see.

53
Surveyspeak (2)
  • Interviewer When did you last visit a dentist?
  • Respondent About two months ago.

Survey follow-upWould you say that you are very
certain / somewhat certain / neither certain nor
uncertain / somewhat uncertain or very uncertain
about the date you just gave? (or attempt to get
date)
53
54
Scalespeak in an importance scale
  • very important
  • somewhat important
  • neither important nor unimportant
  • somewhat unimportant
  • very unimportant

This is somewhat unimportant
This is very unimportant
This is somewhat important
55
4. Good questions (and researchers) as persistent
problems
56
4. Good questions (and researchers) as persistent
problems
  • What makes a question good?
  • Q measures what it should
  • Q understood as intended
  • Q salient for respondents
  • Q answerable and answered
  • Demonstrated quality through testing and use

57
Good questions "fit" their context
  • Validity and reliability are facilitated by
    common ground, shared speech community usage and
    social norms
  • Intended meaning of question and answer options
    perceived meaning

58
  • A visual example of a good instrument

59
Chinese diagnostic doll
Patient remains clothed, doll is naked
  • Cultural norms on disclosure observed to enable
    response

Cultural norms on sensitive topics observed
60
  • The diagnostic doll reflects and accommodates the
    cultural embedding of the instrument (doll), the
    researcher (doctor) and the respondent (patient)

61
When "good" questions go travelling...
  • In different contexts, good questions may be poor
    cultural fits
  • change in "meaning"
  • different conceptual coverage
  • socially difficult to ask or to answer
  • lose or gain saliency

62
When "good" questions go travelling...
  • In different contexts, good questions may be poor
    cultural fits

Translation may then be an inappropriate means
to "ask the same question"
63
Consequences what can go wrong .and why
64
TRANSLATION
SourceCommunication about my illness at home is
poor
  • Translation (Spanish)
  • We do not talk much about my illness at
  • home

Why? Remedy?
65
PERCEPTION
SourceDo you have difficulty sitting for 2 hours?
  • Translation (French)
  • Do you have difficulty standing for 2 hours?

Why? Remedy?
66
CULTURE or INTENDED MEANING
SourceWould you take part in a demonstration?
  • Translation (German)
  • Would you take part in a demonstration that
  • blocks the traffic?

Why? Remedy?
67
GLITCH
SourceHave you ever felt like hitting someone
  • Translation (Turkish)
  • Have you ever felt like shooting someone

Why? Remedy?
68
Translation process highlights design issues
  • Do you provide financial support for grown-up
    children or grandchildren?
  • How many hours TV do you watch on an average
    weekday?
  • Please give me the initials of your first and
    last name
  • Do you prefer OTC or prescription medicines?

69
Unavoidable design changes
  • English
  • neither satisfied nor dissatisfied

isiZulu and Hebrew neither nor ? not ...
not dissatisfied ? not satisfied
70
Unavoidable design changes
  • English
  • neither satisfied nor dissatisfied

isiZulu and Hebrew "not satisfied and not not
satisfied" cf. Henningsson et al, 1998, Harkness
et 2005
71
IV Towards solutionsconcluding remarks
72
IV Towards solutions
  • See the source as the source
  • Source of information
  • Source of challenges
  • Source needs to be appraised and tested for
    suitability for new contexts and languages

73
IV Towards solutions
  • See the target as the target
  • Determine aims for target language questions
  • Appraise source question goals and means for
    target realization
  • Engage in deep processing
  • Target may need degrees of freedom

74
IV Towards solutions more progress
  • Progress will be a process
  • Base any change on evidence
  • Research and documentation
  • Create critical mass
  • Collaboration in initiatives
  • Record and share lessons learned

75
Thank you
76
Related Literature
Harkness, J. (2008) Comparative Survey
Research Goals and Challenges. Foundation
chapter in Dillman, D., Hox., J. and de Leeuw,
E. (eds.) International Handbook of Survey
Methodology, Hyattsville, VA Erlbaum. Harkness,
J., Schoebi, N., Joye, D., Mohler, P., Faass, T.
and Behr, D. (2007) Oral Translation in
Telephone Surveys. In J.M. Lepkowski, C.
Tucker, J.M. Brick, E. de Leeuw, L. Japec, P.J.
Lavrakas, M.W. Link and R.L. Sangster. Advances
in Telephone Survey Methodology, John Wiley and
Sons, Hoboken, New Jersey. Harkness, J. Pennell,
B.-E., Schoua-Glusberg, A. (2004) Survey
Questionnaire Translation and Assessment. In
Presser, Stanley, Rothgeb, Jennifer, Couper,
Michael, Lessler, Judith, Martin, Elizabeth, and
Singer, Eleanor (Eds.) Questionnaire Development
Evaluation and Testing Methods, Wiley Series in
Survey Methodology. New Jersey John Wiley Sons
Inc.
77
Harkness, J. (2003) Questionnaire Translation.
In Harkness, Janet A., Van de Vijver, Fons J.R.,
Mohler, Peter Ph.(Hrsg.) Cross-Cultural Survey
Methods. Wiley Series in Survey Methodology. New
Jersey John Wiley and Sons Inc Harkness, J.
Van de Vijver, F. J. R. Johnson, T. P. (2003)
Questionnaire Design in Comparative Research. In
Harkness, Janet A., Van de Vijver, Fons J.R.,
Mohler, Peter Ph.(Hrsg.) Cross-Cultural Survey
Methods. Wiley Series in Survey Methodology. New
Jersey John Wiley and Sons Inc Harkness, J.
Schoua-Glusberg, A. (1998) Questionnaires in
Translation. In Harkness, J. (Hrsg.)
Cross-Cultural Survey Equivalence.
ZUMA-Nachrichten Spezial Band 3. Mannheim
Zentrum für Umfragen, Methoden und Analysen 1998,
S. 87-128
Write a Comment
User Comments (0)
About PowerShow.com