Assessing Impact: Study Abroad at Two Institutions

1
Assessing Impact Study Abroad at Two Institutions

• Elaine Meyer-Lee
• Saint Marys College
• Carol Warfield
• Auburn University
• 2006 AIEA Annual Conference
• February, 2006

2
Why Assess Outcomes?

• 1,022 US institutions reported information on
  their study abroad programs
• 174,629 U.S. students studied abroad
• Data from Open Doors Report on International
  Educational Exchange 2003-2004

3

• Inform continuous quality improvement
• Advocate for international education
• Satisfy regional accrediting bodies
• Add to basic understanding of student growth in
  this area

4

• Young, wide-open field
• Much need for better methods and more data on
  many different dimensions
• This presentation is not a step-by-step overview,
  but two examples
• (See Forums upcoming Guide to Outcomes
  Assessment in Education Abroad)

5
Multi-method Longitudinal Assessment of
Intercultural Engagement, IC Sensitivity,
Identity, and Goals Preliminary Results

• Elaine Meyer-Lee
• Director of International Intercultural
  Learning
• Joy Evans
• Research Analyst-Center for Womens InterCultural
  Leadership
• Saint Marys College
• Notre Dame, Indiana
• 206 AIEA Annual Conference
• February 25, 2006

6
Presentation Overview

• Context
• Assessment Methods
• Review of Instruments
• Preliminary Results
• Contributions to Study Abroad Assessment

7
Saint Marys College

• Catholic Womens Liberal Art
• Established 1844
• About 1600 enrollment
• 98 full-time
• 82 on-campus
• 85 Catholic
• 9 students of color
• 20 of students study abroad

8
Mission of the Center for Womens InterCultural
Leadership

• Saint Mary's College's mission includes
• preparing women to make a difference
• in an increasingly complex and
• interdependent world. The role of the
• Center for Women's InterCultural
• Leadership (CWIL) within this mission
• is to foster the intercultural knowledge
• and competence critical to educating
• the next generation of women leaders.

9
CWIL at Saint Marys

• Encompasses much more than study abroad (see
  brochures)
• Is adopting Byrams definition of intercultural
  competence
• Is launching a portfolio-based certificate
  program in Womens Intercultural Leadership

10
Study Abroad Programs

• Year/ Semester
• Ireland (Year only)
• India (Semester only)
• Rome
• Spain
• France
• Australia
• Austria
• South Africa

11
Study Abroad Programs

• 4-2 Weeks
• European Studies (4)
• Internships in Europe (4)
• Honduras (3)
• Ecuador (3-2)
• Greece (2 ½)
• S. Korea (2)
• 1 Week
• Prague/Poland
• Jamaica
• Mexico

12
Assessing Study Abroad Going Beyond Student
Satisfaction
13
Evaluation Assessment Goals

• Evaluate programs effectiveness
• Measure Students
• Changes in sensitivity to cultural differences
• American identity development
• Own sense of growth toward goals
• Explore correlations of change with
• Intercultural engagement while abroad
  (interaction and reflection)
• Program or demographic characteristics

14
Assessment Methods/Design

• Longitudinal Research Design
• Pre-test/Post-test (some grad follow-up)
• Procedures 70 response rate so far
• Control Group
• 30 First-year students
• Focus Group
• Study Abroad Re-entry Course
• Written reflection and Kolbs LSI

15
Assessment Instruments Pre-Test

• Control Group
• Intercultural Development Inventory (IDI)
• American Identity Questionnaire

• Study Abroad Participants
• Intercultural Development Inventory (IDI)
• American Identity Questionnaire
• Goals/motivations from Saint Marys College Study
  Abroad Survey

16
Assessment Instruments Post-Test

• Study Abroad Participants
• Intercultural Development Inventory (IDI)
• American Identity Questionnaire (AIQ)
• SMC Study Abroad Survey Program Impact Measure
  of Intercultural Engagement (MIE)

• Control Group
• Intercultural Development Inventory (IDI)
• American Identity Questionnaire

17
Data Collection Timeline
18
Review of Instruments
19
Intercultural Development Inventory (IDI)

• 50-item Self-Assessment
• Valid Reliable Psychometric Measure
• Culture general in focus
• Quantifies the subjective experience of cultural
  difference
• Most widely used currently
• Based on Milton Bennetts Developmental Model of
  Intercultural Sensitivity (constructivist stage
  model)

20
Slide Available Onlinehttp//www.coe.int/T/E/Nort
h-South_Centre/Programmes/3_Global_Education/c_GEN
E/GENE20Web-page20-20(3)20Hesse20article.asp
21
IDI Measurement Scales

• Denial/Defense
• Reversal
• Minimization
• Acceptance/Adaptation
• Encapsulated Marginality

22
Overall Profile
23
American Identity Questionnaire

• 10-item Likert-style Self Assessment
• Adaptation of Jean Phinneys Multigroup Ethnic
  Identity Measure (good track record)
• Based on Eriksonian identity development theory.
  Measures 2 Factors
• identity search (e.g. I have often talked to
  other people about what it means to be an
  American.)
• affirmation, belonging, and commitment (e.g.
  Being American plays an important part in my
  life.)
• One goal of study abroad is more complexity in
  understanding own culture, and students in pilot
  focus group felt it captured an important
  dimension
• Currently refining adaptations psychometric
  properties

24
Saint Marys College Study Abroad Survey

• Semester/Year-Long Programs
• Core Survey
• 111 Items
• 4 open-ended questions
• Customized Addendum

• Summer/Short-term Programs
• Core Survey
• 58 Items
• 1 open-ended questions
• Customized Addendum

25
Saint Marys College Study Abroad Survey

• Semester/Year-Long Programs
• Question Sections Include
• Demographics/Background
• Goals
• Pre-Departure
• Academic Program
• Cultural Tours Excursions
• On-Site Student Services
• Intercultural Engagement
• Overview of Program
• Program Impact (Goals)
• Volunteer Interests

• Summer/Short-term Programs
• Question Sections Include
• Demographics
• Pre-Departure
• Living Arrangements
• Intercultural Engagement
• Overview of Program
• Volunteer Interests

26
Self-reported Growth in

• Understanding of different cultures
• Different perspective on American culture
• Greater understanding of the US in world affairs
  and history
• Different perspective on your ethnic
  heritage
• Understanding of my faith
• Self-awareness
• Maturity and self-confidence
• Interpersonal skills and the ability to
  adapt
• Intellectual development in the liberal
  arts
• Ability to communicate in a foreign language

27
Measure of Intercultural Engagement

• Asks students to quantify the frequency of their
  engagement through
• Interaction with cultural resources
• Interaction with local people
• Explicit reflection on these interactions

28
Interaction with cultural resources

• Theater/opera/symphony/concerts
• Museums/ historical sites
• Forests/parks/beaches
• Traveling outside of the host country
• Traveling within the host country

29
Interaction with local people

• Shopping/visiting local markets
• Movies/entertainment catering to locals
• Restaurants/pubs/cafes catering to locals
• Visiting local families/friends in their homes
• Participating in worship services
• On-campus informal conversations
• Events sponsored by the study abroad program
• Extra-curricular activities (student clubs,
  intramurals, etc.)
• Volunteering, service-learning, internships,
  shadowing
• Reading or listening to local news

30
Explicit reflection on these interactions

• Journaling/Blogging
• Photography
• Other artistic expression (painting, drawing,
  collages, etc.)
• Class discussions
• Informal conversations with locals
• Informal conversations with students studying
  abroad
• Calling/e-mailing family/friends/faculty in
  America about experience

31
Pre-test Results IDI Worldview Profile
Pre-Abroad
32
IDI Worldview Profile Pre Controls
33
Pre-test Comparison to Control American
Identity Questionnaire

• Affirmation, belonging and commitment
• ? no significant difference
• Identity search
• ? statistical significance (at the .05
  level)

34
Summary of Pre-test Findings

• IDI Students scores do not support
  developmental model
• IDI Low Minimization Score
• Catholic beliefs .Were all Gods children
• Control on both Heritage Seeker Factor
• Are study abroad participants a different
  population than other SMC students?

35
Preliminary Analysis of Change Data for Matched
Set

• No significant change in means for IDI or AIQ yet
  (on paired T-test on pre-post mean scores, n42
  and 27)
• BUT, high self-reported growth (1.7 when 1 is
  high) and wide range in change scores and program
  means

36
What about those who do change?

• Correlation of AIQ with IDI scales (bivariate
  non-parametric correlations)
• Correlations with Intercultural Engagement
• Cultural resource scale with AIQ change
• Overall and reflection with program impact
  (especially interpersonal items)
• 50 did grow on at least one scale (defined as 2
  standard deviations of positive change, n21)

37
How do the growers compare?

• Characteristic Growers Non-Growers
• From N.America 91 95
• White 67 86
• Latina 24 4.5
• Never been abroad 53 63
• Abroad lt 3 months 24 26
• Whole year program 10 46
• Semester program 76 46
• Summer program 14 8
• Rome program 24 32
• Seville program 24 5
• Mexico 14 9
• France 14 9
• Ireland 10 46

38
Methodological Findings

• Importance and Challenges of Control Group
• Procedural effects (e.g. on-line vs. paper)
• Signs of internal consistency in measures
  (inter-item correlations)

39
Benefits to Study Abroad Assessment
Challenges to Institutional Research

• Intercultural Development Inventory
• Standardized instrument
• Ability to compare study abroad program outcomes
  with domestic intercultural programs

• Intercultural Development Inventory
• Requires qualifying training to administer
• Highly rigorous and proprietary
• Labor- and cost-intensive for large scale use

40
Benefits to Study Abroad Assessment
Challenges to Institutional Research

• Program Impact MEIM
• Comparable program specific data on students
  intercultural interactions
• AIQ
• Examine identity dimension, connect to literature
  on this

• Program Impact MEIM
• Labor-intensive development and administration
• AIQ
• Reliability and Validity of adaptation
• Needs refining

41
Next Steps

• Analysis
• More on change and changers with full dataset
• More on correlations with intercultural
  engagement, program variables (including Engle
  Engles category), and student variables
  (including original motivation)
• Qualitative
• Psychometrics on AIQ, Program Impact, MIE
• Data collection
• Add 1 year follow-up
• Draw on Womens Intercultural Leadership
  Portfolios?
• Applications
• Many improvements to (and rationales for)
  programs already. Overhauling re-entry course.

42

• For more handouts or bibliography on assessing
  intercultural outcomes of study abroad programs,
  go to
• http//www.saintmarys.edu/cwil/php/intercultural
  .learning/IILOutcomes.php
• or email me meyerlee_at_saintmarys.edu