The Most Significant Change Technique MSC

1
The Most Significant Change Technique (MSC)

• Dr Jessica Dart
• Clear Horizon

2
MSC

• Form of qualitative, participatory ME
• Based on stories of significant change
• Developed by Davies 1996 - Bangladesh
• Now used in numerous development programs and in
  the public sector

3
Qualitative vs quantitative monitoring

• Quantitative
• Focus on measurement
• Closed questions
• About proving
• Easy to aggregate
• Deductive
• Static
• Goal displacement can be a problem

• Qualitative
• Focus on questioning
• Open questions
• About learning
• Hard to aggregate
• Inductive
• Dynamic
• Goal displacement is not an issue

4
Limitations of indicator based monitoring

• Goal displacement
• Creaming
• Reporting bias
• Not about learning

5
Qualitative monitoring

• Can be used in conjunction with conventional
  output monitoring
• Is usually more aimed at learning than
  accountability

6
Why stories?

• People tell stories naturally - indigenous
• Stories can deal with complexity and context
• People remember stories
• Stories can carry hard messages /undiscussables
• But stories not known for accuracy/truth

7
Use of stories in MSC

• Collection of stories systematic, collective
  interpretation storytelling can be effectively
  harnessed for participatory evaluation
• Because interpretations tell another story
  process has beneficial outcomes for evaluation
  utilisation

8
Overview of MSC

• 1. Determine sorts of change to monitor
• 2. Collect stories
• 3. Review filter stories regularly
• 4. Collate selected stories for funders review
• 5. Monitor the process and verify the stories

9

• Demonstration with stories from Australian Dairy
  Farmers

10
Imagine you are a project steering committee
trying to empower Victorian dairy farmers to make
better business decisionsSelect the story that
best represents what you value
11
Overview of MSC

• 1. Determine sorts of change to monitor
• 2. Collect stories
• 3. Review filter stories regularly
• 4. Collate selected stories for funders review
• 5. Monitor the process and verify the stories

12
Example

• Target 10 Dairy Extension Project
• Four regions in Victoria, 50 staff
• 1999-2000 trail of the approach
• Still continues today

13
Step 1- Selection of domains of change

• 3 broad domains of changes to be monitored at
  the project level
• Changes in on-farm practice
• Changes in farmer-decision making skills
• Changes in profitability
• Any other type of change
• Not precisely defined

14
Step 2 - Collect stories

• During the last month, in your opinion, what do
  you think was the most significant change that
  took place in terms of on-farm practice of
  participants in the project?
• The respondent (farmer, extension worker or
  industry rep) answers in 2 parts
• 1) descriptive 2) explanatory

15
Step 3 Review filtering process

• The stories were reviewed by
• The regional committees (every 2-3 months)
• Statewide Executive (every 2-3 months)
• The stories are reviewed using a facilitated
  process at the state and funder levels

16
(No Transcript)
17
Funder meeting
State meetings
flow of stories
feedback
Region 1
Region 2 Region 3 Region 4
Story tellers
18
Example of feedback

• Not all stories concerned change
• Need better links between stories projects
• Stories written in the first person by farmers
  are powerful
• Writing style is important
• Quality of stories increased as participants
  learn

19
Step 4 - Collate review selected stories

• In total 134 stories were collected - 80 from
  extension staff
• A booklet containing 24 selected stories
  accompanied by the selection criteria comments
• Purchasers provide feedback to project after
  reviewing booklet

20
Step 5 Monitor process verify stories

• A database was developed to keep track of all
  stories
• Secondary analysis at end of reporting period
• In this case selected stories were not verified
• Storytellers were asked to check final stories in
  report

21
Impact of MSC

• Staff gained more fully shared vision
• Process boosted their morale
• Process saw farmers, staff, collaborators sitting
  together and interpreting qualitative data
  casting evaluative judgements
• Project committees became better at
  conceptualising impact

22
Use of stories

• In addition to reporting, stories were used
• To improve planning
• To help explain a point to a farmer
• To recruit new participants
• To help explain a point to another member of
  staff
• For PR

23
Why is MSC different?

• Collection coupled with systematic process of
  collective interpretation
• The interpretations themselves tell another story
  
• Process of interpretation can have beneficial
  outcomes for utilisation
• Extra step of collectively sharing and
  interpreting stories adds a new dimension

24
What MSC is NOT ABOUT

• Not only about collecting good news stories
• Not aimed at marketing /PR
• Not an easy and quick way to measure unexpected
  outcomes
• No quantifiable results
• Not recommended as a stand alone approach for
  assessing impact

25
Purpose of MSC in ME

• Primary purpose to facilitate improvement by
• focusing direction of work towards explicitly
  valued directions
• eg. what do we really want to achieve and how
  will we produce more of it?
• Contributes to summative evaluation
• Information about unexpected outcomes
• Performance information concerning very best
  success stories
• Can inform criteria used to judge projects
• Encourages stakeholder participation

26
MSC

• Creates space for stakeholders to reflect, to
  make sense of complex changes that occur as
  result of intervention
• Provides dialogue to help various actors to make
  sense of each others values
• Facilitates dynamic dialogue ie. what do we
  really want to achieve and how will we produce
  more of it?
• Particularly useful for participatory programs
  with diverse, complex outcomes, multiple
  funders/ stakeholders groups