Moving Beyond the Loading Dock: A MultiDimensional Approach to Assessment

1
Moving Beyond the Loading Dock A
Multi-Dimensional Approach to Assessment
C. Roncoli1, T. Crane1, J. Paz1, N. Breuer2, K.
Broad2, G. Hoogenboom1
1University of Georgia, 2University of Miami
7th Climate Prediction Applications Science
Workshop Norman, OK, March 24-27, 2009
2
Core Concepts

• Salience/Relevance
• Credibility/Legitimacy
• Access/Equity

3
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Evaluating the utility of tools from end-users
  perspectives
• Assessing decision support systems in terms of
  their actual uses and impacts
• Eliciting lessons learned and transferring
  results

4
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Evaluating the utility of tools from end-users
  perspectives
• Assessing decision support systems in terms of
  their actual uses and impacts
• Eliciting lessons learned and transferring
  results

5
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Evaluating the utility of tools from end-users
  perspectives
• Assessing decision support systems in terms of
  their actual use and impact
• Eliciting lessons learned and transferring
  results

6
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Assessing the utility of tools from end-users
  perspectives
• Evaluating decisions support systems in terms of
  their actual use and impact
• Eliciting lessons learned and transferring
  results

7
Understanding Decisions

• Sample 38 farmers
• Sites 21 counties in South Georgia
• Methods Semi-structured interviews
• Weather and climate information systems
• Risk management strategies
• Potential uses of climate forecasts

8
Understanding DecisionsSalience/Relevance
9
Understanding DecisionsSalience/Relevance

• Relative uncertainty of climate forecasts
  compared to non-climate drivers
• Commodity prices, input prices
• Credit, insurance, government payments
• Trade policies, labor laws, immigration
  regulations
• Inflexibility of highly-capitalized operations
• Large acreage
• Infrastructure/equipment investments
• Indebtedness

10
Understanding DecisionsCredibility/Legitimacy

• Lack of familiarity with climate forecasts
• Absence of personal experience or a track record
• Diffidence based on failure of other predictive
  information (i.e. hurricane warnings)
• Differences in farmers and scientists
  understandings of key concepts
• Accuracy
• Probability
• Normal

11
Understanding DecisionsAccess/Equity

• Farmers have limited time and mental energy to
• Process additional information
• Devise alternative strategies
• Farmers often rely on others for information
  management such as
• Wives and children
• Crop consultants
• Buyers and suppliers

12
Understanding DecisionsAccess/Equity

• Potential for actors to leverage info over
  farmers
• Lenders
• Insurers
• Brokers/buyers
• Not all producers reached by conventional
  extension services
• Limited-resource, small-scale
• Organic/sustainable
• Minority, immigrant

13
Understanding DecisionsAccess/Equity
14
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Evaluating the utility of tools from end-users
  perspectives
• Assessing decision support systems in terms of
  their actual uses and impacts
• Eliciting lessons learned and transferring
  results

15
Evaluating Tools
www.AgroClimate.org
16
Evaluating ToolsMethods

• Lab-based usability test (N4)
• IT experts review (N12)
• Evaluation surveys (N75)

• Extension agent workshops (N4)
• Ag classroom evaluations (N4)
• Feedback at outreach events

17
Evaluating ToolsMetrics
1Strongly Agree, 2Agree, 3Neither, 4Disagree,
5Strongly Disagree
18
Evaluating ToolsSalience/Relevance

• More crops
• Multiple varieties
• Multiple grasses
• More cotton tools
• Hurricane landfalls
• First and last freezes
• Humidity levels
• Soil moisture
• Pests
• Competing areas forecasts
• Climate change

19
Evaluating ToolsCredibility/Legitimacy

• Show the people behind the website
• Provide personal contact info for scientists
• Make institutions logos prominent
• Highlight public nature of website (.edu, .gov)
• Explain where data comes from
• Link to trusted sources of information
• Add editorials by county agents, testimonials
• by farmers

20
Evaluating ToolsCredibility/Legitimacy

• Foster habitual use and experiential learning
• Provide track record to assess accuracy
• Provide Outlooks more frequently
• Update information regularly
• Keep things consistent across site
• Improve user support, tutorials, manual
• Provide saving and customizing options
• Create recognizable identity for site

21
Evaluating ToolsAccess/Equity

• Show you grasp what it means to be a farmer
• Show that farmers are primary audience
• Write in language farmers can relate to
• Make more user-friendly, info easier to find
• Provide options for slow connections,
• different computer skill levels
• Layer information for different users

22
Evaluating Tools
2008
2005
2007
23
Evaluating Tools
24
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Evaluating the utility of tools from end-users
  perspectives
• Assessing decision support systems in terms of
  their actual uses and impacts
• Eliciting lessons learned and transferring
  results

25
Assessing Impacts
Effectiveness La Niña forecast ? change in
peanut varieties from long-cycle to
short-cycle, Irwin County
26
Types of Assessment

• Identifying users, understanding their decision
  processes, and the role of climate forecasts
• Assessing the utility of tools from end-users
  perspectives
• Evaluating decision support systems in terms of
  their actual use and impact
• Eliciting lessons learned and transferring
  results

27
Transferring Results
SECC CLIMAS Collaboration
AgroClimate for the Southwest US
28
Conclusions

• Salience/Relevance What constitutes success?
• For the SECC
• For assessment
• Credibility/Legitimacy Who are we accountable
  to?
• End users
• Funding agencies
• Universities, scientific community
• Access/Equity Who are our clients?
• Agricultural extension
• Agricultural producers (which ones)
• Any actor in the agricultural sector?

29
Conclusions

• Meteorologist is the only occupation where you
  can be wrong every day and still have a job
• Farmer, Bullock County, GA