Robert Lempert Steven Popper

1
Shaping Our Long-Term Future Policy Analysis
Under Conditions ofComplexity and Deep
Uncertainty

• Robert Lempert Steven Popper
• May 29, 2002

2
Today We Face Many Long-Term Policy Challenges

• War on terrorism
• Retirement of baby boomers
• Global sustainable development
• Guiding the biotech revolution

. . .
3
National Policy Often Made With Long-Term Future
Clearly in Mind
4
As Are Many Decisions in Everyday Life
5
Ignoring Long-Term Policy Analysis, or Doing It
Poorly, May Have Severe Consequences

• Opportunities turn into problems
• Solvable problems are hurried into crises
• Festering problems become unsolvable without
  major penalties

6
New Analytic Methods Can Improve Long-Term
Policy Analysis

• Today we will discuss
• Why Long-Term Policy Analysis (LTPA) is Hard
• Four Key Principles of Successful LTPA
• Demonstrate Robust Adaptive Planning for LTPA
• Conclusions

7
What is Long-Term Policy-Making?

• When the menu of potential near-term actions is
  significantly affected by consideration of events
  that may occur many decades in the future
• The goal of LTPA is thus to
• Identify, assess, and choose among near-term
  actions that shape options available to future
  generations

8
Long-Range Policy Analysis is Hard Because,We
Cannot Accurately Predict the Future...

• For instance, even the most optimistic 1973
  forecasts of future energy use greatly
  overestimated demand

9
(Even in Fantasy the Future is Often Only a
Mirror of the Present)
Missions to Mars Tomorrowland ca 1955
10
And Because Future Generations Will MakeTheir
Own Decisions

• Recall how common values about
• race
• environment
• womens roles in the workplace
• have changed in the US over the last 50 years.
• How will values change over the next 50 to 100
  years?

11
LTPA is an Example of Decision-Making Under
Conditions of Complexity and Deep Uncertainty

• Deep uncertainty is when we do not know, and/or
  key parties to the decision do not agree on
• the system model
• prior probabilities and/or
• values
• Complexity is when the behavior of a system can
  not be simply understood from the sum of its
  parts

12
Deep Uncertainty Is Intimately Related to the
Problem of Avoiding Surprise ...

• Our stupendous unreadiness at Pearl Harborwas
  just a dramatic failure of a remarkably
  well-informed government to call the next enemy
  move It is not true that we were caught
  nappingRarely has a government been more
  expectant. We just expected wrong. We were so
  busy thinking through some obvious Japanese
  moves that we neglected to hedge against the
  choice they actually made.
• Thomas C. Schellings forward to Pearl
  Harbor Warning and Decision
• by Roberta Wohlstetter

13
... And to the Challenge of Exploiting Models of
Complex Adaptive Systems

• CAS models most useful, compared to other
  mathematical representations, when
• Information about micro behaviors has
  significant, non-obvious implications for macro
  behavior
• Emergence has potential for surprise
• Ability to adapt is key part of problem
• Under such conditions, CAS models can reflect
  unpredictability of the real world and require
  new tools for decision analysis

14
Outline

• Why Longer-Term Policy Analysis (LTPA) is Hard
• Four Key Principles of Successful LTPA
• Demonstrate Robust Adaptive Planning for LTPA
• Conclusions

15
Traditional Means of LTPA Contain Key Insights
into Addressing Challenges

• Prognostication
• Historical Analogy
• Foresight Exercises
• Scenario Planning
• Formal Decision Analysis
• Simulation modeling

16
Four Key Principles of Long-Term Policy Analysis

• Consider ensembles of large numbers of scenarios,
  which contain more information than any single
  model
• Seek robust, rather than optimal, strategies,
  which satisfice across a broad range of plausible
  scenarios and values
• Employ adaptive strategies, which evolve over
  time in response to new information to achieve
  robustness
• Use data and models to support inductive
  reasoning, helping users to interactively
  discover and test hypotheses about best
  strategies

17
Consider Ensembles of Many Scenarios

• On the occasion of the 1893
• World Columbian Exposition
• 74 experts wrote essays predicting what the US
  would look like in 1993
• Most were wrong
• Some were strangely close to truth

18
Seek Robust Strategies
A robust strategy has low regret, that is
performs well compared to the alternatives,
across many plausible futures

19
Employ Adaptivity to Achieve Robustness
20
Outline

• Why Longer-Term Policy Analysis (LTPA) is Hard
• Four Key Principles of Successful LTPA
• Demonstrate Robust Adaptive Planning for LTPA
• Conclusions

21
Todays Example The Challenge of Sustainable
Development

• What actions can we take in the near-term to help
  ensure strong economic growth and a healthy
  environment over the course of the 21st century?

• This sustainability question represents
• a well-developed debate

• with available models and data

• a category of long-range policy analysis issues
  where
• undesirable outcomes
• arise from pursuit of goals
• held valuable by individuals and societies

22
Can Compare Scenarios With Alternative Paths
Into Future
Comparative Trajectories of Global Scenario Group
250
Conventional Worlds
Great Transition
Eco-communalism
Policy Reform
Reference
Gross World Product ( trillions)
New sustainability paradigm
Fortress World
20
1990
Breakdown
Barbarization
10
5
Population (billions)
23
Consider This Set of Scenarios Tracing
Alternative Paths Into Future
Comparative Trajectories Inspired by Global
Scenario Group
Market
Great Transition
Breakdown
Gross World Product ( trillions)
Population (billions)
24
We Will Value Alternative Scenarios Using
Quasi-HDI Measure
Set Out Key Indicators With Two Regional
Definitions
North South

• Generate a family of quasi-HDI measures by
  choosing
• discount rates
• perspective years
• weights
• between factors
• between regions

Four Alternative Weightings
The shorthand
Ave. Annual ? to 2100
scorecard
25
Using Historical Analogy to Suggest Scale for
(Non-Green) Quasi-HDI
26
Define Strategies to Compare Across Scenarios
Assume near-term policy continues until changed
by future generations
Present
Future
Select near-term policy
Does the carrying capacity change?
NO
Implement policy
YES
Choose policies to maximize utility
27
Near-Term Policies That Speed Innovation by 1
Can Have Significant Effect
Near-term policy impact on Global Scenario Group
Trajectories
Market
Breakdown Averted
Great Transition
Gross World Product ( trillions)
Population (billions)
28
Robust Adaptive Planning Approach to LTPA

• Identify candidate adaptive strategies
• Compare strategies across all plausible scenarios
• Select most robust strategies given goals
• Discover and examine failure modes for candidate
• Based on analysis, consider means for hedging
• Iterate using new candidate strategy set

29
Pictures Help Tell Complex Stories to a Wide
Audience
Map Over Many Scenarios
30
Map Shows a Wide Range of Plausible Scenarios
5.0
4.0
3.0
Innovation rate
2.0
1.0
0
1.0
1.0
2.0
3.0
4.0
0
Economic growth rate
31
Speeding Innovation Performs Well in Many Futures
Using North HDI Measure...
Slight speed-up
5.0
Innovation rate
0
1.0
1.0
2.0
3.0
4.0
0
Economic growth rate
32
... But Often Fails for Global Green Measure
Slight speed-up
5.0
4.0
3.0
Market scenario
No regret Mild A lot Overwhelming
Innovation rate
2.0
1.0
0
1.0
1.0
2.0
3.0
4.0
0
Economic growth rate
33
Speeding Innovation by 1 Only HelpsAcross a
Small Range of Futures
Slight speed-up
No-action strategy
5.0
5.0
4.0
4.0
Market scenario
3.0
3.0
Market scenario
Innovation rate
2.0
2.0
1.0
1.0
0
0
1.0
1.0
1.0
2.0
3.0
4.0
0
1.0
2.0
3.0
4.0
0
Economic growth rate
Economic growth rate
34
Scenario Generator Should AddressFour Types of
Factors
X (eXogenous uncertainties)
L (policy Levers)
M (Metrics)
R (Relationships)
35
We Chose the Wonderland Model As Scenario
Generator
36
Navigating with Scenarios

• Compare the performance of alternative near-term
  strategies over a very wide range of scenarios
• Use regret to measure performance
• Regret is the difference between the performance
  of a strategy in a given scenario, given some
  measure of value, and the performance of the best
  strategy in that scenario
• We seek robust strategies
• Robust strategies have small regret over a wide
  range of plausible scenarios

37
Innovation Policy Fails When Base InnovationLags
and Environmental Stress Is High
Slight speed-up
1.1
1.0
0.9
0.8
0.7
0.6
Sustainability challenge
0.5
0.4
0.3
N
W
No regret Mild A lot Overwhelming
0.2
Market scenario
NG
WG
0.1
0
1.1
2.0
1.0
0
1.0
2.0
3.0
3.0
Innovation rate
38
Typical Failure Scenario foran Innovation Policy
Carrying capacity over time for slight speed-up
and crash effortpolicies for slight speed-up
Worst Case Scenario
2.8
2.8
2.4
2.4
2.0
2.0
Carrying capacity preserved
Carrying capacity collapses
1.6
1.6
Output per capita
1.2
1.2
0.8
0.8
N
N
W
W
0.4
0.4
S
S
0
0
2005
2020
2035
2050
2065
2080
2095
2005
2020
2035
2050
2065
2080
2095
Year
Year
Slight speed-up withGlobal Green Measure 19
Crash effort withGlobal Green Measure 5
Worst Case identified by Monte Carlo search
over 2000 futures
39
More Aggressive Innovation Acceleration Can
Impose Significant Costs in Many Futures
Crash effort
5.0
5.0
4.0
4.0
Market scenario
Market scenario
3.0
3.0
Innovation rate
2.0
2.0
1.0
1.0
0
0
1.0
1.0
1.0
2.0
3.0
4.0
0
1.0
2.0
3.0
4.0
0
Economic growth rate
Economic growth rate
No regret Mild
A lot Overwhelming
40
None of the Innovation Policies Performs
Consistently Well Across All Futures and Values

• Use computer to search for breaking scenarios
  for each strategy
• Look for highest regret for each strategy using
  all four types of values
• Every innovation strategy had breaking
  scenarios with extremely high regret

41
Start with a Milestone and Then Select the Policy
Best Suited to Achieve It
Present
Future
Select near-term milestone
NO
Does the carrying capacity change?
Determine best policy to meet milestone
YES
Implement policy
Choose policies to maximize utility
42
No Increase in Emissions Intensity
GoalPerforms Well Over Many Futures and Values
No increase
5.0
5.0
4.0
4.0
Market scenario
Market scenario
3.0
3.0
Innovation rate
2.0
2.0
1.0
1.0
0
0
1.0
1.0
1.0
2.0
3.0
4.0
0
1.0
2.0
3.0
4.0
0
Economic growth rate
Economic growth rate
No regret Mild
A lot Overwhelming
43
But Still Can Fail Catastrophically
No increase
1.1
1.0
0.9
0.8
0.7
0.6
Worst Case
Sustainability challenge
0.5
24
20
0.4
16
0.3
N
Output per capita
12
0.2
8
W
4
0.1
S
0
2005
2020
2035
2050
2065
2080
2095
0
Year
0.1
1.0
2.0
3.0
4.0
5.0
0
Innovation speed-up cost
44
Start with a Milestone, But Evaluate Progress
Early and Modify Milestone If Necessary
Present
Future
Select near-term milestone
Does the carrying capacity change?
NO
Determine best policy to meet milestone
YES
Implement policy
Is milestone achievable with current approach?
Choose policies to maximize utility
YES
NO
Relax milestone
45
Safety Valve Strategy Is Highly Robust
Safety valve
5.0
5.0
4.0
4.0
Market scenario
Market scenario
3.0
3.0
Innovation rate
2.0
2.0
1.0
1.0
0
0
1.0
1.0
1.0
2.0
3.0
4.0
0
1.0
2.0
3.0
4.0
0
Economic growth rate
Economic growth rate
No regret Mild
A lot Overwhelming
46
What About Surprises?

• Safety Valve strategy is robust against a very
  wide range of futures
• many of which we might regard as surprising
• Test strategy against discontinuous surprises
• Rapid technological advance that eliminates
  emissions
• Plague that decimates population for twenty years
• Future generations whose utility is completely
  disconnected from the environment

47
Safety Valve Still Robust with Surprises
Safety valve
5.0
No surprise
4.0
3.0
Innovation rate
2.0
1.0
0
1.0
48
These Robust Adaptive Planning MethodsHave
Addressed a Wide Variety of Problems

• Robust milestones for sustainable development
• Climate change policy
• Responding to ambiguous, adversarial warnings
• Military strategy
• Technology and product planning

49
Example Multi-Agent Model to Address Technology
Strategies for Climate Change

• Model represents
• Heterogeneous agent preferences and circumstances
  
• Learning among agents
• Increasing returns to scale

Net economic output
We judge each scenario by the present value of
aggregate output through 2050
Damages
Emissions
Economic Agents
Technology subsidies
Each agent chooses level of energy consumption
and type of energy-producing technology to
maximize its forecasted utility
Emissions permits/ Carbon taxes
50
Compare Regret of Alternative Strategies Over
Visualizations Suggested by Statistical Analysis
Getting prices Right only
51
Robust Regions DisplayHypothesis About Best
Strategies
52
Summary

• Long-Range Policy Challenges are ubiquitous and
  hard
• New analytic methods offer the potential to
  improve the practice of longer-range
  policy-making
• Quantitative tools designed to support methods
  decision-makers use in practice
• Four Key Principles of Successful Long-Range
  Policy Analysis
• Use ensembles of multiple scenarios
• Seek robust strategies
• Employ adaptivity
• Use analytics to support human hypothesis
  generation and testing

53
Backup Slides
54
Where Do We Wish to Be in 98.6 Years?
2000 HDI
2100 HDI
2100 measured by 2000 HDI
55
... But Still Fails in Futures with HighCosts
and Serious Sustainability Challenges
Safety Valve
Best Alternative (NS01)
18
18
Carrying capacity collapses
Carrying capacity preserved
16
16
W
14
14
12
12
S
10
10
Population
W
8
8
S
6
6
4
4
N
N
2
2
0
0
2005
2020
2035
2050
2065
2080
2095
2005
2020
2035
2050
2065
2080
2095
Year
Year
Global HDI 8
Global HDI 2
56
Now Were Left with Critical, IrreducibleChoices
for the Present Generation

• Safety Valve strategy is highly robust
• But it fails in scenarios where speeding
  innovation is very expensive, but crucially
  important
• More detailed model would provide more policy
  options, and the potential to craft strategy
  robust across more scenarios
• But often no strategy will be robust against all
  scenarios
• The last step in Long Term Policy Analysis is to
  characterize the irreducible choices for the
  present generation

57
Choose Safety Strategy UnlessOver 90 Sure It
Will Fail
4.5
4.0
3.5
3.0
2.5
Expected regret ()
2.0
1.5
1.0
0.5
0
0
10
20
30
40
50
60
70
80
90
100
Probability of a Safety Fails scenario ()
58
Key Components of Long-Term Policy Analysis
59
Formal Sensitivity Analysis Over Regretfor
Modified Wonderland
60
Description of Scenario Generator