Net-to-Gross: A Few Observations Vis-

1
Net-to-Gross A Few Observations Vis-á-Vis the
Long Term

• CALMAC
• San Francisco, California
• July 17, 2007

Michael RufoItron Inc. 1111 Broadway, Suite
1800 Oakland, CA 94607 510-844-2881
2
Theory

• Love em, hate em, but
• Free ridership and spillover/MT
• Important concepts, reasonably well understood,
  important to program design, forecasting/procureme
  nt, policy making
• But terms and measurements dont always capture
  and convey what we want
• Free ridership and spillover/MT
• Two sides of the same coin
• Its short-sighted to focus on one while ignoring
  the other
• What matters is long term market change
• Totality of Direct program participation,
  market effects, codes/standards, and other
  influences
• Free ridership is limited as a short-term,
  participant-driven metric

3
Why Worry About Free Ridership?

• Non-participants perspective
• Dont give my money to someone else to do
  something they were doing anyway
• Program efficacy
• Competition for scarce public purpose dollars
• Concerns cant be cavalierly dismissed
• and someone will always raise them!
• But
• when programs are effective over the long-term
• and participation is widespread
• concerns can be mitigated, if not eliminated

4
Program Effects are Often Acceleration (RER 2001)
5
Practice

• Butthere are problems, mountains of them
• Adoption is non-linear and so is free ridership
• Markets are dynamic and when interventions
  succeed change is accelerated
• Line AD in previous slide is not what we observe
• What is a true naturally occurring baseline 25
  years after the first market interventions?
• Todays free riders are often yesterdays market
  effects
• Measurement techniques are limited
• No pure control groups
• Whatever happened to experimental design!
• The vagaries of self reports
• The pain of market tracking

6
Practice

• Measurement challenges partially related to under
  and inconsistent investment
• 1-2B/yr over past 15 years on programs
• How much on evaluation and longitudinal
  baselines?
• Evaluation efforts spotty and often half hearted
• Real-learning the same inconclusive stuff over
  and over instead of conducting rigorous,
  consistent, and, yes, sometimes more costly,
  long-term research
• No surprise we have very few reliable
  longitudinal data sets of market saturation,
  penetration, costs
• Nationally, reported program data is weak!
• Doesnt support econometric analysis

7
NTG Application to Implementers

• Extreme cases
• Linear scaling of reward/penalty, threshold
  triggers
• No financial feedback
• Nationally, some fallout from the1990s?
• Fear and stipulation
• Lets call the whole thing off
• Maybe if we dont measure it it will go away
• But ex post NTG can provide vital feedback
• Critical to improving program design
• In some cases, partially reward/penalize
  depending on what administrators can
  realistically control
• Not a substitute for multi-year market effects
  analyses

8
NTG Under Aggressive Program Funding

• Over the long term
• EE program essentially savings and investment
  fund
• Customers are really using their own money
• To enhance buying power
• Stimulate new markets
• Mitigate market barriers
• Explicit in Use it or loss it approach for
  large CI
• Short-term free ridership becomes less of a
  concern
• If long-term participation is widespread
• Significant market effects are accomplished
• Programs/policies adapt quickly to
  accomplishments/failures
• Latter requires continuous measurement of both
  short-term program impacts and long-term market
  effects

9
Conclusions

• Not measuring should not be an option
• But traditional focus on free riders is
  suboptimal and can lead to wrong conclusions
• Free rider term itself problematic/derogatory
• Not consistent with traditional economic use of
  term
• What matters are
• Short-term marginal program effectiveness (MPE)
  
• Lets find a better term that gets at what we
  care about
• Long-term market effects
• And, yes, associated program attribution.

10
Conclusions

• Attribution is obviously challenging
• Precise attribution will always be difficult
• Chunky attribution less so
• Results should be used to
• Optimize program and portfolio design
• Know when to
• declare victory, admit failure, redesign, or move
  on
• Appropriately direct and motivate implementers,
  w/out
• Penalizing for factors outside direct control
• Creating perverse incentives
• (e.g., maximizing short-term versus long-term
  impacts)
• Improve forecasts and influence on procurement

11
Wrap
We should not let short-term, and sometimes
controversial applications of NTG, overly
influence research and evaluation planning
objectives which should be driven by both short-
and long-term perspectives on measuring changes
in total societal energy efficiency (and energy
consumption) in ways that are robust in the face
of changing program strategies and policy
regimesNor should we let measurement challenges
lead to measurement avoidance.