Select%20Issues%20in%20Attribution%20and%20Net-to-Gross%20

1
Select Issues in Attribution and Net-to-Gross
Practical Examplesfor Discussion

• Presented at
• CALMAC Meetings
• July 18, 2007
• By
• Daniel M. Violette, Ph.D.
• Summit Blue Consulting
• Boulder, Colorado
• dviolette_at_summitblue.com

2
Attribution and Net Savings through
Self-reporting

• Net Program Impacts account for free ridership
  and spillover
• Self-reporting is often the least expensive
  option for estimating net program impacts, BUT
• Self-reporting may be biased (recall,
  self-selection)
• How do we incorporate and quantify qualitative
  responses?
• Financial incentives for the Program
  Administrator
• At NYSERDA, financial incentives for the Program
  Administrator are not tied to net savings
• In California, if incentives are at stake, should
  net savings estimation follow a specified
  approach that limits investigator bias?

3
Attribution and NTG Issues

• Two types of issues have been arising in our
  research
• Attribution of impacts to programs when more than
  one entity in the region is implementing a
  similar program
• Within a specific program, how to address free
  ridership and spillover (or market effects) due
  to the program.
• As more entities are becoming involved in DSM
  implementation the first type of attribution is
  becoming more important in assessing
  organization-specific cost-effectiveness.
• Also, costs for kW and kWh savings may need to
  aggregate the costs from more than one program
  effort.
• Entities implementing programs include utilities,
  federal agencies (EPA), state governments, and
  regional energy agencies such as the NW Energy
  Efficiency Alliance (NW Alliance)

4
Example from NW Alliance

• Periodic reviews of the Alliance are called for
  by the NW Alliance Board according to the
  Alliance charter to determine whether the
  Alliance benefits outweigh its costs.
• Approach
• Review of the Market Progress Evaluation Reports
  (MPERS)
• Review of the most recent Alliance Cost
  Effectiveness (ACE) model
• Determine threshold analysis what is breakeven
  for that program
• Determination of pivot assumptions
• Dimension uncertainty around these assumptions
• Simulate results under different values for the
  pivot assumptions.
• A concern was expressed by some Board members
  that the NW Alliance was claiming savings that
  were actually due to other effects or to other
  organizations.

5
Examples of Alternative Hypotheses

• Energy Star Residential Lighting (most
  controversial program)
• 1. Energy crisis of 2001 drove sales (energy
  costs and media awareness indicators)
• 2. BPA and local utility coupon program spillover
• 3. Field performance is not as anticipated
  (installation, removal rates, retention, )
• 4. CFL stocking practices were driven by other
  market factors (availability, infrastructure)
• 5. Relationship to EPA/Energy Star programs drove
  sales
• Energy Star Residential Windows
• 1. Baseline assumptions were different
  (nationally and regionally)
• 2. Builders changed installation preferences for
  other reasons than Alliance activities
• 3. Manufacturers changed processes for other
  reasons than Alliance activities
• 4. Distribution of electrically heated home is
  different than assumptions

6
Approach to Scenario Analysis

• A six-step approach characterized each program
  scenario analysis
• Step 1 Begin with the cost-effectiveness
  analyses for the four programs identified for
  detailed analysis.
• Step 2 Select pivot assumptions that influence
  cost-effectiveness.
• Step 3 Trace assumptions to MPERs or other
  reference documents.
• Step 4 Conduct interviews with other
  organizations to bracket impacts, key assumptions
  and develop scenarios.
• Step 5 Seek ranges for key values.
• Step 6 Delineate breakeven scenarios and
  distributions of economic outcomes

7
CFL Scenarios

• Estimated Value Alliance receives credit for all
  CFL sales that were not utility coupon or
  giveaway sales minus assumed baseline of 100,000
  CFL sales (this baseline comes from the ACE model
  for the CFLs project)
• Low Scenario The low scenario might assume that
  the many utilities in the region that developed
  their own CFL programs actually were the more
  important driver.
• 30 of the CFLs that the Alliance is taking
  credit for in the Estimated Value case are
  actually spillover from the utility coupon and
  giveaway programs to other sales of CFLS.
• The awareness was created by the utility
  programs and that CFLs would have been available
  in adequate supply such that the utility programs
  were a more significant driver of total CFL sales
  than is assumed in the estimated value base case.
• High Scenario The high scenario assumes that
  spillover goes in the other direction and that
  due to Alliance efforts.
• Utilities are able to sell 30 more CFLs than
  would otherwise have been the case since the
  Alliance helped set up coupon programs, the
  redemption center and encouraged retailers to
  stock CFLs.
• The end result is that without the Alliance
  efforts the utility achieved sales would have
  been 30 less

8
Scenario Implications

• (1) LOW attribution scenario (Total) -
  (Utility) - (30 spillover i.e., impact on
  non-utility sales cause by utility efforts) -
  (baseline) 2.8 million
• (2) MEDIUM -- Estimated value (Total) -
  (Utility) - (baseline) 4.0 million.
• (3) HIGH attribution scenario (Total) -
  (Utility) (30 spillover, i.e., alliance
  efforts make utility sales 30 higher than would
  otherwise have been the case) (baseline)
  5,170 thousand or roughly 5.2 million

9
What would one want to know?

• How likely is each of these scenarios to occur?
• Are scenarios other than these three as likely or
  more likely to occur?
• What is meant by low, medium and high?
• Is the low scenario the lowest conceivable value?
• Is the high the highest conceivable value?
• Just knowing these three values may not tell us
  very much and might not capture the expert
  judgment and ancillary information available very
  well
• As a result, a distribution approach was used.
  This process was used for a small set of programs
  that accounted for the vast majority of the
  savings claimed by the NW Alliance.

10
Distribution approach

• Interviews were conducted with regional experts
  familiar with the regional programs no purely
  quantitative approach seemed adequate.
• Opinions were obtained about the likelihood of
  the three scenarios.
• Example of a distribution-base analysis

11
Completing the Analysis

• Two other distributions were selected for pivot
  factors for the CFL analysis
• Number of lamps sold due to Alliance activities.
• Savings in Watts for each lamp sold (takes into
  account installation, retention, wattage, and
  other factors).
• 1. The final distribution was based on 5,000
  random draws of values from each distribution
  using _at_RISK.
• 2. For each draw (or value) from that
  distribution, the final attribution value is
  calculated for that set of values.
• 3. This provides 5,000 values which are graphed
  to give us the final distribution of impacts
  attributable based on the literature review and
  expert judgment.

12
Results Most controversial Program

• Since the evaluation team could not directly
  measure who was responsible for each CFL sale, it
  relied upon responses from interviews with
  retailers, utility program managers, and other
  knowledgeable experts.
• Taking into account of all factors, the
  cumulative savings due the from 70.4 aMW to about
  26 aMW. Due both to sales of CFL and lower
  savings per CFL.
• Other programs were much closer to initial
  Alliance estimates.
• Conclusions
• The program was cost-effective at these levels
  using mean values.
• Levelized cost was still below the cost of power
  in the region.
• Also, calculated program risk, i.e., the
  probability that the program was not
  cost-effective (a small but positive value).

13
A NYSERDA Example

• Net-to-Gross Ratios w/o billing data
• Based on work on-going at NYSERDA
• No DSM incentives are dependent upon these
  results
• Goal is to make the best decisions regarding
  program implementation and attribution of
  impacts.
• Looked at both free-ridership and three different
  types of spillover
• Internal project spillover by participants
• Participant spillover at other projects
• Non-participant spillover
• CONCEPT Prove existence first, then try to
  dimension effect and incorporate uncertainty in
  estimates.

14
Estimating a Net-to-Gross (NTG) Ratio

• Net Savings
• Gross Savings (e.g., from program data base and
  infield validation) x NTG ratio
• Summit Blues Approach
• NTG Ratio Net Factor x Market Factor
• Net Factor 1 minus Free Ridership
• Market Factor 1 Spillover(1)
  Spillover(2) Spillover(3)
• where
• Spillover(1) is Inside Spillover within Program
  projects
• Spillover(2) is Outside Spillover from Program
  participants
• Spillover(3) is Non-participant Spillover (aka
  Market Effects)

15
Estimating Free Ridership
16
Estimating Spillover

• Inside and Outside Spillover for Participants
• Is there evidence of spillover (Yes/No)?
• What share of the market does it apply to?
• Number of projects/facilities
• Relative size of projects (physical and savings)
• Attribution of savings to the program influence
• Non-Participant Spillover
• Non-participant survey
• Participating ESCOs, vendors, etc.
• Baselines and market activity
• NOTES
• 1. Constructed distributions around all
  estimates based on the dispersion of responses.
• 2. Distributions based on the assumption that if
  the population of participants responded to the
  same set of questions they would show a variance
  similar to that found in the sample.
• Use of estimated based on conservative
  assumptions, e.g., the 33rd fractal.
• Now using integrated data collection (IDC) to get
  better estimates.

17
Other Examples

• Ontario
• Shared savings incentives based on NTG
  calculations.
• Some of the programs designed by the natural gas
  utilities were picked up and operated by Natural
  Resources Canada (NR Canada)
• Attribution to the utilities made in a regulatory
  proceeding based on filed arguments.
• Other Northeast utilities also have shared
  savings and need attribution to their programs in
  a multi-entity delivery setting.
• Within NYSERDA, program-level attribution is
  difficult due to overlapping program marketing
  and measures e.g., equipment rebates and new
  construction programs.

18

• Contact Information
• Daniel Violette, Ph.D.
• Summit Blue Consulting
• 1722 14th Street 230
• Boulder, Colorado, 80302
• Phone [phone]
• dviolette_at_summitblue.com