Process Evaluation Plans for 200608 EE Programs

1
Process Evaluation Plans for 2006-08 EE Programs

• Outline for discussion with
• CPUC Energy Division Staff
• November 30, 2006

2
Tasks for All Process Evaluations

• Update program theory, document program
  activities, and compare program operation with
  theory.
• Assess and make recommendations on program
  documentation and processes needed to support
  energy savings estimation.
• Investigate barriers to program participation and
  other obstacles to program implementation.
• Evaluate how the program meets SCE portfolio
  needs.
• Recommend improvements and solicit implementer
  feedback on usefulness of the recommendations.

3
Key Issues for Process Evaluations of Major
Programs

• The following slides identify a few key issues
  that will be explored for each major program.
• We solicit Energy Divisions input on issues and
  concerns they have with individual programs or
  evaluation plans.

4
SCE 2500Appliance Recycling Program

• Issues with room air conditioner recycling
• Impact of increased rebate for freezers on the
  level of freezer recycling
• Effectiveness of co-promoting program with other
  SCE programs
• New implementation issues given two recycling
  services contractors in the program
• Tracking databases
• End-to end process
• Quality assurance, control, and monitoring
  mechanisms

5
SCE 2501Residential Energy Efficiency Program

• Effectiveness of point of sale (POS)
• Incentives design, marketing, trade allies
• Product quality
• How the new electronic rebate application is
  meeting customer needs
• Limited POS customer contact data available that
  needs to be shared between the process and impact
  studies

6
SCE 2502Multifamily EE Rebate Program

• Three-fold increase in funding level
• Getting a wide variety of measure installations
  in the programs
• A better handle on the installation practices for
  the lighting measures
• Product quality, installation locations,
  documentation
• Effectiveness of increased program exposure to
  contractor networks and multifamily trade circles
• Approaches for small, medium, and large
  properties

7
SCE 2505 California New Homes Program

• No plan available yet.
• Any issues that Energy Division or the Master
  Evaluation Contractor would like to see included?

8
SCE 2507Comprehensive HVAC Program

• Start-up issues with the major, new component
  the AC Quality Program
• procedure development, recruitment of
  Verification Service Providers (VSPs),
  development of contractor training material,
  marketing etc.
• Program dependence on the midstream market actors
  
• Consistency across processes, quality of the
  service, and level of results
• Program Tracking and Documentation
• evaluability assessment to identify gaps in
  program data tracking and documentation affecting
  impact evaluation, especially for early
  replacement component

9
SCE 2508Retro-Commissioning Program

• Hold off on a formal process evaluation until
  2008. Instead, conduct an internal assessment in
  2007, because the program manager is still making
  many enhancements to the program operations
• Other Proposed Activities
• Verification of baseline of a few early
  participants, to test program review processes.
• Measure Drop-Off Study, using 2007 program
  participants Estimate the average drop-off from
  the starting number of retro-commissioning
  recommendations (and their energy savings) to the
  subset of recommendations actually
  installed/implemented. This can be useful for
  program planning and results forecasting.
• Persistence Study Identified by various
  published reports as an area of concern. Can
  additional customer training improve it? Were
  short of funding to do a study of this.

10
SCE 2509Industrial Energy Efficiency Program

• As a new program, the Framework specifies an
  examination of the following

11
SCE 2509Industrial Energy Efficiency Program

• In addition, SCE will building on empirical work
  to date by examining
• How typical barriers to industrial energy
  efficiency programs are impacting SCEs program
• implementation uncertainty, order fulfillment,
  capital allocation criteria/strategy,
  productivity concerns
• How SCEs Project Champion structure is
  mitigating these typical barriers

12
SCE 2510Agricultural Energy Efficiency Program

• Since this regrouping and tailoring of audit and
  rebate offerings creates a largely new program,
  the process study will examine

13
SCE 2510Agricultural Energy Efficiency Program

• Furthermore, the study will build on 2003 and
  2004-2005 SCE Pump Test Evaluation results by

14
SCE 2511Nonresidential Direct Install Program

• Explore additional measures for the program
• Program is lighting-dominated other measures
  tried have had problems
• Review and recommend marketing approaches that
  could increase customer participation rates above
  30.
• Review/recommendations on inspection processes
• Review/recommendations on customer eligibility
• Review/recommendations on savings estimation
  methods, given high variability across sites
• Review/recommendations on inspection processes

15
SCE 2512 Savings By Design

• No evaluation plan yet available.
• Any issues that Energy Division or the Master
  Evaluation Contractor would like to see included?

16
SCE 2513 Education, Training Outreach

• Education, Training Outreach Program
• Seven distinct program elements
• Energy Centers (CTAC AgTAC)
• Technology and Test Center (TTC)
• Energy Design Resources (EDR)
• Non-residential Remote Energy Audits (NRREA)
• Mobile Education Unit (MEU)
• Building Operator Certification (BOC)
• Customer Language Efficiency Outreach (CLEO)

17
SCE 2513 Education, Training Outreach

• Ensure program participant tracking systems
  support indirect energy savings estimation
• Assess marketing outreach and program targeting
  strategies, marketing collateral, training
  materials curriculum design
• Evaluate effectiveness of program delivery
  channels in reaching intended targets
• Explore apply best practices in adult ed.
• Investigate barriers to participation
• Assess participant satisfaction with programs

18
SCE 2515Emerging Technologies Program

• Program Operations and Theory Alignment
  (cross-program)
• Once an ET is adopted into a resource acquisition
  program, what drives successful deployment?
• Baseline Knowledge and Awareness
• How do ETP stakeholders rate the ETP as a
  credible and timely information source (as
  compared to DOE, ACEEE)?
• Marketing Effectiveness
• Are the media and message used by ETP appropriate
  for the stakeholders? Is the website being used
  effectively to communicate assessment results?
• Stakeholder Needs Assessment
• Who are the stakeholders of ETP, and which of
  their needs is ETP fulfilling?

19
SCE 2516 Statewide Codes Standards Program

• No evaluation plan completed yet.
• Any issues that Energy Division or the Master
  Evaluation Contractor would like to see included?

20
SCE 2517Business Incentives Services Program

• Process Evaluation Focuses
• Addition of the Audit Services Program
• Encouragement of Comprehensive Projects
• Inclusion of Added Load Projects
• Integration of Demand Response Opportunities
• The Green Building Module
• Pilot Program For Joint Gas Electric Audits

21
SCE 2518 SCE 2525Local Government Partnership
Programs

• Best Practices
• A cross-program process evaluation will compare
  and contrast operations to determine best
  practices.
• Factors delineated in the 2004-2005 Local
  Government Initiative evaluation will be used to
  guide this best practices evaluation.
• Baseline Knowledge and Awareness
• How aware are local government officials and SCE
  customers of SCEs resource acquisition programs?
  How reputable are SCE and govt officials as
  sources of EE information?
• Marketing Effectiveness
• How well do officials and customers understand
  the benefits of participating in SCE resource
  acquisition programs?
• Customer Needs Assessment and Fulfillment
• Why did customers choose to participate? What
  were their expectations, based on the marketing?

22
SCE 2526 SCE 2530Institutional Partnership
Programs

• Best Practices
• Program processes and results will be compared
  and contrasted with each other as well as other
  notable institutional energy efficiency
  partnership programs in the U.S.
• Education Effectiveness
• These programs will train state agency staff to
  identify and scope retrofit and RCx projects. Can
  the training courses be improved, and can they be
  shared with CTAC and other RCx programs?
• Partner Needs Assessment and Fulfillment
• How well do all partners share responsibilities?
  Are the expectations and needs of the partners
  and program stakeholders met?
• Communications/Staffing
• The CDCR program will serve as the template for
  the State of California Partnership. Partners
  will be interviewed throughout the early period
  of the program implementation for early feedback
  on program operations.

23
SCE 2531 SCE 2548 - IDEEA SCE 2549 SCE 2552
- InDEE

• Small, innovative programs to fill gaps in
  existing program offerings and test new
  technologies, with the ultimate goal of
  mainstreaming successful efforts
• subject to the same benefit cost criteria as
  other programs
• the time frame may not be sufficient for new
  program options to be fully explored
• cannot afford many implementation errors and
  still pass the benefit-cost test.
• An evaluability assessment for each program
• Identify start-up issues
• Identify gaps in documentation, program tracking
  and information

24
SCE 2554 SCE 2556Statewide Marketing Programs

• No current plans for process evaluation
• Any issues that Energy Division or the Master
  Evaluation Contractor would like to have
  addressed?