Evaluation, Measurement and Verification (EM

1
Evaluation, Measurement and Verification (EMV)
of Energy Efficiency Programs???????,?????
(EMV)

• Steve Kromer, P.E.
• Consultant /EVO Board Member
• ??/EVO?????
• Beijing, China ????
• February 26-27, 2009

2
EMV Topics??

• ??
• ???????
• ??????????
• ??,?????
• ???????

• Introduction
• Evaluation of Energy Efficiency Programs
• Measurement and Verification of Energy Efficiency
  Projects
• Procedures, Principals and Methods
• Evaluation Management and Planning

3
1. Introduction
1. ??

• ??????????
• ????????????
• ????????
• ????????

• Why is evaluation important?
• Evaluation as part of the Program Cycle
• Programs versus projects
• Institutions Required to Perform Evaluation

4
Why Evaluate Efficiency Programs?????????

• Things that are measured tend to improve
• ???????????
• Economist John Kenneth Galbraith
• ??????.???.???

5
How do we increase performance?????????

• Athletes ???
• Set goals ????
• Develop training plans ??????
• Track progress ????
• Compete! ??

6
Example Cycling Training Program??-???????
Heart rate ??
Power ??
7
Energy Investment Decisions ??????Investment in
Supply Side ?????
Investment ??

• What Information is needed to support Supply-Side
  
• ??????????
• Investment decisions ? ?????
• Implementation ? ???
• Settlement ? ???

Supply-Side ???
Demand-Side ???
Generation ?? Transmission ?? Distribution
??
Efficiency ?? Demand Response ????
8
Efficiency Investment Decisions ??????
Information Needs ????
Investment ??

• What Information is needed to support Demand-Side
  
• ??????????
• Investment decisions ? ?????
• Implementation ? ???
• Settlement ? ???

• Lighting ??
• Boilers ??
• Plant Improvements
• ????
• Controls ??
• High Efficiency Motors
• ????

Demand-Side ???
Supply-Side ???
Generation ?? Transmission ?? Distribution
??
Efficiency ?? Demand Response ????
9
Balancing Investment in Supply and
Demand????????????
Investment ??

• Lighting ??
• Boilers ??
• Plant Improvements ????
• Controls ??
• High Efficiency Motors ????

Demand-Side ???
Supply-Side ???
Generation ?? Transmission ?? Distribution
??
Efficiency ?? Demand Response ????
10
EXAMPLE California 2009-2011 Energy Efficiency
Programs??????? 2009-2011????
Utility Investment ?????? 4 Billion US 40???

• Lighting ??
• Boilers ??
• Plant Improvements
• ????
• Controls ??
• High Efficiency Motors
• ????

EMV Protocols Based on EMV ???? IPMVP
Demand-Side ???
Supply-Side ???
Generation ?? Transmission ?? Distribution
??
Efficiency ?? Demand Response ????
11
EXAMPLE California 2006-2008 Energy Efficiency
Portfolio??????? 2006-2008??????
Evaluation Best Practice ??????

• Applies Standard Protocols ??????
• EMV Budget 100 Million
• EMV??1???
• Thousands of Projects Measured ???????

Demand-Side ???
Supply-Side
Generation ?? Transmission ?? Distribution
??
Efficiency ?? Demand Response ????
12
Evaluation Objectives
????

• ????
• ????????????,?????????????????
• ?????????????,???????????????
• ????
• ???????
• ?????????

• Primary objectives
• Document and measure the effects of a program and
  determine how well an efficiency program met its
  goals
• Help understand why those effects occurred and
  identify ways to improve current and future
  programs
• Other objectives
• Determining the cost-effectiveness of a program
• Documenting compliance with regulatory
  requirements

13
Evaluation is Part of the Program
Cycle???????????

• Evaluation results are used to make informed
  decisions on program improvements and future
  program designs and offerings
• The program implementation cycle is one in which
  programs are designed, then implemented, and then
  evaluated
• Using results of evaluations, programs are
  re-examined for design changes and then modified
• This cycle provides for a continuing process of
  program improvement

• ????????????,?????????????????
• ?????????????,????,?????????
• ??????, ????????????????????
• ????????????????????

14
Program Evaluation Cycle??????
Step 1 Program Goal Setting ??1?????? Setting
evaluation goals and reporting expectations ?
???????????
Step 2 Program Design ??2???? Preliminary
evaluation plan and budget ?????????
Step 3 Preparing for Program Launch
??3?????? Prepare detailed evaluation plan
collect baseline data as needed ????????-????????
Step 4 Program Implementation ??4???? Evaluatio
n data collection, analyses and reporting
(perhaps continuing on after program is
completed) ??????,?????(?????????????)
Feedback for future programs ???????
Feedback for current program ???????
15
Evaluation Best Practice??????

• Ample budget 5-8 of program budget
• Appropriate Governance
• Protocols and Community of Expertise
• Institutions
• Features
• Complete and transparent
• Balances risk, reliability and costs
• Consistent to allow comparison across programs
  and technologies

• ???????????5-8
• ?????
• ?????? (??)??
• ??
• ??
• ?????
• ????,??????
• ??????????????????

16
Evaluation Institutions????

• ????
• ????2??2.2.3.8
• 2.4 ?????
• ??
• ???
• ????
• ????
• ??

• DSM Institutional Framework DSM
• Section 2 of Manual 2.2.3.8
• 2.4 Measurement and Verification
• Responsibility
• Contractors
• Skilled Professionals
• Professional Societies
• Certification

17
EMV FrameworkEMV ??

• Institutions National, Provincial, Cities,
  Utilities, Consulting Companies
• Professional Development
• Training
• Certification
• Documents / Protocols
• Tools Metering Data

• ??-??,?,?,??????,????
• ????
• ??
• ??
• ??/???
• ??-??-??

18
Programs versus Projects?????

• ??????????????
• ???????????????
• ?????? ??????
• ??gt ??
• ????????????
• ????? (MV) gt ??
• ?????????????,????????????

• A portfolio is a collection of programs
• A program is a collection of multiple, similar
  projects
• A project is a single energy efficiency
  activity
• Evaluation gt programs
• Assess the performance and quality of a program
• Measurement and Verification (MV) gt projects
• data collection, monitoring and analysis project

19
2. Evaluation of Energy Efficiency
Programs???????

• ??????
• ?????????
• ????
• ????
• ????(GHG)????

• Program Evaluation Types
• Impact -Total and Net Calculations
• Process Evaluation
• Market Effects
• Getting Credit for Reduced Emissions (GHG)

20
Types of Evaluations????

• ??????
• ??(kWh)???(kW) ???????????
• ??????????????(??????,?????????)
• ??????????
• ??????????????????????????????
• ?????????????????,????????????????

• Impact evaluations determine
• Energy (kWh) and Demand (kW) savings and avoided
  emissions
• Other benefits (such as health benefits, job
  creation, and water savings) that directly result
  from a program
• Can also include cost-effectiveness analyses
• Process evaluations assess how well a program
  was, or is, being implemented
• Market effects evaluations estimate a programs
  influence on encouraging future energy efficiency
  projects because of changes in the energy
  marketplace

21
Impact Evaluations Total and Net?????????

• Total Energy Impact - The change in energy
  consumption that results directly from actions of
  participants in the DSM program.
• Net Energy Impact - The total change in energy
  use that occurred because of the DSM program
• Total Energy and Net Energy impacts are not
  usually the same
• Net Savings/Total Savings Net to Total Ratio
  (NTR)

• ???????DSM????????????????????
• ?????????DSM??, ?????????????
• ????????????????
• ???/??? ????????(NTR)

22
Net to Total Ratio????????

• Net Energy Impacts adjust the Total
• Energy savings that would have occurred without
  the program
• free riders
• (reduces total energy savings)
• Energy savings that occur without direct
  influence of program
• free drivers
• (increases total energy savings)
• Energy savings or energy increases due to change
  in operations
• for example, if a factory operates motors more
  hours because cost of running the system is lower
  with the measure installed
• (reduces total energy savings)

• ???????????
• ?????????????
• ???
• (??????)
• ??????????????
• ??????
• (??????)
• ??????????????????
• ??,,????????????,??????????????????
• (???????)

23
Net Impacts Methods??????

• ??????
• ???????????????,????????
• ?????????
• ??????,????????????
• ???????
• ??????????????????????,????????????/????
• ??????????
• ??????

• Self-reporting surveys
• Information is reported by participants and
  non-participants, without independent
  verification or review.
• Enhanced self-reporting surveys
• Self-reporting surveys, interviews and
  independent documentation review and analysis.
• Econometric methods
• Statistical models compare participant and
  non-participants energy/demand patterns with
  adjustments using econometric analyses and survey
  input.
• Stipulated net to total ratio Based on historic
  information

24
Process Evaluations????

• Recommendations can consist of suggestions for
  changing
• Programs structure
• How the program is implemented
• Program design
• Program delivery
• Program goals
• Should test for the use of best practices and
  consider consumer satisfaction
• Should assist timely feedback to make program
  changes as needed or to review early findings.

• ?????????
• ????
• ??????
• ????
• ????
• ????
• ?????????????????????
• ?????????,??????????????????????

25
Process Evaluations are Valuable when ????????,?

• The program is new or has many changes
• Energy savings are less than expected
• There is low participation by consumers and other
  participants in programs
• Program has a slow start-up
• Participants are reporting problems
• Program appears to not be very cost-effective

• ??????????????
• ?????????
• ????????????????
• ???????
• ??????????
• ??????????

26
Process Evaluation Methods??????

• As with impact evaluations, process evaluation
  requires a plan
• It is best if the process evaluation is done by a
  group independent of the program implementation
• Process evaluation is done with data collection
  from
• Surveys
• Individual interviews
• Group interviews
• Written surveys
• Observation, measurement and analyses

• ??????,??????????
• ???????????????????????
• ????????????
• ??
• ????
• ????
• ????
• ??,?????

27
Market Effects Evaluation??????

• Market Effects Study evaluates increases in the
  adoption of energy efficient products, services
  in the market.
• Market effects evaluations determine whether the
  market is changing. Example What types of high
  efficiency products are generally available, such
  as CFLs?
• Market effects evaluations usually consist of
  surveys, reviews of market sales data, and
  analysis of the survey results and collected data.

• ?????????????????????????
• ??????????????????????????????????,???????(CFLs)?
  
• ??????????????,????????,???????????????

28
Evaluations can be uses to Calculate Avoided
(GHG) Emissions??????????????(GHG) ???

• ?????
• (???????)x (????)
• ????????
• ???
• ????
• ??????????(CDM)??

• Avoided Emissions (Net Avoided Energy Use) x
  (Emission Factor)
• Issues that must be addressed
• Additionality
• Boundary Issues
• Clean Development Mechanism (CDM) methods
  required

29
Greenhouse Gas Reporting??????

• Corporate Accounting and Reporting Standard.
  Standards, guidance, and web-based calculation
  tools to help companies, regulators, and others
  develop an organizational-wide greenhouse gas
  emissions inventory.
• GHG Project Accounting and Reporting Protocol.
  Requirements and guidance for quantifying
  reductions from greenhouse gas mitigation
  projects, for example, those used to offset
  emissions or to generate credits in trading
  programs.
• GHG Protocol Guidelines for Quantifying GHG
  Reductions from Grid-Connected Electricity
  Projects
• These documents are available at
  http//www.wri.org/climate/

• ????????????,?????????????,????,??????????????????
  ??????
• GHG???????????????????????????,???????????????????
  ???????
• ???????????????????(GHG)?????
• ???????????http//www.wri.org/climate/

30
3. Measurement and Verification of Energy
Efficiency Projects??????????

• Measurement and Verification Documents
• The IPMVP - International Performance
  Measurement Verification Protocol
• IPMVP Options (A,B,C,D)

• ???????
• IPMVP-???????????
• IPMVP??(A,B,C,D)

31
Measurement and Verification (MV)????? (MV)

• MV is an evolving form of research, although
  common practices exist
• These practices are documented in several
  guidelines, including
• The International Performance Measurement
  Verification Protocol - IPMVP (2007)
• U.S. FEMP MV Guidelines Measurement and
  Verification for Federal Energy Projects Version
  3.0 (2008)
• ASHRAE Guideline 14 Measurement of Energy and
  Demand Savings (2002)

• ?????????????????,?????????
• ????????????,??
• ???????????-IPMVP (2007)
• ????????(FEMP) MV???????????????? 3.0 (2008)
• ????,??????????(ASHRAE )?? 14??????????(2002)

32
Determining the Savings????
33
MV Options MV ??
Options A and B are retrofit-isolation
methods ??A?B??????? Options C and D are
whole-facility methods ??C?D??????? The
difference is where the boundary lines are
drawn ??????????
34
MV Options MV ??
The Options differ in their approach to the
level, duration, and type of baseline and
reporting period measurements. Options A and B
measurements are taken at the end-use system
level (e.g., lighting, motor system) Option C
evaluations are conducted at the whole-building
or whole-facility level. Option D evaluations,
which involve computer simulation modeling, are
also made at the system or the whole-building
level.
??????????????????,?????????????
??A?B???????????????(????,????) ??C????????????
????? ??D??,?????????,?????????????
35
MV Options (cont) MV ??(??)
??????? ??A- ??????????????????????
,?????????????????? ??B?C-???????????,??,?/??????
?????(??B)?????(??C)? ??D-??????,??,?????????????
?????
In terms of type of measurement Option A -
involves using a combination of both stipulations
and measurements of the key factors needed to
calculate savings in engineering models. Options
B - and C involve using data collected from
spot, short-term, and/or continuous measurements
in engineering models (Option B) or regression
analyses (Option C). Option D - may include
spot, short-term, or continuous measurement data
to calibrate computer simulation models.
36
Example IPMVP in Evaluation Application -
California??IPMVP?????-?????

• In the 2006 California Energy Efficiency
  Evaluation Protocols - MV plans must adhere to
  the IPMVP.
• A basic level of MV is defined as involving a
  statistically adjusted engineering model with
  metering/monitoring per IPMVP Option A.
• An enhanced level of rigor is also defined that
  would be based on IPMVP Options B (retrofit
  isolation) or D (calibrated simulation).
• Details are defined in a program MV plan that is
  reviewed on a case by case basis.

• ?2006?,???????????- MV??????IPMVP??
• MV ?????(??)?????IPMVP??A???????/??,?????????????
  
• ?????????????IPMVP??B(????)?D(????)
• ?????????MV????, ????? MV???????

37
Example IPMVP in Evaluation Application - New
York State Programs??IPMVP????-?????

• Commercial/Industrial Performance Program (CIPP)
  pay-for-performance program
• Approximately one third of NYSERDA savings
• MV protocols based on IPMVP and FEMP MV
  Guideline 2.2
• Employs IPMVP Options A D
• MV evaluation of NY States CIPP impacts
• Multiple approaches ranging from stipulated
  savings values to using IPMVP options A D
• Contracts specify that reviews be consistent with
  IPMVP

• ??/??????(CIPP)?????????????
• ????????????????????????
• MV????IPMVP?FEMP MV ??2.2?
• ??IPMVP??A-D
• MV?????CIPP?????
• ?????????????IPMVP??A-D
• ?????????IPMVP

38
4. Procedures, Principals and Methods??,?????

• Issues to Address in Evaluation Planning
• Resource Documents

• ???????????
• ????

39
Common Evaluation Issues To Address in Planning
??????????????

• Define evaluation goals and scale (relative
  magnitude or comprehensiveness)
• Set time frame for evaluation and reporting
  expectations
• Set spatial boundary for evaluation
• Define program baseline, baseline adjustments,
  and data collection requirements
• Establish a budget in context of information
  reliability goals
• Select impact evaluation approaches for total and
  net savings calculations, and avoided emissions
  calculations
• Select who (or which type of organization) will
  conduct the evaluation

• ?????????(????????)
• ????????????
• ????????
• ????????,????,????????
• ?????????????
• ?? ?????????????????????????
• ???(?????)?????

40
EMV Source Documents 5.5 EMV ???5.5

• 2007 US National Action Plan for Energy
  Efficiency Model Energy Efficiency Program Impact
  Evaluation Guide.
• This guide describes a structure and several
  industry-standard approaches for calculating
  energy, demand, and emissions savings resulting
  from facility (non-transportation)
  energy-efficiency programs that are implemented
  by cities, states, utilities, companies, and
  other similar entities. www.epa.gov/eeactionplan

• 2007??????????????????????
• ?????????????????????,??????????(???)?????????????
  ,????????,??????????,?,????,?????????????????
• www.epa.gov/eeactionplan

41
Evaluation Resources California????-?????

• California Public Utility Commission
• Energy Efficiency Evaluation Protocols
  Technical, Methodological, and Reporting
  Requirements for Evaluation Professionals
  (California Evaluation Protocols) (2006)
• Evaluation Framework (2004)
• The California Evaluation Protocols and Framework
  can be found at
• California Measurement Advisory Council (CALMAC)
  http//www.calmac.org
• Searchable Database of past evaluations
• Consortium for Energy Efficiencys Market
  Assessment and Program Evaluation (MAPE)
  Clearinghouse http//www.cee1.org/eval/clearingho
  use.php3

• ?????????
• ????????????????,????????(???????) (2006)
• ????(2004)
• ????????????????
• ????????????( CALMAC ) ??http//www.calmac.org
  
• ???????????
• ??????????????(MAPE)???? http//www.cee1.org/eval
  /clearinghouse.php3

42
Other Evaluation Resources??????

• Evaluating Energy Efficiency Policy Measures
  DSM Programmes, International Energy Agency,
  December 2006. http//dsm.iea.org
• EERE Program Analysis and Evaluation Management
  Guide, U.S. Department of Energy Office of Energy
  Efficiency and Renewable Energy (US DOE EERE)
  2007. The guide is available athttp//www1.eere.e
  nergy.gov/ba/prog_mgmt_guide.html
• Impact Evaluation Framework for Technology
  Deployment Programs, U.S. Department of Energy
  Office of Energy Efficiency and Renewable Energy
  (US DOE EERE), 2007. The guide is available at
  http//www.eere.energy.gov/ba/pba/km_portal/docs/p
  df/2007/impact_framework_tech_deploy_2007_main.pdf
  

• ?????????DSM??,?????http//dsm.iea.org
• ??????????(EERE)???????????,??????????????????(US
  DOE EERE)2007?????????http//www1.eere.energy.gov
  /ba/prog_mgmt_guide.html
• ?????????????,??????????????????(US DOE EERE)2007
  ????????? http//www.eere.energy.gov/ba/pba/km_po
  rtal/docs/pdf/2007/impact_framework_tech_deploy_20
  07_main.pdf

43
5. Evaluation Management and Planning???????

• Project Tracking Databases
• Database of Energy Efficiency Resources
• Evaluation Planning using Uncertainty

• ???????
• ?????????
• ????????????

44
Database Types?????

• Project Tracking
• ????
• Evaluation Planning Management
• ???????
• Measure Results
• ????
• Evaluation Results
• ????

Database for Energy Efficient Resources ????????
45
Applying Uncertainty Analysis????????

• Apply at Portfolio Level (Top down)
• ???????????(????)
• Allocation of EMV resources
• EMV????
• Apply to Programs ?????
• Evaluation planning by targeting most uncertain
  measures
• ?????????????
• Apply to Project MV (Bottom up)
• ?????MV (????)
• Identifying the variables to measure
• ?????????
• Optimal allocation of resources?
• ???????
• Based on cost of reduced uncertainty
• ???????????

44
46
Summary of Presentation????

• MV is used to assess savings from individual
  projects to calulate total program and portfolio
  savings.
• Evaluation uses MV and other methods to
  determine savings and improve programs.
• There are many available resources to build an
  excellent evaluation capability in China.

• MV????????????, ??????????????
• ????MV ???????????????????
• ??????????????????????

47
Evaluation helps you achieve your energy
performance goals!???????????????
Steve Kromer

• Thank You Good Luck ??!