Quality Control Review of E3 Calculator Inputs

1
Quality Control Review of E3 Calculator Inputs

• Comparison to DEER Database
• Brian Horii
• Energy and Environmental Economics, Inc.
• November 16, 2006

2
Overview

• Purpose was to review how well data entered into
  the E3 Calculator matches the DEER database.
• A review of all IOU submissions reveals that very
  few measures actually state the DEER Run ID or
  Measure ID
• The review also shows many measures that were
  entered, but had no planned installations.
• 9,236 total measures for all IOUs
• 6,136 with some installations
• 2,424 RunID matches
• 1,304 with installations
• 155 Measure ID matches
• 151 with installations
• The lifecycle and peak impacts of these measures
  are shown in the next slides

3
Lifecycle Gross kWh

• Small share of measures have DEER identifiers in
  E3 Calculator.
• Values not adjusted for NTG ratio.
• Values use entered EUL

4
Gross Peak Reduction
5
Lifecycle Gross Therms
6
Run ID Matching

• For the subset that had Run ID inputs, we
  compared how the entries match the DEER database.
• Several criteria for matches were used. All
  matches were to only 2 or 3 significant digits to
  allow for rounding
• Matches for either DEER common or code impacts
  AND either Incremental or Full costs.
• Matches that ignore a commodity (G or E) if no
  savings claimed
• Matches on ratios of Impacts to Costs. Either
  Common or Code can match. Choice of Incremental
  or Full costs for denominator based on DEER
  Application and CostBasis.
• Full costs used with Common impacts 86 of the
  time.
• Incr costs used with Code impacts 69 of the
  time.
• Matches if entered ratios are lower than DEER
  ratios.
• Note that these methods will NOT match cases
  where direct install costs are excluded from IMC
  and put into Admin.

7
RunID Details

• 994 measures match DEER
• 1,070 match if entry ignored when no savings
  claimed
• 1,077 measures match for Impact/Cost ratios
• 1,893 match if all impact/cost ratios LOWER than
  DEER are deemed OK
• 244 measures do not match under any test.
• 41 measures have negative peak impacts that were
  not entered (3.4 MW)
• 244 measures have cost and impact units that do
  not match. Of those, 149 passed one of the
  matching tests, we did not perform any unit
  reconciliation tests.
• Impact of changing inputs for the 244 non-matches
  is shown on the next table. (i.e. how different
  are those inputs?)

8
Run ID Subset

• Impacts for the measures with a DEER Run ID that
  match the DEER database, versus the non-matches
  are shown in the top half of the table.
• The bottom half of the table shows the change in
  impacts if we modified the inputs for the
  non-match cases to match the DEER database.
• There is little effect on the kWh forecast, but
  significant impact on kW, and less so on Therms.

9
kWh Detail for Measures w/ RunIDs

• Match includes measures where entered ratios are
  less than DEER ratios.
• Bottom table shows the change in kWh needed to
  match the DEER ratios.
• Negative value indicates that entered values have
  larger impacts than DEER

10
kW detail

• SCE overestimates are largely (if not entirely)
  due to pasting error mentioned at the prior
  QA/QC workshop.

11
Lifecycle Therm Detail

• Note that negative correction values for Sempra
  matches are due to cases where the measure had
  a negative therm reduction, but no therm impacts
  were claimed in the input section.

12
Other Run ID Findings

• kW (Impact of matching to DEER shown in
  parentheses)
• 104 cases where entered kW significantly above
  DEER. (228 MW)
• 2 cases where DEER Watts entered as kW, plus 4
  other conversion errors. (0.4MW)
• 6 additional cases where negative DEER impact is
  entered as positive (0.7 MW)
• 41 cases where negative kW impacts are not
  entered. Of those, 30 have no installations.
  (3.2 MW)
• Therms
• 5 cases where DEER values are Therms, not kBTU
  (24.4 MTh)
• 4 cases where impact and cost units differ
  despite DEER indicating same (12.1 MTh).
• Other findings
• 33 case where measure is RETROFIT, but base for
  impacts is CODE, not COMMON. This is
  conservative, and probably not an error, but
  highlights that users could use the DEER database
  to arrive at very different results by mixing the
  two sets of inputs and costs.

13
Review of Measures with No Run ID
14
Process

• Develop avg max ratios of impacts per GIMC
• Utility submissions that match DEER RunID, by
  measure end use categories (based on end use
  shapes in the E3 calculator)
• DEER measures by 60 subcategories.
• Map Measures
• Match measure end use shapes.
• Manually map 1739 cases.
• Compare entered ratios to maximums from step 1

15
Ratio test overview

• 6916 measures did not have RunID matches in the
  input data
• 4681 have some installations
• 317 have no GIMC
• Initial filtering using DEER ratios
• 579 measures with kWh ratios in excess of the
  DEER samples maximum
• But if average ratios are used, the utility
  submissions are very conservative in aggregate
  for kWh and kW.
• 419 measures with Therm ratios in excess of the
  DEER samples maximum
• The therm overestimate would be slightly higher
  in aggregate if the average ratios were used for
  all measures.
• Problems with filtering analysis
• Impact and cost unit mismatches make comparisons
  difficult
• 53,014 DEER runs have different units (out of
  about 120K)
• Results shown exclude all DEER runs where units
  are not the same
• Mapping of utility measures to categories is
  imprecise
• Aggregation into categories is imprecise

16
Impact Ratios by End Use

• Average and Maximum kWh ratios

17

• WattsMax Average
• kBTUMax Average

18
Results based on DEER Extract

• Based on reductions relative to CODE.
• Similar results if larger of CODE or COMMON is
  used

19
kWh ratios for DEER sub categories
20
Watt ratios by DEER subcategories
21
Using DEER groups yields comparable results

• Based on ratios using 60 DEER subcategories,
  results are similar to end-use matching
• Matches for gas measures remains poor

22
Unmatched Measures

• We used the maximum impact per GIMC as a very
  generous criteria
• Yet, even with that criteria, 562 cases where kWh
  ratio is exceeded, 406 cases for kW ratio, and
  490 cases for Therm ratio. (1,045 unique cases).
• Cases by utility
• PGE 393
• SCE 121
• SDGE 167
• SoCal Gas 364
• Note may indicate an input problem, or a problem
  with the assumed mapping or a problem introduced
  by the large number of DEER runs excluded due to
  unit mismatches.

23
With better data, the ratio test could be a
useful screen

• Distribution of kWh ratios from the utility
  measures
• Largest ratios are for measures such as
• Pre-rinse spray valve electric water heating
• Strip curtains
• Lighting

24
Identification of measures with largest ratio
mismatches

• Based on sorting lifecycle amounts
• The measures will differ from column to column

25
Some DEER Measure Gaps

• Non-Res freezers and refrigeration
• Non-Res ovens
• Non-Res pool heaters
• Computers
• Non-Res gas measures

26
Summary of Findings

• 76 of cases with some installations have no easy
  links to RunIDs.
• Note that for PGE, 27 of their 1668 cases w/o
  DEER RunIDs are for calculated measures
• The Max Ratio test is a blunt instrument, but
  more precision would be very time consuming given
  the data in DEER and in the E3 calculators.
• With that caveat
• KWh estimates appear reasonable. (3 above max)
• kW appears high, compared to DEER data (11 above
  max)
• Therms are hard to judge via DEER (54 above max)
• The max ratios pass 80 of the measures (4,274
  out of 5,319)
• Recommendations
• Future tool should require users to explicitly
  indicate if savings are relative to common or
  code, and if costs are installation or
  incremental.
• Need a way to link to DEER sub categories, at a
  minimum, to allow for automated checks
• Need a process for creating new approved
  measures, and for updating DEER.
• EULs should be reconsidered for retrofits
  (remaining life gets common benefit, and EUL-
  remaining life gets code benefit)
• Secondary impact on other fuels should be
  considered.
• Direct install costs should be input on a measure
  basis (not moved into lump sum admin costs) to
  allow for QA/QC review.