Statistical Testing for AMR Programs

1
Statistical Testing for AMR Programs
CB Associates AMR Seminar March 30, 2004
2
Why Use a Statistical Testing Plan?

• Focuses testing on the proper meters.
• Minimizes number of meters to be tested
• usually requires less than 30 of what a
• periodic testing plan requires.
• Can provide data and analysis tools for use in
• understanding what is happening with meters
• installed in the field or for use in the
  purchasing
• of new meters.

3
Acceptable Statistical Testing Plans
ANSI C12.1-2001 Code for Electricity Metering
Guidance
Paragraph 5.1.4.3.3 Statistical sampling
plan The statistical sampling plan used shall
conform to accepted principles of statistical
sampling based on either variables or attributes
methods. Meters shall be divided into
homogeneous groups, such as manufacturer and
manufacturers type. The groups may be further
divided into subdivision within the
manufacturers type by major design
modifications. NOTE - Examples of statistical
sampling plans can be found in ANSI/ASQC Z1.9,
the ANSI version of MIL-STD-414 and ANSI/ASQC
Z1.4, the ANSI version of MIL-STD-105.
4
Homogeneous Population(s)

• The groups or populations being sampled and
  tested
• are made up of the same or similar items,
  items
• which operate in the same way and were made
  in the
• same manner.
• For electric meters, this has traditionally
  been
• interpreted as being meters of a specific
  meter
• type from a manufacturer (i.e. AB1, J5S, MX,
  etc.).
• AMR programs have helped to make the overall
• populations more homogenous. This makes a
  utility
• with an AMR system better prepared to take
• advantage of a statistical sampling plan.

5
Suitably Sized Samples

• The sample size for each group must be large
• enough to provide a statistically valid
  sample
• for the groups population.
• The larger the groups population, the greater
  the
  
  
  
  savings for statistical testing over
  periodic
  
  
  
  testing and the more
  statistically reliable the testing
• AMR implementation generally results in larger
• group populations. The larger the population,
  the
• more suitable for statistical testing.

6
Random Sample Selection

• Every item within the group or population has
• an equal chance of being selected as part of
  the
• sample for testing.
• Random sample selection is critical to
  providing
• for a statistically valid sample.
• AMR programs help to update and overhaul
• meter record systems. Having the records for
  the
• entire meter population updated allows for a
  better
• chance that any meter may be selected as part
  of
• the sample for testing.

7
Population Fits the Statistical Model

• The statistical model being used for the
  sampling/testing plan needs to
  match the actual distribution of the
  population.
• In most circumstances, one is looking at a
  normal or Gaussian distribution (i.e. a Bell
  curve).
• This can be checked using a histogram plot or a
  chi-square analysis. For mechanical and
  electromechanical meters, a normal
  distribution fits the actual data very well.
• For electronic or solid-state meters, there is
  some question due to the failure modes of these
  meters. These meter types are fairly recent
  designs, and not enough data has been seen yet to
  verify a normal distribution.

8
Population Fits the Statistical Model

• AMR programs put either retrofitted
  electromechanical meters in the field or solid
  state meters. Electric Utilities must be in a
  position in the near future to determine if the
  solid state meters have a normal distribution.
  The only way to determine this is to aggressively
  begin testing and evaluating the in-service solid
  state meters.

9
Statistical Testing Plan w/ AMR

• By definition an AMR system no longer has a pair
  of human eyes checking the installation each
  month. Statistical testing allows the Utility to
  quickly identify which areas may have a problem.
  
• Potential problems that could be caught by
  aggressive testing.
• A faulty batch of meters
• Design or premature equipment failures
• Poor installation due to a poorly trained crew
• Location related failures
• Energy Diversion

10
Population Fits the Statistical Model

• Test an installation and not just a meter. Test
  programs for AMR systems need to involve testing
  and checking the meter performance as well as
  checking and testing the installation. This more
  extensive test check list should be done for the
  higher revenue CI customers.

11
Statistical Sampling and Revenue Protection
One of the significant benefits to the
statistical sampling of AMR meters is the
potential to spot energy diversion more readily.
Statistical testing of meters will indicate the
overall health of the meter population. Coupled
with historical revenue information and meter
tamper flags statistical testing can become a
powerful tool for combating energy diversion.
Utilities will be in a better position than ever
to spot trends toward energy diversion more
readily and on a closer to real time basis.
12
Statistical Testing with AMR

• Statistical testing to monitor AMR programs will
  also point up
• design or manufacturing deficiencies
• installation or post-installation problems (some
  of which may or may not be energy diversion).
• All should be pursued and the root cause
  understood.

13
Statistical Testing with AMR
Statistical Testing in preparation for AMR
14
Statistical Testing Plan w/ AMR

• For the business case a good statistical testing
  program can determine how well the existing
  system is working and what revenue gains might be
  expected from replacing all of the meters over a
  36 to 48 month time period.
• A good statistical testing program can also be
  used to make sound business decision as to which
  meters should be retrofit and which should be
  replaced.

15
Statistical Testing with AMR
Setting up Testing Programs to monitor AMR
installations and post AMR performance
16
Statistical Testing with AMR

• As you implement your AMR program problems and
  exceptions will seem to pour out of the wood
  work. The key is to stay focused on the primary
  issues and not on the isolated occurrence.
  Statistical testing can help you to differentiate
  between the two.
• Testing during installation will help utilities
  to spot trends early on. Root cause analysis
  will help to determine if there is a design
  issue, a manufacturing issue, an installation
  issue, a communication issue, or a training
  issue. All of these will occur to one degree or
  another. Some can be corrected easily. All will
  require expensive revisit work.

17
Statistical Testing with AMR

• As you are completing your AMR installation the
  following is a brief check list and discussion of
  topics to cover
• What waivers may be available from your Utility
  Commission?
• How long will these waivers be effective?
• How will revenue protection work with your AMR
  program?
• Identify people responsible and whether or not
  they are interested in working together to
  develop a more comprehensive and informative
  program.

18
Statistical Testing with AMR

• Areas to cover continued
• Who is responsible for the meter performance -
  the utility or the vendor?
• Who determines when there is or is not a problem
  inside and outside the Utility?
• Frequency of sampling and objectives
• Design concerns
• Support concerns
• Installation concerns

19
Statistical Testing with AMR

• Example of an AMR test program
• Best time to start to develop the program is
  while the meters are being installed.
• Use installation reports to determine if there
  is any initial concerns about the meters being
  installed.
• Typical reports that should be available
• Failed Meter Report, Project to Date
• Electric Meters on Network Report

20
Statistical Testing with AMR

• For this utility the failed Meter Report listed
  nearly 30 failure or return categories into which
  the manufacturer classified returned meters which
  failed either before, during, or after
  installation. Additionally, returned meters
  which were pulled because of abnormal remote
  polling but passed a multi-function test (MFT)
  are listed in a separate category. The data for
  each category was broken down into three groups
• Recalls - Meters that were recalled prior to
  installation. These normally came from automatic
  recalls generated by meter module status flags.
  Some recalls were manually generated.
• Not Installed - Meters with problems found by the
  installer and not installed.
• Maintenance - Meter problems reported outside of
  the automated process and changes using a manual
  paper process.

21
Statistical Testing with AMR

• Of the various failure or return categories, the
  seven largest failure categories associated with
  start-up failures were analyzed in detail for
  possible trends. These seven categories
  represent 85 of all failures and 95 of failures
  not related to programming errors or electric
  surge damage. The seven categories included
• New meters Unable to Read Meter Module
• Retrofits Unable to Read Meter Module
• Abnormal Cumulative Count
• Packet Error in Meter Module
• Broken Leg/Base/Glass
• Burnt Meter/Base/Leg
• Defective 1S Meters

22
Statistical Testing with AMR

• Data for the categories was tabulated into a
  spreadsheet
• Data tables and graphs for each category were
  created.
• Summarized data and graphs for these seven
  categories, both collectively and individually
  were evaluated and presented to the management
  team.
• The graphs provided a visual picture of the
  growth of each failure category and were based on
  the associated summary data. Where appropriate
  and useful, graphs showing the percentage of the
  meter population for a failure category were
  included.

23
Statistical Testing with AMR

• Percentage graphs were done for the following
  categories
• Summary of Top 7 Failure Categories and Failed
  Meters Passing MFT Testing
• New Meters Unable to Read Meter Module
• Retrofits Unable to Read Meter Module
• Meters Reported as Failed but Passing MFT
  Testing
• Failure percentages were calculated for all
  categories, but due to the small percentages
  involved for some categories, graphs were only
  produced for the above four categories.
  Percentage data is tabulated on the data table
  for each category.
• Data on the installed new meters and overall AMR
  meter population was obtained from the Electric
  Meters on Network reports.

24
Statistical Testing with AMR

• Data Evaluation and Conclusions
• After monitoring the situation for nearly 2-1/2
  years and evaluating 32 months worth of data the
  following conclusions were made regarding
  statistical testing and monitoring of the newly
  implemented AMR system meters
• The overall meter failure rate, including
  returned meters passing MFT testing, was X of
  the installed population. Of this half are
  actual failures and half are returned meters
  passing MFT testing.
• This final number is considered to be fairly
  accurate since two months were allowed to pass to
  let the backlog of failed meters returned to the
  manufacturer be tested and added to the Failed
  Meter Report.

25
Statistical Testing with AMR

• For most failure categories, it was not possible
  to breakdown the failures between new meters and
  other AMR meters. The Unable to Read Meter
  Modules categories was the exception. The failure
  rate in this category for new meters was eleven
  times that of retrofit meters. New meters
  Unable to Read Meter Module is the largest
  failure category representing about half of the
  actual electric meter failures for the AMR
  project.
• Of the lesser failure categories, the failure
  rates were well under 0.10.

26
Statistical Testing with AMR

• Recommendations
• Since the AMR deployment has been completed, the
  following recommendations are made for follow-on
  monitoring of the AMR meter population
• In-service testing will be critical for
  determining the actual state of the installed
  meter population. For the random sample
  in-service testing program, all efforts should be
  made to ensure that a sufficient sample of new
  meters (at least 200 per group) is pulled from
  the field for in-service testing.
• Minor design changes in the new meter over the
  course of the AMR deployment could mean that the
  in-service performance of the meters may differ
  depending on the exact age of a meter and its
  design variations. Therefore, the in-service
  test results for the new meters should be
  analyzed in detail to see if there are obvious
  performance differences between different
  sub-groups of meters.

27
Why Do All of this Testing?

• Installation of AMR programs move at seemingly
  breakneck speeds with all focus on schedule. At
  the same time, problems and exceptions seem to be
  pouring out of the woodwork. Upper management
  wants to hear about project milestones and
  budgets and not about the problems. Especially
  not any publicly embarrassing problems associated
  with an AMR installation.

28
Why Do All of this Testing?

• The meter engineer will have only limited
  resources to address this multitude of problems
  and exceptions. Statistical testing will allow
  you to more readily identify where the problems
  are and where there were simply anomalies. The
  testing will help differentiate between training
  and equipment problems. The testing will also
  help to identify potential weak areas in the
  system that may bear closer scrutiny as the
  system goes into service. Putting a good testing
  system into place during the implementation will
  help to keep you on schedule, on budget, and out
  of trouble during the installation and will
  ensure that there will be a good system in place
  with the self discipline and understanding to
  administer the system.

29
Summary
AMR provides the Utility with the opportunity to
get even more and better business information
from their installed meter base. Statistical
Sampling of these in-service meters can help to
point up deficiencies in the installed system
during installation as well as shortly after
system implementation. The sampling can help to
identify potential energy diversion and can help
catch design inadequacies in the meters. Once a
problem is identified additional statistical
testing can help to zero in on a problem and help
to identify potential solutions. Statistically
testing the installed meter population will allow
the utility to more fairly meter the entire
population without unfairly charging any one
customer and without unfairly subsidizing any
group of customers. Statistical sampling plans
are also lower cost plans to use than the
traditional periodic plans.