Lessons learned from

1

• Lessons learned from
• Development and Application of
• Plant Monitoring Annunciator System
• 2008.11.04
• Seung Min Baek ( )
• (smbaek_at_kopec.co.kr)
• Korea Power Engineering Company, Inc.

2
Contents

• Introduction to KOPEC
• Development of Plant Monitoring and Annunciator
  System
• History of PMAS Development
• Background
• PMAS Prototype Development
• PMAS Functions
• Application to Actual Projects
• PMAS Work Characteristics
• Lessons learned from PMAS Development and
  Application
• Conclusion and Discussion

3
Introduction to KOPEC
President/CEO
Auditing Executive
Quality Assurance
Management Strategy
Auditing Dept.
Planning Admin. Division
Eng. Research Division
Nuclear Project Division
NSSS Eng. Project Division
Plant Eng. Construc. Division
Strategic Project Division
Power Eng. Research Institute
4
Introduction to KOPEC - KOPEC NEDs Scope of
Work

• NSSS Design and Engineering for PWR and PHWR
• Safety Analysis
• Preparation of Guidelines for Installation,
  Testing and Operation
• Start-up and Licensing Services
• Engineering Services for Operating Plants
• Training Services for Engineers and Operators
• Research and Development
• Project Achievement for Nuclear Power Plant
  Construction
• Yonggwang NPP units 3, 4, 5, 6
• Ulchin NPP units 3, 4, 5 6
• Wolsung NPP units 2, 3 4
• Shin-Kori units 1, 2, 3, 4
• Shin- Wolsung NPP Units 3 4

Under Construction
5
Introduction to KOPEC - Organization of KOPEC
NED
6
2. Development of PMAS - History of PMAS
Development

• Plant Monitoring Annunciator System(PMAS)
• An Integrated System of Plant Monitoring Computer
  System (PMS) and Plant Annunciator System (PAS)
• History of PMAS through KSNP Project
• KSNP Phase 1 1998.011999.01
• Feasibility study for the integration of PMS and
  PAS
• KSNP Phase 2 1999.102001.10
• System Design of PMAS
• Generation of System Design Requirements and
  Design Specification
• Prototype Implementation
• Project Impact Analysis for SKN-12 and SWN-12
  Project
• Application to SKN-12 and SWN-12

7
2. Development of PMAS - Background of PMAS
Development

• PMS and PAS features
• PMS Plant Computer System Plant Data
  Acquisition System
• Major Functions
• Plant Process Monitoring, NSSS BOP Application
  Programs including COLSS, SPDS and etc.
• Historical Data Storage and Retrieval, Alarming,
  Logging and etc.
• Scope of Supply
• KOPEC NED System Design, Implementation of MMI
  S/W, Application S/W and Operator Support
  functions
• WEC Implementation of System Software,
  Integration with KOPEC S/W, FAT and Equipment
  Supply
• PAS Plant Annunciator System
• Collecting alarm signals, Processing them and
  presenting through Annunciator windows
• System designed by KOPEC A/E

8
2. Development of PMAS - Background of PMAS
Development
9
2. Development of PMAS - Background of PMAS
Development

• Motivation for the integration of PMS and PAS
• Inconsistency between PMS and PAS indicated by
  KINS
• Display inconsistency from the aspect of HFE
• Information inconsistency
• Unnecessary Redundant Device/Signals for similar
  information
• PAS CRTs and PMS Alarm CRTs,
• PAS Printer and PMS Alarm Printer
• Separate Cables to PAS and to PDAS for same alarm
  signal
• Inefficiency in Operation/Maintenance
• Inefficiency in training of the maintenance
  engineer
• Inefficiency in Spare part procurement and
  Management
• Frequent Design Changes and Design Improvement
  requested by customer

10
2. Development of PMAS - Background of PMAS
Development
11
2. Development of PMAS - PMAS Prototype
Development
...
...
EOF
NERC
NDL
PR
PR
GATEWAY SERVER 1
GATEWAY SERVER 2
MAINTENANCE SERVER
WS 1
WS2
WS 3
WS 4
WS 5
WS 6
WS 7
ALM WS 1
ALM WS 2
WS 8
Modem
PR
PR
ALM PR
PR
SYSTEM LAN (REDUNDENT SWITCH)
CEDMCS
DATA SERVER 1
DATA SERVER 2
ALARM SERVER 1
ALARM SERVER 2
SYSTEM SERVER 1
SYSTEM SERVER 2
RMS
BOP PCS N1
ICCMS A
ICCMS B
DPPS 1
DPPS 2
TCS
BOP PCS N2
PLANT LAN
CPCS
INPUT LAN
TIME LAN
PDAS A
PDAS C
PDAS N1
INSC A1
INSC A2
PDAS B
PDAS D
PDAS N2
INSC B1
INSC B2
MUX N1
MUX N2
MUX N3
MUX N4
MUX N5
MUX N6
MUX N7
HJTC HTR
HJTC HTR
TR/METER
TERM A1
TERM A2
TERM A3
ISOL A
ISOL C
TERM B1
TERM B2
TERM B3
ISOL B
ISOL D
12
2. Development of PMAS - PMAS Prototype
Development
PAS TERM
SIMULATOR
WINDOW BOX CABINET
PCS SERVER A
PCS SERVER B
PDAS CH A
PAS INCS A1
Alarm
Graphic
Maintenance
I/O
Graphic
Alarm
Graphic
Display 2
Display 1
Server
Simulator
Display 2
Display 1
Display 3
13
2. Development of PMAS - Software Environment

• Software Environment
• Server
• OS Unix (or Linux)
• PL c/c
• fotran for specific applications
• Display
• OS Linux
• PL c/c
• Graphic Tool ILOG Views

14
2. Development of PMAS - PMAS System Software

• RTIPS (Real-Time Information Processing System)
  is the system software package developed for
  reliable and robust real-time monitoring
  functions in the PMAS.
• Task Scheduling
• Input/Output Processing including alarm
  processing
• Communication among subsystems
• Real-Time Database
• Time Synchronization
• Failover
• RTIPS provides the following features
• Information Processing with Real-Time Task
  scheduler Database
• Easy interface with plant specific application
  programs
• C/C based software package
• High flexibility
• Advanced man-machine interface with ILOG display
  tools

15
2. Development of PMAS - Operator Support
Functions

• Point Dictionary
• Point Dictionary allows the user to quickly
  search the specific point in database.
• The user can search the point by entering Point
  ID, Tag ID, or description.
• Point Dictionary supports drag and drop operation
  within the display page.

16
2. Development of PMAS - Operator Support
Functions

• Point Viewer
• Point Viewer allows the user to view database
  contents with its current value or status for a
  specific point.

• The user can select the format of the point value
  as integer, decimals, exponent, etc.
• Point Viewer provides the capability to display
  the current value on the monitor as full size
  when the user wants to see the point value or
  status from far away distance.

17
2. Development of PMAS - Operator Support
Functions

• Annunciator Window Viewer
• Annunciator window viewer allows the operator
  easy access to the annunciator windows alarm
  information through the operator s monitor.

• Link to alarm list or alarm procedure

18
2. Development of PMAS - Operator Support
Functions
Alarm Procedure
Alarm List Viewer
19
2. Development of PMAS - Operator Support
Functions

• System Status Overview
• System Status Overview provide on-line display
  that graphically represents the status of the
  system hardware, components, communication
  network, and software execution.

• System Status Overview provides the capability
  with access to more detailed displays of
  subsystem equipment to allow the operator to
  acquire more detailed information.

20
2. Development of PMAS - Operator Support
Functions
Bar Chart and Real-time Trend Curves
21
2. Development of PMAS - Operator Support
Functions

• Log Report
• Log Report automatically accumulates periodic
  data on an hourly, shift, daily, and monthly
  basis. The values printed out for each periodic
  log consists of averages, minimum, maximum,
  and/or summation.

• Periodically saved to the permanent storage by
  HDSR Backup function
• Format and point lists of each log to be set up
  by Report Builder.
• Demand Log automatically displays the log report
  according to the predefined time interval.

22
2. Development of PMAS - Historical Data
Storage and Retrieval

• Historical Data Storage and Retrieval (HDSR)
  records, stores, and retrieves the historical
  information.

• HDSR stores and retrieves the data with a
  resolution of the scan rate of the points.
• HDSR stores SOE reports, alarm logs, and
  scheduled log reports.
• backup to permanent storage such as DAT, or DVD.

23
2. Development of PMAS - Historical Data
Storage and Retrieval
Historical Trend displays the historical data
with similar format of Real-Time Trend.
24
2. Development of PMAS - Engineering Tools

• Engineering Tools
• Point Configurator
• Summary of the important attributes in database
  records
• Provision to change point attributes such as scan
  status, scan rate, alarm status, alarm priority,
  alarm limit, deadband, and entered value
• On-Line DB Configurator
• Provision for I/O database modification such as
  addition or deletion of I/O points into the
  database while a system is running on-line.
• Setup for Trend Curves, Group, Pen Recorder,
  Sensor Calibration
• Composed Point Builder, Report Builder
• Security Builder

25
2. Development of PMAS - Engineering Tools
On-line DB Configurator
Report Builder
Engineering Tools
Composed Point Builder
Point Configurator
26
2. Development of PMAS - Application Programs

• Plant specific application programs can be easily
  accommodated
• in the PMAS. In the KSNP(OPR-1000), following
  application
• programs are provided
• Core Operating Limit Supervisory System
• Safety Parameter Display System
• Control Rod Monitoring System
• BOP Performance Calculation Program
• Xenon Prediction and Reactivity Balance Program
• Pressure and Temperature Limit Curve Program
• Mid-Loop Operation Support Program
• RCS Leak Rate Calculation Program
• Safety System Setpoint Channel Deviation
  Monitoring Program

27
2. Development of PMAS - Application Programs

• Safety Parameter Display System
• Safety Parameter Display and Evaluation System
  (SPADES) Safety Parameter Display System (SPDS)
  implementation in the PMAS
• Presentation of critical safety function status
  with the supporting operational displays to
  assist control room personnel in evaluating the
  safety status of the plant
• Support of Normal Operation, Standard Post Trip
  Action, Diagnostic Action, Operational Recovery
  Procedure, and Functional Recovery Procedure

28
2. Development of PMAS - Application Programs
Safety Parameter Display System

• SPADES display Hierarchy
• Level 1 Overview information on the status of
  the Critical Safety Functions
• Level 2 Alarm logic of Critical Safety Functions
  and Resource Assessment Tree (RAT) for Success
  Path
• Level 3 Major system, subsystem and component
  status information

29
2. Development of PMAS - System Mimic Display
30
2. Development of PMAS - Achievement through
PMAS Development

• Improvement in Sequence Of Event (SOE) function
• Relocation SOE modules to PAS from PDAS N1E
• Reduced Time Delays
• Cable Reduction
• Time Synchronization among PMS and PAS
• Unnecessary Redundant Signals to both PDAS and
  PAS assigned to either PDAS or PAS according to
  the signal type
• Reduction of PDAS and PAS Cabinets
• Two(2) Aux Relay Cabinet removed in PAS
• One(1) PDAS N1E Cabinet removed
• Improvement in Input Scan Rate
• Digital1 or 10 msec, Analog 100 or 200 msec
  (Depending on customer request)

31
2. Development of PMAS - Achievement through
PMAS Development

• Sharing of MMI Devices
• Monitors, Printers, and Storage Devices
• Reliability enhanced by incorporating On-line
  real-time diagnostics functions
• Consistency between PMS and PAS Information
  Display
• Development of SPDS which can support EOP SPADES
• Improvement in HDSR functions
• All PMAS I/O points including PAS alarms are
  stored and retrieved with scan rate resolution.
• Most of reports including SOE report and Log
  reports are stored.
• Convenient operator interface with zoom-in and
  zoom-out capability
• Many other operator support functions enhanced
  based on Operator Experience Review.

32
3. Application to Actual Projects

• Kori-2 Plant Computer System Replacement
  (2002.2.28 2003.12.31)
• Delivery to the Site 2002.07.29 2003.08.30
• 90 days Availability Test 2002.09.18
  2002.12.19
• Successful completion of SAT 90 days
  availability test
• SKN-12 and SWN-12 Project (2002.08.09 )
• SKN-1 PMAS Delivery to the plant site
  2008.06.30
• UCN-12 Plant Computer System Replacement
  (2008.4.1 )
• 1st Delivery to the site 2008.09.10

33
3. Application to Actual Projects - Hardware
Configuration applied to Kori-2
34
3. Application to Actual Projects - Hardware
Configuration applied to SKN-12
35
3. Application to Actual Projects - PMAS
Interface with other systems
PCS Application Programs Alarm Display HDSR PID
Mimic Trend Curves/Bar Charts Real-time Database
AI (1160) CI ( 688) AO( 28) CO(208) Others (24)
PDAS
PAS
ALM (4300) SOE ( 800)
ICCMS
ICCMS (186)
AI Analog Input CI Contact Input AO Analog
Output CO Contact Output SOE Sequence of
Event AL Analog Link DL Digital Link
NSSS IC CPCS, DPPS CEDMCS
CPCS (612) DPPS (136) CEDMCS (236)
BOP IC TCS, RMS, BOP PCS
AL (1000) DL (1300)
Total No. of I/O 10,678
(except for Internal Points)
36
4. PMAS Work Characteristics

• Software Oriented System
• Inherent property of S/W from No Silver
  Bullet by Fred Brooks, Jr.
• Complexity, Conformity, Changeability,
  Invisibility
• Very complex Non-Safety Grade Computer System
• Versatile operator support functions
• Interface with various IC systems
• S/W Implemented by a team, not by a few S/W
  engineers
• Large scale of Testing required
• Frequent design change or improvement
• Interface design defined lately
• HFE TA or VV results
• Customer request for change after delivery

37
5. Lessons learned from PMAS
Development and Application

• Benefit of Prototype Development
• Verification of design concept
• Better Configuration for the integration of PMS
  and PAS
• Preview of customer requirements / user interface
• Minimization of schedule impact in actual project
• Completion of preliminary design specification
• Evaluation of design tools
• Technical review of interface including
• Other NSSS/BOP Systems
• Architect Engineering
• Early setup of software development environment

38
5. Lessons learned from PMAS
Development and Application

• Major causes for failures
• Configuration Control Problem
• Database
• Database design real-time response, sufficient
  data capacity and on-line maintenance capability
• Complexity of the On-line database modification
  mechanism
• Validation routine required to reduce database
  input errors
• Network
• Major devices for Distribution and Integration
• Failure of backbone network means system failure
• Backbone network to be designed for higher
  reliability
• Data Communication Protocol design during design
  phase
• Analysis of all system functions associated with
  data communication including functions for
  computer system operation and maintenance as well
  as plant operation

39
5. Lessons learned from PMAS
Development and Application

• Major causes for failure
• Time Synchronization
• Time server failure may cause whole system level
  failure because critical part of S/W uses time
  tag information.
• Time synchronization to be performed redundant
  and diverse ways
• Essential to make each sub-system to work with
  its own internal timer in case of time
  synchronization failure

40
5. Lessons learned from PMAS
Development and Application

• Importance of Configuration Control
• IEEE Standards 828 and 1042
• Configuration control is the means to keep
  integrity and traceability of the S/W.
• Several factors for Configuration Control
• S/W developers are reluctant to put their
  software into Configuration Control environment.
• System administrator should have appropriate
  authority for S/W Configuration Control.
• S/W developers correct understanding on the
  Configuration Control is much more important.
• A unified S/W configuration control environment
  is recommended in case of multiple organizations
  participating in a project.
• Test need to be performed using tasks newly
  generated from configuration Control server.

41
5. Lessons learned from PMAS
Development and Application

• Lessons from Testing
• Appropriate testing to be performed in each phase
  of software development
• The result of the testing and resolution of
  errors occurred to be documented
• Test Procedure
• System test procedure to be prepared based on
  system design document (high level requirements)
• Prerequisites for the test to be written very
  specifically in details
• Pre-test is recommended before issue of the
  procedure and testing and will make the testers
  familiar with test environment.
• For complicated S/W, test procedure can not be
  perfect. Test engineers role is very important.

42
5. Lessons learned from PMAS
Development and Application

• Lessons from Testing (continued)
• Test Engineers
• Very familiar with system design requirements and
  enough experience for similar testing
• Not to be a software programmer
• Need a capability to check, analyze software
  hardware problems, and document the problems
• Testing beyond Test Procedure
• Retest not only failed test procedure but also
  all procedures associated with S/W correction
• Test Equipment
• Test depends on Test Equipment capability (Signal
  generator, S/W Simulator, Interface System
  Emulator)

43
5. Lessons learned from PMAS
Development and Application

• Lessons from Testing (continued)
• Test Result Documentation
• Test result marked up in the Test Procedure
• Test procedure changes marked up with reason,
  date and signature for revision
• Failed test to be notified to S/W developer and
  retested after S/W correction or Test environment
  change
• Documentation to include the resolution of the
  failed test
• An additional means to share test result, S/W
  correction, improvement, questions and answers
  between testers and developers

44
5. Lessons learned from PMAS
Development and Application

• Flexibility of the GUI tool
• to meet customer requests getting more complex
• to resolve the discrepancies found from the Human
  Factor Engineering (HFE) review
• to reflect the field change request after
  delivery
• Cyber Security
• Korean Regulatory Guide for Cyber Security issued
  in Dec. 2007 based on US NRC Reg. Guide 1.152 rev
  2.
• Recommend to reflect cyber security requirements
  in the design document so that they could be
  verification and validation items. Then they can
  be verified and validated in each design phase as
  stated in Reg Guide 1.152.
• Utility needs to set up cyber security policy and
  plan.

45
5. Lessons learned from PMAS
Development and Application

• Other Lessons
• Reliable Manpower Operation
• Qualified engineers consistent and extended
  involvement in time through the development
• Tech Manual Preparation
• Not a work after completion of Final Acceptance
  Test
• Part of Tech Manual (ex Startup procedure,
  System Setup Procedure) to be utilized during
  development by testers, software and hardware
  engineers
• Major problem corrections to be reflected to the
  tech manual and not to repeat same problems in
  the site

46
6. Conclusion and discussion

• PMAS
• First localization of a large scale IC system
  for NPP in Korea
• Considerable Enhancement in Functionality and
  Performance
• Summary of the lessons learned
• Benefit of the prototype building
• Importance of software configuration control
• Several lessons in Testing
• Flexibility of the GUI tools
• Qualified engineers consistent and extended
  involvement in time through the development

47
6. Conclusion and discussion
Thank you for your attention.

• Questions?