Darlington Digital Control Computer System Replacement Approach and Experience

1
Darlington Digital Control Computer System
Replacement Approach and Experience

• Presented by Eric Hung, Yong Du

2
Acknowledgement

• The paper is prepared by
• Andy Zupan (OPG)
• James Liu (OPG)
• Eric Hung (OPG)
• Yong Du (University of Toronto)
• Eric Yu (University of Toronto)

3
Background

• Darlington Nuclear Plant
• CANDU reactor
• 4 Units (880MW)
• First Unit in service in 1990
• Each reactor is controlled by two digital control
  computers (DCCs) in a master/standby
  configuration
• Typical CANDU configuration
• good track record (gt 300 reactor years)

4
Highlight of Darlington DCCs

• Two independent Computers (Digital Equipment
  PDP11/70, CAE process control I/O)
• Each computer is capable of controlling the
  reactor designed with online maintenance in
  mind. Maintenance on one computer can take place
  while the unit is controlled by other computer
• Hot standby fault in the controlling computer
  (master) will results in automatic transfer to
  standby computer
• Fail safe unit shuts down automatically if both
  DCCs fail
• Internal Self checking subsystem detect
  hardware and software fault, which could result
  in transfer of control to the other computer

5
DCC Replacement

• Replacement project was initiated in later 1990s
  due to reliability and obsolescence issues
• Some constraints of the project
• Installation is to be done online high risk to
  production
• No spare part available
• Hardware technology used in the DCCs is out dated

6
Approach

• The team decided on a very conservative approach
  due to high production risk
• Upgrade will be done incrementally each
  installation is within 1 shift
• Improve reliability, fault tolerance/detection
  capability. Functional and throughput
  improvement is secondary
• Keep the process I/O system intact as the
  reliability is acceptable
• Provide flexibility for future expansion
• Keep software changes to minimal

7
Approach

• Replacement is to be done in stages
• Stage 1 improve reliability
• Stage 2 address obsolescence issue
• Stage 3 future expansion/enhancement
• Funding has been secured for the first two stages

8
Simplified diagram of DCCs

9
Stage 1

• Focus on replacing unreliable and high
  maintenance items
• Memory
• Power supply
• Moving head disk (MHD)
• Magnetic tape (MT)

10
Memory replacement

• Why we do it
• Increasing rate of memory error
• What we have done
• Replace original DEC memory with PEP 70
• Installed bus repeater to clean up bus signal

11
Power supply replacement

• Why we do it
• Unreliable cannot regulate to the spec
• Increase failure rate
• What we have done
• Replace with a in-house design power supply
• Much more efficient and precisely regulated
• Improve reliability by having dual redundant
  supply each is capable to provide adequate
  power

12
MHD and MT replacement

• Why we do it
• High maintenance items
• Spare parts is becoming more difficult to get
• What we have done
• Replace with an in-house design device (using
  FPGA)
• No moving parts improve reliability
• Improve error detection/correction with two banks
  of memory
• Replace both MHD and MT with same unit
• Allow online reading (not writing) of data by
  external devices (such as a PC) using IEEE 1394
  protocol (firewire)
• Improve error logging capabilities

13
Results and Lessons learned

• Results
• Good improvement in the system health of DCCs
• Lessons learned
• Review field conditions as early in the project
  as possible
• Modern devices may be more sensitive to power
  fluctuation than older devices
• Good power supply potential give you a huge
  payback.

14
Improved System Health Trend
15
Stage 2

• Goal Address obsolescence -replace obsolete
  components using up to date technologies
• CPU
• Massbus Controller
• Fixed Head Disk
• Unibus backplane
• Display/printer (in preliminary engineering
  stage, wont discuss further)

16
Approach

• Working with original hardware vendor and another
  smaller company
• Original vendor (OEM) has overall responsibility
  and to provide oversight to address proper QA
  (Z299 etc)
• OEM form a joint venture with another smaller
  company that specializes in PDP11 emulation,
  which has previously produced a PDP11/70 emulator
  and was used in the Fuel Handling system at
  Darlington for the last 10 years

17
Architecture

• System architecture
• star configuration
• high speed serial bus (links)
• error detection/ correction at end of links
  (better error checking)
• direct memory access by CPU
• Reduce UniBus traffic by removing MPC and FHD
  from it
• Allow future expansion using the serial bus

18
Validation

• Validation process
• Design reviews
• FMEA
• Testing
• Phased installation

19
Design reviews, FMEA

• Design reviews
• Recommended by standards
• PDR CDR
• FMEA
• Conducted on high level system design

20
Testing

• Test strategy
• additional to FAT and SAT
• focus on functional testing
• start component testing early, do not wait for
  complete system
• test result compared with PDP 11/70 result
• full regression testing after each stage
• phased installation

21
Test plans

• Generic
• SEM
• CP
• Unit DCC

22
Linking to requirements

• Objective the test strategy and test plans are
  good enough to verify the high level system
  requirements
• high level requirement at least as reliable as
  the original PDP 11/70 system
• decompose into lower level requirements
• every lowest level requirement is covered by the
  test strategy, test plans, and some test cases

23
Linking structure

• General linking structure to be built

24
Linking structure

• Arguments to be made (bi-directional)
• For each test case specified
• it is created to address an issue in a test plan
• For each issue in a test plan
• it is raised to verify a system requirement
• For each lower level requirement
• it is required due to a high level requirement,
  and
• the satisfaction of lower level requirements
  meets the higher level requirement

25
Installation

• General approach phased installation
• install on monitoring computers (SEM CP) before
  installing on control computers (Unit DCC)
• install on less complex computers before
  installing on more complex computers (SEM, then
  CP, then Unit DCC)
• allow for soak time after each installation

26
Summary

• Having an overall plan early in the project is
  crucial
• Incremental approach is a conservative and safe
  approach
• Improvement is evident in the early stage of the
  project, hence upper management is more willing
  to support
• Design reviews and FMEA are valuable tools

27

• Questions?