KEKB Accelerator Control System

1
KEKB Accelerator Control System

• KEKB Controls Group
• 2001-08-29
• by T. Katoh

2
Control System

• Control Computer System
• Timing Control System
• Beam Gate Control System
• Personnel Safety Control System
• Communication System
• Control Room and Operators Consoles

3
Contents

• Construction History of KEKB Accelerator Control
  System
• KEKB Control System Design
• System Hardware Configuration
• System Software Configuration
• Central Control Room

4
Construction History

• 1994.4 Started System Design for KEKB
• 1995.2 Controls Systems Evaluation
• 1995.5 Committee Recommendation to adopt EPICS
• 1995.8 First Version of Computer System
  Specifications
• 1996.1 Started to Dismantle TRISTAN Main-Ring
• 1996.3 Final Version of Specifications
• 1996.6 Opened the Bids
• 1997.3 abco1 and 15 IOCs were Installed
• 1997.4 Hardware Specifications were Fixed
• 1997.10 BT Lines were Commissioned
• 1998.3 The Rest of IOCs were Installed

5
Construction History(contd.)

• 1998.4 Started Software Installation with
  Link-Persons and Engineers from the Company
• 1998.12 Commissioning of KEKB Accelerators
• 1999.3 Commissioning with the BELLE Detector
• 1999.4 Started Designing PF-AR Control System
• 2000.10 Installation of abco2 and 10 IOCs for
  PF-AR
• 2001.2 Started to Dismantle PF-AR Devices
• 2001.7 Started to Install or Re-install Equipment
• 2002.1 Commissioning of PF-AR

6
KEKB Control System Design

• System Requirements
• Constraints
• Basic Concepts
• System Architecture

7
System Requirements

• Sources
• Equipment Groups
• Magnet and Power Supply
• RF
• Beam Monitor
• Vacuum
• Beam Transport
• Feedback, Facilities, Physics, etc.
• Operations Group
• Accelerator Physicists or Commissioning Group
• Controls Group

8
System Requirements(contd.)

• Requirements
• All the data that are possible to take should be
  taken.
• All the data that are taken should be saved for
  later analyses.
• All the operation should be recorded for later
  inspection.
• All the machine parameters and information about
  the machine components should be stored in the
  database.

9
System Requirements(contd.)

• Requirements(contd.)
• The Man-Machine Interface should be
  Operator-Friendly.
• The Programming Environment should be
  Programmer-Friendly.
• The Overall Response Time to an Operators
  Request should be less than a Second unless the
  Progress of the Process is Indicated.

10
Constraints

• Use CAMAC as an Equipment Interface
• Schedule
• Control Systems Final Design January, 1995
• Removal of MR equipment January, 1996
• Control Systems Installation September,
  1996
• Completed Hardware Installation March,
  1997
• KEKB Commissioning December, 1998

11
Constraints(contd.)

• Limited Man-Power
• 9 KEKB Controls Group Members
• 3 Physicists and 6 Engineers
• 12 Link-Persons from Other Groups
• 2 from Magnet and Power Supplies
• 2 from RF
• 2 from Vacuum
• 2 from Beam Transport
• 2 from Beam Monitor and Feedback
• 1 from Linac, 1 from Physics

12
Basic Concepts

• Standard Model Architecture
• International Standards
• CAMAC, VME, VXI, GPIB, FDDI, etc.
• Existing Software Environment EPICS
• Separate Computer-bus and Field-buses
• Link-person System Applications
• Out-Sourcing 5 People from Companies

13
Standard Model Architecture

• Presentation Layer (Server)
• Man-Machine Interface(OPI)
• Logging, Analyses, Alarm Displays, Database,
  Global Feedbacks
• Equipment Control Layer (IOCs)
• Data Acquisition, Sequence Control, Local
  Feedbacks
• Device Interface Layer (Field-buses)
• Hardware Interfaces

14
The most Important Points

• Presentation Layer
• Software Development Tools
• User/Programmer Friendly Tools
• Ergonomic Consoles
• LCDs, Macintoshs, PCs, PDPs, Flat Desks
• Equipment Control Layer
• VMEbus High Reliability, Flexibility
• Device Interface Layer
• CAMAC High Reliability, Well-known

15
Presentation Layer

• Operators Consoles X-Terminals
• Database Management ORACLE 7
• Alarm Generation/Recording
• Data Logging
• Data Display CATV Network
• Simulation SAD Program
• High-Speed Network
• FDDI, Distributed Shared-Memory Network,
• 100 Base/TX and 10 Base/T
• Gateway to KEK Laboratory Network acsad

16
Equipment Control Layer

• Provides Standard Interfaces
• CAMAC Serial Highway Drivers(Hytec)
• MXI-bus Drivers for VXI Main Frames(HP)
• ARCNET Drivers, GPIB Drivers(NI), etc.
• Computers VMEbus based IOCs
• FORCE CPU-40(MC68040)
• FORCE CPU-60(MC68060)
• FORCE PowerCore6603(PPC 603e)
• FORCE PowerCore6750(PPC 750)
• Operating System VxWorks

17
Device Interface Layer

• Standard Interfaces
• CAMAC RF and Vacuum
• VXI Beam Position Monitors
• GPIB RF, Vacuum, Magnet Readouts, etc.
• ARCNET Magnet Power Supplies
• TV Signal Switches
• MODBUS PLCs
• RS232 Vacuum Measuring Instruments

18
System Architecture(1)

• Functional Configuration

19
System Architecture(2)

• Actual Configuration

20
System Configuration
21
Central Control Room
22
abco1 Server Workstation

• PA-RISC 7200 Architecture
• 120MHz CPU Clock
• 4 CPUs
• 2GB of Main Memory
• 4GB Hard Disk Drives
• FDDI Interface
• 20GB RAID Disk

23
acsad Server Workstations

• Compaq Alpha Server
• 4 Alpha CPUs of 440MHz
• 6 Alpha CPUs of 330MHz
• 1 GB Memory
• 50 GB RAID
• FDDI Network Interface
• True 64 Unix Operating System

24
Former KEKB Control System
25
KEKB/PF-AR Control System
26
abco2 Server Workstation

• PA-RISC 8500 Architecture
• 440MHz CPU Clock
• 2 CPUs
• 1GB of Main Memory
• 36GB Hard Disk Drives
• FDDI Interface
• 140GB RAID Disk

27
IOC Configuration

• From CCR

28
VME IOC System

• Subrack
• Schroff 16 Slots Subrack
• Power Supply Module
• 259 Watts, 5V 35A, 12V 5A, -12V 2A
• 14 HP Wide, Schroff MPS8-7746
• System Monitor Module
• Mitsubishi DRSJ-01
• Remote System Reset
• DC Power Lines Monitor

29
VME Modules Used

• CPUs
• FORCE CPU-40 MC68040 33MHz
• FORCE CPU-60 MC68060 66MHz
• FORCE PowerCore 6603e PPC 603e
• FORCE PowerCore 6750 PPC-750 266MHz
• FORCE PowerCore 6750 PPC-750 400MHz
• CAMAC Serial Highway Driver
• HYTEC VSD 2992

30
VME Modules Used (contd.)

• GPIB Controller
• National Instruments GPIB 1014
• ARCNET Driver
• Advanet ARCNET-4
• VME-MXI Driver
• Hewlett Packard VME-MXI
• National Instruments VME-MXI II
• MODBUS Interface Controller

31
Number of IOCs and VME Modules
32
Equipment Connected

• 2,517 Magnet Power Supplies
• 176 ARCNET Segments
• 800 Beam Position Monitors

33
EPICS Records on IOCs

• 242,597 EPICS Records on 94 IOCs
• 25,147 EPICS Records on IOCMGD06
• 2,788.5 EPICS Records in Average
• 48,149 kB Max. Allocated Memory
• 1,324 kB Min. Allocated Memory
• 12,142 kB Ave. Allocated Memory

34
Hardware Summary

• Use Standard Buses
• Reliability VME, Compact PCI, CAMAC
• Not ISA, EISA, PCI, Proprietary Buses
• Separate CPU Bus from Field Buses
• Isolation, Analog Signal Handling
• Choose Components Carefully
• Power Supplies
• Capacitors, Fans, Connectors
• Use Ergonomic Equipment

35
Control Consoles
36
TRISTAN Consoles
37
KEKB and PF-AR Consoles
38
KEKB Consoles

• Voice Generator using Power Macintosh
• Multiple Screen Display for Consoles
• DELL Optiplex NX1
• Intel Pentium II 400MHz, 1GHz
• Colorgraphic Evolution 4
• Macintosh with Multiple Video Cards
• X-terminal using IBM Network Station
• Power PC 406(133MHz)
• 48MB Memory
• Disk-less / Fan-less System

39
Control Consoles

• 18 TFT LCDs(1280x1024 pixels)
• 16 TFT LCDs(1280x1024 pixels)
• 14 TFT LCDs(1024x800 pixels)
• Book-type Personal Computers
• DELL Optiplex NX1 and IBM Network Station
• Multi-Screen Display Controllers
• Wireless Keyboard/Mouse
• Power Macintosh with 2/4 Video Controllers
• Six 40 Plasma Displays as TV Monitors
• Easy to change Configuration

40
Software
41
Software

• Two Language Architecture
• TRISTAN Experiences NODAL and PCL
• Interpretive Language for Applications
• SAD and python
• Short Turn-around Time
• Safe Debuging Environment
• Not Dedicated Application Software Programmer
• Compiler Language for Low-Level Software
• C or C for Low-Level Software
• Quick Response Time
• Runs Fast
• Realizes Everything You Want

42
Software (contd.)

• Relational Database Software
• From the Designing Stage
• Put All the Accelerator Information
• Cable Connection Lists
• Equipment Parameters
• Physical Parameters Sizes, Weight,
• Calibration Constants, Fitting Curve Parameters,
  
• Histories
• Installation Dates, Repair Histories, Costs, .
• Property Numbers and more
• Generates EPICS Database Automatically

43
Software (contd.)

• EPICS Software Toolkit
• medm
• Generated from Relational Database
• Ah Alarm
• Ar Archiving
• SNL State Notation Language

44
EPICS Software Used

• medm
• SAD
• python

45
medm Applications
46
python Applications
47
SAD Applications
48
EPICS Databse Files
49
Linac Portable CA Server
KEKB CA Clients
Linac Console
Linac Main Controls
Linac CA Server
Linac Sub-Controls
KEKB IOCs
Linac Equipment
KEKB Equipment
50
Linac Portable Channel Access
51
Application Software in Use

• SAD 141
• medm 74
• python 42
• Miscellaneous 6
• Total 263

52
EPICS Records on IOCs

• 242,597 EPICS Records on 94 IOCs
• 25,147 EPICS Records on IOCMGD06
• 2,788.5 EPICS Records in Average
• 48,149 kB Max. Allocated Memory
• 1,324 kB Min. Allocated Memory
• 12,142 kB Ave. Allocated Memory

53
Software Summary

• EPICS R3.13 Toolkit on HP-UX 10.2
• medm for Man-Machine Interface
• X-Window Server Software on the Terminals
• python for Application Programming
• SAD for Accelerator Simulation / Operation
• VxWorks on IOCs
• Tornado Cross Software Development Sys.
• CAPFAST for EPICS Runtime Database Dev.
• ORACLE 7 for Relational Database
• Portable Channel Access Server for Linac Controls

54
VME CPU Board Benchmarks
55
CPU Board Benchmark Test

• FORCE Pcore 6604 FORCE CPU-40
• CPU Power PC 604e MC 68040
• Clock 200 MHz 25 MHz
• Memory 16 MB 16 MB
• L2 Cache 512 kB --

56
CPU Load Arising from Scanning Database

• PCore 6604 CPU-40
• 1.0 sec 0.60 5.8
• 0.5 sec 1.2 12
• 0.2 sec 3.0 27
• 0.1 sec 6.1 56

57
Transactions Time Required for Channel Access

• PCore 6604 CPU-40
• ca_search 953 us 1,189 us
• ca_put 23 us 109 us
• ca_get 58 us 118 us
• ca_putca_get 76 us 246 us

58
VME-MXI Problems
59
VME-MXI Problem

• HP VME-MXI Driver Module National Instruments
  Products
• FORCE PowerCore 6603 or 6750 CPU Modules Power
  PC
• Tandra Universe or Universe II Chips PCI-VME
  Bus Bridge Chips
• CPU Module Halts and Never Restarts Responding to
  the Reset Signal without Power is switched off

60
Improvements in ARCNET

• A HUB Box contains 3 ARCNET HUBs
• 33 HUB Boxes were Installed for 97 ARCNET Lines
• Decreases 102/Day/Line ARCNET Reconfiguration
  Errors down to Zero

61
Configuration of the Test Bench
CPU module
VME-MXI module
System Monitor module
VXI-MXI Controller
MPX module
MPX module
RF Voltmeter
CPU Modules tested FORCE CPU40 MC68040, 33
MHz FORCE CPU64 MC68060, 66 MHz FORCE
PowerCore 6603 PPC 603, Universe Chip FORCE
PowerCore 6750 PPC 750, 266 MHz, Univ. II
FORCE PowerCore 6750 PPC 750, 400 MHz, Univ. IIB
62
Normal and Abnormal Bus Cycles
Normal Bus Cycle Abnormal Bus Cycle
63
Abnormal Bus Cycle
64
Normal and Abnormal DTACK Signals
PowerCore 6603, with Universe Chip
Normal
Abnormal
65
Abnormal DTACK Signal
PowerCore 6750, 266 MHz, with Universe II
Normal
Abnormal
66
Realtime Linux
67
Linux

• Disk I/O causes 20 - 30 ms of Latency, possibly
  Latency can be around 100 ms or more.
• Cause
• Non-preemptive Kernel
• Scheduling Algorithm
• Interrupt Disabling

68
Non-Preemptive Kernel

• Interrupt

69
Preemptive Kernel
70
EPICS iocCore
Network
Channel Access
Run-time Database
Device Access Layer
I/O Buses (VME/PCI/ISA)
71
EPICS under L4-Linux
72
Linux System Calls
73
RT-thread Preempts Linux
74
RT-thread Calls Linux
75
What Happens if RT-thread Calls Preempted Linux?