Instrumentation Workshop

1
Instrumentation Workshop

• An Operations Perspective

2
In preparing this talk, I have done several things

• I consulted with several members of other
  Operations Departments worldwide and asked them
  for some really good stories.
• So, dont assume that the stories I will tell are
  from Fermilab. I have changed the names and
  circumstances to protect the supposedly innocent.
• This also protects me from my own Instrumentation
  Dept.

3

• I have modified the stories so that you cannot
  tell where they are from.
• If you think this came from your lab, you are
  wrong. We have contributions that are worldwide.
• I have added the disclaimer that operations
  doesnt just refer to the Accelerator Operators,
  but includes Physicists, Engineers, etc.

4
Users

• In operations, we deal with users every day
• Users are another name for experimenters
• Users are all of our reasons for existence
• They whine and complain
• They are never appreciative
• Frequently, they dont tell us the whole story

5

• To be fair, the data for the graduate student is
  going to make or break their career.
• It is usually a year late when they finally get
  to take their data.
• They are usually running on coffee and days of no
  sleep in other words, they are often grumpy.
• Users are often a pain in the Rump

6

• In preparing for this talk, I realized that
  operations is a user also.
• we use instrumentation
• we whine and complain
• we are never appreciative
• We become exactly what we dont want to be
  users.

7

• Operations is a triad of three groups
• Physicists
• System and Support department experts
• Operators

8

• There are physicists
• who commission accelerators
• who improve existing accelerators
• who troubleshoot accelerator problems
• who do studies
• who frequently are the ones who request
  instrumentation
• who do the specification, and define where the
  instrumentation is needed

9

• There are System Experts within the different
  system and support groups
• who also tune beam and are in need of your
  instrumentation

10
Operators

• Most Operations groups have a high turnover rate
  shift work sucks.
• Biggest problem is training Operators.
• This means Operations groups need consistency and
  simplicity much more than they need leading-edge
  interfaces.

11

• Operations have to worry about all the machines
  and all systems including power supplies, RF,
  controls, vacuum, high voltage systems, water
  systems, etc. and all the safety rules and
  regulations.

12

• Operators have to deal with all types of people
• grumpy people
• people at all hours of the day and night
• people that dont want to come in
• arrogant people
• even good people
• and users

13

• Operators are generally fresh out of school, or
  the military are usually in a steep learning
  curve. For a while, Operators see the world as
  magical.
• Operators get to see everyones systems. This
  includes the good, the bad and the ugly.
• In general, Operators do not like to wake people
  in the middle of the night. They would rather
  solve the problems themselves.

14

• Operators want simple interfaces to deal with
  equipment. The have much to deal with and
  reducing complexity is crucial to their survival.
• Operators must be able to trust in the accuracy
  of instrumentation systems.
• Total failure, is better than partial failure.
• Pagers are a good thing.
• Yellow green color blindness among Operators is
  not that rare. Over the years I have worked with
  several color-blind Operators

15

• The purpose of all of us in the accelerator game
  is to provide beam to the experimenters (users).
  They are the reason for our existence

16
Instrumentation WE NEED YOU!

• We need your skills
• We need your cleverness
• Your equipment is our eyes and ears
• Without you we cannot see what we are doing.
• Your equipment, to us, is magical.
• There is never too little instrumentation.

17

• We whine and complain, we never give out
  compliments, but let me assure you WE NEED YOU
  AND YOUR INSTRUMENTATION

18

• A few years ago, the position detectors in a
  beamline did not work.
• Attempts were made to tune it up with just loss
  monitors. After 8 hours of tuning, the beamline
  was still not tuned up.
• After 10 hours, the position detectors were
  fixed.
• It then only took us 10 minutes to tune up that
  line.
• We truly miss your detectors when they dont
  work.

19
Story 1
In the early 70s there was a shutdown and a
magnet was moved in one of the beamlines. Once
back up, it was found that radiation losses
downstream of the magnet were horrific! After
hours of tuning, beamline experts were called for
long consultations over the phone. The beamline
was shut down, and an access was made to check
the magnet.
20
It was verified that the vacuum valves were all
open. Unable to find any problem, the beamline
experts were asked to come in. At 0700, a member
of the Instrumentation Department stopped by the
Control Room and asked how are the losses at the
downstream end of the beamline. Say What? Pray
tell why do you ask?
21
Just the day before, hed changed the gas in the
loss monitors to a different gas at the end of
the enclosure. He was just curious as to how the
losses looked.

• ON SHUTDOWN DAYS, WE NEED TO KNOW WHAT YOU ARE
  WORKING ONespecially something that might be an
  experiment

22
The 10 Decrees of Operations

• Thou shalt not install a major modification
  without informing the triad of operations people.

23
Story 2

• A shutdown day ended.
• Power supplies were racked in and turned on.
• Known-good save files were restored.
• Save-and-compare files were compared.
• Beam was turned on, but would not circulate.
• Loss monitors indicated high in one area.
• Work lists indicated no work in that area.
• Local trims and tune bumps moved some of the beam
  losses, but still no circulating turns

24

• After 4 hours of further work and calling in an
  expert, an old turn-by-turn system was
  reactivated and it was discovered there were
  actually lots of turns going around the ring.
• Everything was shut back down an access was made
  into the tunnel to investigate.
• It was discovered that a major detector that was
  the accustom to use for turn-by-turn beam had
  been disconnected by an instrumentation expert
  while doing calibration tests.

25

• Thou shalt not install a major modification
  without informing the triad of operations people
• Thou shalt not do any maintenance unless it is on
  some shutdown list, as even the most parasitic
  job can and eventually will come back and bite
  thee

26
Story 3

• Polarized electrodes were used to accelerate
  electrons produced by the ionization of the
  carbon jet onto a phosphor screen.
• In principle these electrodes were
  self-compensated the dipolar field was applied
  reversely outside the interaction region in order
  not to create any orbit distortion to the
  antiproton beam.
• Every day during one full week, the people in
  charge of the device would slightly increase the
  polarization of the electrodes, without telling
  anyone, certainly not operations.

27

• This was happening during an ultra-low energy
  operation period (5MeV) and every day operations
  would finely retune the working point to optimize
  the stochastic extraction, wondering what was the
  cause of the minute observed drifts.
• Then, on Friday evening, the machine stopped
  extracting altogether. A measurement of the
  working point showed that it had been moved so
  much that the machine was no longer on the
  extraction resonance at all!

28
What had happened was that after a full week of
high voltage tweeking the instruments designers
thought it would be safer to turn off their
device for the weekend.
29

• Thou shalt not install a major modification
  without informing the triad of operations people
• Thou shalt not do any maintenance unless it is on
  some shutdown list, as even the most parasitic
  job can and eventually will come back and bite
  thee
• Thou shalt not tweek the accelerator while it is
  running without informing operations, even though
  its parasitic. Parasites eventually kill the
  host.

30

• Again, the setting is a typical shutdown day
• Upon coming up, power supplies were turned on and
  save files were restored
• Beam would not circulate
• Losses were found to be unacceptably high in one
  area (loss monitors worked)
• Worklists indicated no work in this area
• Local trims and bumps managed to get some beam
  around, but it was at half the efficiency it
  should have been

Story 4
31

• Everything was turned off and locked out and an
  access was made
• It was found that a motorized detector was
  abnormally radioactive.
• Contact with the primary expert was not possible
  hed left for a long weekend at a remote cabin
  (no telephone).
• We contacted a second expert, who fortunately was
  available and came in.
• Using a small pocket laser (9.99 at Pamadi Dept.
  Store) the detector was a realigned.

32

• Everything was turned back on.
• Position readbacks were drastically different
  from pre-shutdown conditions, but beam was OK and
  losses were reasonable.
• It appears expert 1 recalibrated the readbacks
  and told no one. Operations restored things by
  putting the motorized detector in its normal
  position.

33

• Thou shalt not do any maintenance unless it is on
  some shutdown list, as even the most parasitic
  job can and eventually will come back and bite
  thee
• Thou shalt not tweek the accelerator while it is
  running without informing operations, even though
  its parasitic, parasites eventually kill the
  host
• Thou shalt not calibrate anything unless the
  machine experts and Operators have been told
  ahead of time

34
Story 5

• A fancy new motorized detector was designed and
  built an exciting project
• It came with fancy, complex, readbacks
• It had settable position limits, which allowed
  one to bracket the beam which was done in
  software, to allows setups for different modes
• It had a maximum (software) limit, since
  traveling 1mm too far would put it off its track
• A software bug sent it beyond its maximum limit
  and off its track it went

35
Motorized devices should utilize old fashioned
(hardware) limit switches, getting fancy with
software can be a recipe for disaster. Anything
that is motorized needs to have readbacks and
HARDWARE limit switches.
36

• Thou shalt not tweek the accelerator while it is
  running without informing operations, even though
  its parasitic, parasites eventually kill the
  host
• Thou shalt not calibrate anything unless the
  machine experts and Operators have been told
  ahead of time
• Thou shalt not covet new technology at the
  expense of reliability, for new is not always
  better.

37
Story 6

• Beam tuning was being done in a beamline
• Many motorized detectors were in the beam
• Data was being taken just before the beginning of
  a shutdown
• During the shutdown, a vacuum window ruptured and
  a massive pressure burst occurred
• The pressure burst found 4 detectors in the beam
  pipe at a 90º angle all were damaged by
  shrapnel.

38

• The ultimate fix was that the motorized detectors
  were connected to the safety system. Upon access
  then , the detectors automatically were removed
  from the beam.
• Detector designers should always assume the
  worst. There are many people who will climb on
  your detectors and do interesting things with the
  software.

39

• Thou shalt not calibrate anything unless the
  machine experts and Operators have been told
  ahead of time
• Thou shalt not covet new technology at the
  expense of reliability, for new is not always
  better.
• Thou shalt assume the world of operations and
  maintance is out to get thee. Defend thyself.
  Make thy systems resistant to slugs!

40
Story 7
In the old days, instrumentation was a detector
connected to some electronics, which was then
connected to a patch-panel in or near the Control
Room and subsequently connected to a scope. Now
instrumentation is interfaced to a Control
system. No matter how new the Control system
is, when it is installed it is already becoming
obsolete
41
Operations now view your instrumentation through
their control system. No matter what your
opinion of that control system is, your equipment
needs to be interfaced to it. Example
developing instrumentation through LabView is
fine but operators need the actual interface as
part of the operational control system.
42
The days of being only an Electronics Engineering
project are gone. Software and instrumentation
must interface with the current operations
control system. We cannot afford to have
separate multi-favorite control systems for the
operators to learn and use. Again they have
too much to to learn and to watch over. The last
thing they need is to have to deal with a new and
unique control system.
43

• Thou shalt not calibrate anything unless the
  machine experts and Operators have been told
  ahead of time
• Thou shalt not covet new technology at the
  expense of reliability, for new is not always
  better.
• Thou shalt assume the world of operations and
  maintance is out to get thee. Defend thyself.
  Make thy systems resistant to slugs!
• Thou shalt not covet thy own control system. Thy
  instrumentation must integrate into the current
  accelerator controls system

44
Story 8
In the past, proposals for fancy bells and
whistles for instrumentation systems have met
with a resounding NO!. We cannot do that. In
todays world, we have faster electronics. We
have more sophisticated instrumentation
techniques. We have more powerful high speed
computers. AND, we have more transfer of
knowledge such as this workshop. We NOW have
remarkable capabilities for those bells and
whistles.
45
However, this has lead to another series of
problems. Where, in the past we simply needed
ON/OFF, IN/OUT, and one or two timing channels.
NOW the application programs have gotten quite
complicated. The learning curve for the user has
gotten quite steep. The training time for the
use of these programs is quite complex.
46

• Also, remember, there are several different kinds
  of users
• The expert user, who can make the application
  program sing, dance, and fly.
• The everyday user, who just wants the data and
  wants to take it the same way every time.
• The studier who rarely uses the program and needs
  to be lead by the hand every time.

47
More than ever, the actual interface to the user
has become extremely important. The project is
NOT complete until the software user-interface is
complete. Lets repeat this The project is NOT
complete until the software user-interface is
complete.
48

• Thou shalt not covet new technology at the
  expense of reliability, for new is not always
  better.
• Thou shalt assume the world of operations and
  maintance is out to get thee. Defend thyself.
  Make thy systems resistant to slugs!
• Thou shalt not covet thy own control system. Thy
  instrumentation must integrate into the current
  accelerator controls system
• The project is NOT complete until the software
  user-interface is complete.

49
Story 9

• Comments from an operational point of view
• Instrumentation is used by the triad of
  operations personnel.
• The Physicists who request the instrumentation
  and have specific goals of what it needs to do.
• The Experts who have their own layout style.
• The Operators who have more programs they have to
  deal with than either of above two and who look
  for a specific and consistent flavor in the
  programs layout.

50

• Thou shalt assume the world of operations and
  maintance is out to get thee. Defend thyself.
  Make thy systems resistant to slugs!
• Thou shalt not covet thy own control system. Thy
  instrumentation must integrate into the current
  accelerator controls system
• The project is NOT complete until the software
  user-interface is complete.
• The human interface between operations and the
  instrumentation is an important part of the
  project. All 3 members of the operations triad
  need to be involved.

51
Story 10
Operations 1st goal is to keep the machine
running! Operators do not like waking up groups
of people during the night, however they are not
afraid to wake up anyone if the the problem is
causing accelerator downtime. Documentation that
would enable operations to do the initial
troubleshooting and repair is extremely
helpful. It is far better to have operations fix
a system than drag you in in the middle of the
night.
52

• An example would be a 4-5 page sheet or, better
  yet, a web page that contains the following
  information
• name of expert 1, 2 and 3
• name of programmer
• name of vacuum expert
• the experts phone and pager numbers
• where to find appropriate spare parts (fuses,
  spare chassis, control cards, etc.)
• how to properly troubleshoot the system (how to
  figure out its controls, local chassis and tunnel
  readbacks, etc.)

53

• a page or two of the most common failures and
  resolutions that have evolved over the years.
• It should be in operator english.
• Any informative documentation or training that
  enables the operations triad to solve the simpler
  problems, makes everyones life easier.

54

• Thou shalt not covet thy own control system. Thy
  instrumentation must integrate into the current
  accelerator controls system
• The project is NOT complete until the software
  user-interface is complete.
• The human interface between operations and the
  instrumentation is an important part of the
  project. All 3 members of the operations triad
  need to be involved.
• Be mindful that any information, documentation or
  training that enables operations to solve the
  simpler problem, makes every ones life easier.

55
Story 11
Develop a relationship with your users find
yourself a champion (or five). In every
operations organization there is always someone
that wants to learn more about what you are
doing. If you find that person, he or she can
be of great assistance to you. They can help you
in many ways.
56

• They can help you test new systems.
• They can give you operations viewpoint.
• They can assist you in getting information about
  your system passed around to all the other users.
• They can help you deal with finding room for
  your equipment.
• Space in the Control Room is at a premium they
  can help.

57

• At Fermilab we have a program wherein we take a
  new Physicist and put him or her on shift with
  one crew for a 5-week rotation. This serves many
  purposes.
• It allows us to brainwash the physicist
• It gives the physicist exposure to the
  operations systems
• It allows the physicist to see whats happening
  over the whole accelerator process

58

• The project is NOT complete until the software
  user-interface is complete.
• The human interface between operations and the
  instrumentation is an important part of the
  project. All 3 members of the operations triad
  need to be involved.
• Be mindful that any information, documentation or
  training that enables operations to solve the
  simpler problem, makes every ones life easier.
• Develop a relationship with your customers and
  know their perspective.

59
Instrumentation from an Operations viewpoint
-Instrumentation experts now have to deal with
mechanical support groups, safety groups, vacuum
system groups, control system groups, as well as
operations, - coordination as important as the
actual engineering itself.
60

• You have to worry about
• who are the keepers of your systems
• training others (as well as your own people
• documentation
• maintenance keeping old and almost-forgotten
  systems alive
• trying to do routine upkeep when the accelerator
  is running

61

• At the same time you have to
• keep your engineering skills sharp
• learn more about programming
• learn and deal with all the safety rules and
  regulations
• All the while, budgets are dropping, manpower is
  being reduced and bureaucracy is on the increase.

62
If it were easy, anyone could do it!
63
In summary What does operations want

• Good communications
• We need to know what youre up to it does
  effect operations
• We want you to understand that we cannot deal
  with multiple control systems as we have much to
  deal with, as well as we have a high turn-over
  rate. Instrumentation needs to be implemented in
  the existing control system.

64

• When developing new systems and you are using new
  technology, ask yourself whether you should have
  used new technology as opposed to tried-and-true
  methods. Newer isnt always better.
• Operations can easily be fooled by partial
  failures. Reliable or dead is what we want. A
  partially-working system is a real disaster for
  us.
• Good troubleshooting documentation enables us to
  do more and keep you home with your families.
• The interface program is the most important part
  of instrumentation to us. Work with us and make
  this interface something we both can use.

65

• Dont trust the rest of the world for we are
  slugs. Protect yourself.
• Finally thank you for the eyes you provide for
  us, tuning blindly can be a very ugly thing.