The Great Network Storm of

1
The Great Network Storm of 03

• A Cast Of Thousands

2
History

• In a May 31, 2003 message to tech-talk Lawrence
  Hoff (Brookhaven National Laboratory) mentioned
• memPartInfoGet bug which corrupted the heap (!!)
• a Wind-River Systems Software Problem Report of
  ill-formed Ethernet packets causing multicast
  storms.
• that BNL had worked around the bug by disabling
  IP forwarding

3
Summary

• HP89441 VSA sends malformed packet
• All devices on .2 subnet receive packet
• vxWorks bug causes IOCs to forward packet back to
  the network (!!)
• Exponential increase in network traffic
  (theoretically 24064 2x10158 packets)
• Control system grinds to a halt

4
The HP89441 as seenby the APS control system
5
Short-Term Solution

• Stop the IOC which was communicating with the
  HP89441.
• Disable packet forwarding in as many IOCs as
  possible
• IOCs with associated MVME162 card must have
  packet forwarding enabled
• Devices over which we have not control have
  packet forwarding enabled
• Any Ngt1 is going to make N64 a big number!

6
Real Solution

• Add packet filter to network switch
• Not possible with our switches
• Testing home-network 2-port router
• Fix bug in all vxWorks machines
• More than just IOCs
• Fix bug in HP89441
• HP/Agilent aware of problem

7
Warning!

• YOUR SYSTEM COULD BE NEXT!