Lecture 4b: Risks and Liabilities of Computerbased Systems

1
Lecture 4b Risks and Liabilities of
Computer-based Systems

• CSCI102 - Introduction to Information Technology
  B
• ITCS905 - Fundamentals of Information Technology

2
Overview

• Historical examples of software risks
• Implications of software complexity
• Risk assessment and management

3
Historical Examples

• Software errors
• Can KILL
• Indirectly
• Directly
• Cost MONEY
• Loss of equipment
• Loss of business

4
Software Aids and Abets Murder 1992

• A New Jersey inmate escaped from
  computer-monitored house arrest in the spring of
  1992
• He simply removed the rivets holding his
  electronic anklet together and went off to commit
  a murder

5
Software Aids and Abets Murder 1992

• A computer detected the tampering
• when it called a second computer to report the
  incident, the first computer received a busy
  signal and never called back

6
Radiation Machine Kills Four 1985 to 1987

• Faulty software in a Therac-25 radiation-treatment
  machine resulted in several cancer patients
  receiving lethal overdoses of radiation

7
Radiation Machine Kills Four 1985 to 1987

• Four patients died

8
 Radiation Machine Kills Four 1985 to 1987

• When their families sued, all the cases were
  settled out of court
• There were several errors,
• among them the failure of the programmer to
  detect a race condition (i.e., miscoordination
  between concurrent tasks)

9
 Radiation Machine Kills Four 1985 to 1987

• It was found that found that accidents occurred
  even after AECL thought it had fixed particular
  bugs
• "A lesson to be learned from the Therac-25 story
  is that focusing on particular software bugs is
  not the way to make a safe system
• "The basic mistakes here involved poor
  software-engineering practices and building a
  machine that relies on the software for safe
  operation

10
Hyphen Costs 80 Million 1962

• A probe launched from Cape Canaveral was set to
  go to Venus
• After takeoff, the unmanned rocket carrying the
  probe went off course
• NASA had to blow up the rocket to avoid
  endangering lives on earth

11
Hyphen Costs 80 Million 1962

• NASA later attributed the error to a faulty line
  of Fortran code
• Somehow a hyphen had been dropped from the
  guidance program loaded aboard the computer,
  allowing the flawed signals to command the rocket
  to veer left and nose down
• ...Suffice it to say, the first U.S. attempt at
  interplanetary flight failed for want of a hyphen

12
Hyphen Costs 80 Million 1962

• The vehicle cost more than 80 million, prompting
  Arthur C. Clarke to refer to the mission as the
  most expensive hyphen in history

13
ATT Long Distance Service Fails 1991

• In the summer of 1991, telephone outages occurred
  in local telephone systems in California and
  along the Eastern seaboard
• These breakdowns were all the fault of an error
  in signalling software

14
ATT Long Distance Service Fails 1991

• Right before the outages
• DSC Communications introduced a bug when it
  changed three lines of code in the
  several-million-line signalling program
• After this tiny change, nobody thought it
  necessary to retest the program

15
ATT Long Distance Service Fails 1991

• These switching errors in ATT's call-handling
  computers caused the company's long-distance
  network to go down for nine hours
• The meltdown affected thousands of services and
  was eventually traced to a single faulty line of
  code

16
Theres a Hole in the Bucket

• Small systems
• form part of larger systems
• A fault within a small part could result in a
  catastrophe later on

17
Theres a Hole in the Bucket

• Designers have an ethical responsibility to
  design the best system possible

18
Bugs

• Bugs exist because
• humans aren't perfect
• Since humans design and program hardware and
  software, mistakes are inevitable
• That's what computer and software vendors tell
  us, and it's partly true
• What they don't say is that software is buggier
  than it has to be

19
Bugs

• Why?
• Because time is money, especially in the software
  industry

20
Bugs

• This is how bugs are born

21
Bugs

• A software or hardware company sees a business
  opportunity and starts building a product to take
  advantage of that
• Long before development is finished, the company
  announces that the product is on the way

22
Bugs

• All the while pressuring the software engineers
  to add more and more features

23
Bugs

• Shareholders and venture capitalists clamour for
  quick delivery because that's when the company
  will see the biggest surge in sales
• Meanwhile, the quality-assurance division has to
  battle for sufficient bug-testing time

24
Bugs

• The simple fact is that you get the most
  revenues at the release of software,
• The faster you bring it out, the more money you
  make. You can always fix it later, when people
  howl. It's a fine line when to release something,
  and the industry accepts defects

25
What Is Risk Assessment and Management?

• Risk and uncertainty are fundamental elements of
  modern life
• They are ever present in the actions of human
  beings and they are frequently magnified in
  large-scale technological systems
• Risk and uncertainty must be managed effectively
  to protect people from injury and to permit the
  development of reliable, high-quality products

26
What Is Risk Assessment and Management?

• Risk is often defined as a measure of the
  probability and severity of adverse effects

27
What Is Risk Assessment and Management?

• In risk assessment, the analyst often attempts to
  answer the following set of triplet questions
• What can go wrong?
• What is the likelihood that it would go wrong?
• What are the consequences?

28
What Is Risk Assessment and Management?

• Answers to these questions help risk analysts
  identify, measure, quantify, and evaluate risks
  and their consequences and impacts

29
What Is Risk Assessment and Management?

• Risk management builds on the risk assessment
  process by seeking answers to a second set of
  three questions
• What can be done?
• What options are available and what are their
  associated trade-offs in terms of all costs,
  benefits, and risks?
• What are the impacts of current management
  decisions on future options?

30
What Is Risk Assessment and Management?

• To be effective and meaningful, risk management
  must be an integral part of the overall
  management of a system
• This is particularly important in the management
  of technological systems, where the failure of
  the system can be caused by the failure of the
  hardware, the software, the organization, or the
  humans