Dr. Paul Dorsey

1
Using Periodic Audits to Prevent Catastrophic
Project Failure

• Dr. Paul Dorsey
• Dulcian, Inc.
• May 22, 2008

2
The Vasa

• Early 17th century (1628)
• The greatest ship of all time
• 3 years to build
• 2 gun decks with 64 cannons
• King Gustavus Adolphus of Sweden set the
  measurements.
• 1000 trees were required
• Triple laminated oak walls (18in/46cm thick)
• Mast 190 ft/57m
• Length 201 ft/61m
• Cost 5 of Swedens GNP

3
Maiden Voyage

• Set sail on a beautiful summer day
• August 10, 1628
• Passed the royal palace of Stockholm
• Fired proud salutes from cannons
• Sailed 1400 yards
• Gust of wind came
• Ship sank in less than 1 minute

4
Why is the Vasa relevant?

• What sank the Vasa sinks FMIS projects.
• We are still building Vasas.
• We cant stop bad decisions.
• We CAN stop ignoring them.
• If you spend 1M and fail, it is bad.
• If you spend 100M and fail, it impacts the whole
  organization
• If you spend 1B and fail, it impacts the
  country.
• If you are going to fail - fail cheap.

5
(No Transcript)
6
The Essence of Project Management

• Leading kids on a hike
• From time to time, climb a tree
• Check for obstacles
• Adjust direction
• Call everyone together
• "LET'S GO!"

7
Periodic Audits

• Critical success factor for failure prevention
• Must be external
• Developers cannot self-assess.
• Big, substantial effort
• Weak audit is worse than useless
• Provides illusion of safety

8
Audit Costs

• Delay project
• Expensive
• 5-10 of project cost
• Intrusive
• Annoying
• Politically costly

9
Team response to audit

• Project Manager
• "Why don't you trust me?"
• "Waste of time"
• Developers
• "We would rather be coding."
• Team may feel threatened
• If the team feels threatened, they probably have
  reason to be.
• If the team welcomes the audit.
• Sign of professional maturity

10
Audit Benefits

• Early detection of failure
• If a 300 million project fails after 20
  million, that is a huge savings.
• Validation of system architecture
• Second set of eyes
• Give team time to take stock
• Mid-project course correction

11
Auditor Independence

• Auditor must be told that there will be no chance
  for follow-on work.
• Otherwise the audit is suspect
• To enforce independence
• No economic attachment to the results
• No incentive to skew the results
• No relationship to the development team

• During the audit
• Limit contact with development team
• "Stockholm Syndrome"
• After the audit, auditors can help with the
  project plan.
• Auditor is the agent of the person who hired
  him/her (no one else)
• Only reports to the contract owner are objective.
• No professional standard or certification for IT
  auditors exists.
• Contract creates objectivity.

12
Audits are never 100 objective

• Auditors bring their own biases.
• There are IT "religions."
• .Net vs. Java
• "Thick database" vs. middle tier logic
• Service Oriented Architecture (SOA)
• Repository-based development
• Business rules
• Agile
• A professional with a different perspective can
  still detect a "Vasa."

13
Justifying Audit Cost

• An extensive audit requires approximately 10 of
  the total project cost.
• Industry project failure rate is 60.
• Example Audit at the halfway point of a 1
  million project
• Cost (50 x 1,000,000) x 10 50,000
• Benefit (50) x 1,000,00) x 60 300,000
• Given the high failure rate, audits are very
  cheap insurance.
• With other benefits, it is obvious that audits
  are a good deal.

14
Finding the right auditor

• Not internal
• Not from the same company as the developers
• Not dedicated auditors
• Must be real developers, DBAs, architects
• Must have built systems of similar scope and
  subject area

15
The Audit Team

• Project Manager
• Experience in the subject area with projects of
  similar scope
• Database Administrator (DBA)
• Experience with same platform and similar
  database size
• Application Architect
• Skill in the same area (Java, .Net, Oracle, etc.)
• Security Specialist
• Military, financial system or health care
  experience

16
Structure of the Audit

• Knowledge transfer
• Deep understanding
• As if auditor were taking over the project
• Understand the system before evaluating
• Infrastructure audit
• Evaluate the existing infrastructure to support
  system
• Every area needs to be assessed.
• Ability to meet current and future user
  requirements
• Auditor must understand the requirements
• Financial review

17
Detailed Structure of Audit

• B. Infrastructure Audit
• Examine from technical soundness perspective.
• Compare to current industry best practices
  document any discrepancies.
• System and User Documentation
• Data Model Audit
• Database Review
• User Interface (UI) Architecture Review
• Distributed System/ETL Audit
• Security Audit
• Hardware/Software/Networking Review
• Backup/Recovery Procedures
• Appropriateness of system upgrade mechanism
• C. Ability to Meet Current/Future Requirements
• Examine the current system requirements, identify
  use cases, and review for suitability,
  specifically
• Compare documented requirements to the existing
  use cases and how they are handled.
• Assess user satisfaction with the existing
  system.
• Are existing backup/recovery procedures
  sufficient to meet maximum downtime requirements?
• Assess system flexibility to meet new
  requirements.
• D. Financial review

• A. Knowledge Transfer
• Allows auditors to understand the entire system
  architecture as if they were taking over system
  development.
• The following areas should be reviewed for the
  knowledge transfer portion of the audit
• System Overview
• Data Model Walkthrough
• Review/Identify Transaction Use Cases
• Review User Interface Code Architecture
• Review Reporting Requirements/Architecture
• Review System Architecture
• Review System Installation/Upgrade Mechanism.
• Internal Control review
• User Access review
• Handling system bugs/enhancements
• Multi-Lingual capabilities
• Basic System Requirements
• Process Flows
• Custom Framework
• Performance
• Standards

18
Knowledge Transfer

• "Seek first to understand." Stephen Covey
• Learn enough to take over the process
• Architecture
• Data Model
• Use Cases
• Report Audit
• Configuration Management
• Internal Controls
• Documentation
• Training
• System may be bad enough that this is not
  possible.
• Do it anyway.
• Preventing the Vasa from sailing is hard work.

19
Infrastructure Audit

• Compare what was learned in the knowledge
  transfer with best practices
• Each area needs to be assessed
• Documentation
• Data model
• Database design
• User interface architecture
• Security
• Backup Recovery
• Configuration Management
• Internal Controls

• Identify weaknesses in each area
• Corrective actions
• Exposure
• Controls needed
• One mistake can sink the Vasa.
• System wont scale
• Security hole
• Inflexible design

20
Ability to Meet Requirements

• Requires careful use case documentation
• Assess user satisfaction
• Sit with users
• Survey
• Request queue is not a good measure. If system is
  poor, users give up.
• Assess each use case
• Functional requirements
• Performance
• Downtime
• Future requirements
• Flexibility
• Deployment

21
Financial Review

• Stewardship Audit
• If requirements change, price can balloon.
• Does this make sense?
• Sunk costs are "somewhat" relevant
• Measure decision quality

• Financial History

22
COTS Project Audits

• Not very different from custom projects
• COTS projects fail just as often.
• Review COTS architecture
• Be careful about how well COTS satisfies
  requirements
• COTS customizations can be VERY expensive.
• Customized COTS cannot be upgraded.

23
Audit Report

• 2-5 page Executive Summary Report
• Are we OK?
• 10-15 page CIO Report
• Are we OK? Why?
• 100 page detailed report
• What we did
• What we found
• What needs to be fixed
• Next steps

24
Acting on the Audit Report

• If report concludes "Vasa."
• Get a second opinion
• Let the development team respond
• Sunk costs are sunk costs.
• The amount of money budgeted for the project is
  irrelevant.
• "Upgrade" is a way to change direction without
  admitting failure.

25
Case Study Ethiopia FMIS

• Harvard University Project
• Small part of major financial reform
• 38 M USD over 12 years
• 3 M USD over 3 years for IT (very frugal)
• Harvard was ending the project and wanted to
  assess quality of system.
• Custom development project
• Dulcian was brought in to do the audit.

26
Audit Scope

• Standard audit
• As described in this presentation
• Total knowledge transfer and team back-up
• Support for any what if? scenarios
• Total system back-up

27
Audit Recommendations

• Maintain current system
• Upgrade system internals
• Business rules approach
• Oracle DBMS
• Ultra-light Web architecture

28
Audit Results

• Government and Harvard accepted recommendations.
• Maintain/evolve current system 1.5M USD
• Redesign architecture 1.5M USD
• Dual nature of the audit (audit handoff) made
  existing team very uncomfortable.
• Top three IT people resigned.

29
Conclusions

• Audits don't prevent failure they just catch
  failures earlier in the process.
• Audits don't replace good design.
• The audit may only help fail more small projects
  rather than one big project.
• Audits are resource intensive, expensive,
  annoying, politically charged.
• But they are cheaper than not doing them at all
  in the long run.
• Auditors must be kept independent.
• No follow-on work
• Both COTS and custom projects need audits.

30
Result of Audit

• Quite a good design
• Excellent documentation
• Very good developer coding
• Supported current requirements
• Some architecture portions needed modification.
• Database design issues
• No Foreign Keys
• Odd design (inherited from previous team)
• Poor performance for parts of system
• Scalability questions
• Would not work on Ethiopian internet due to
  slow/unreliable connections

31
Contact Information

• Dr. Paul Dorsey paul_dorsey_at_dulcian.com
• Dulcian website - www.dulcian.com

Latest book Oracle PL/SQL for Dummies