Agile Methodologies and Chaos Revisited

1
Agile MethodologiesandChaos Revisited

• William (Bill) Myers
• Cinergy Corporation
• Information Builders Summit 2004 User Conference
• May 2004

The statements and opinions herein are those of
the individual author. They are provided for
informational purposes only and should not be
relied upon for decision making.
2
Introduction

• Audience Flow
• Agile Methodologies
• Philosophy Methods
• Management Coding
• General Agile reporting
• Questions
• Follow-up questions at end
• Appendix
• Links
• Additional Details / starts at References slide
• Additional points
• Supplemental Handout

3
What is Agility

• Ability to adapt
• to Changing circumstances
• to Changing requirements
• Impact
• Improves business climate
• Provides competitive edge
• Quick gratification
• Customers Developers
• Faster Return on Investment

4
Point of View - Concerning Risk

• Traditional Methods
• Risks for IT
• Errors
• Delivery on budget, on time, on scope
• Business risks generally ignored
• Agile Methods
• Business risks
• Competitive timeliness
• Effectiveness / the right thing
• IT risks
• Errors
• Budget, time, scope

5
Agile Methodologies / Perspective

• How to Decide
• Relationships Responsibilities
• 80 / 20 rule
• Feature driven
• Customer stories
• Baseline high level
• Emerge/Converge
• Reduce reporting risks

Note Crystal Methodology IS NOT
RELATED to the reporting product
6
What is Agile Development?

• Set of policies and conventions
• Controls risk
• Puts project into safety zone
• Agile Methodologies
• Changing environments
• Philosophies
• Products, Personal relationships, Timing,
  Documentation
• Methods
• Approach Principles
• Project management
• Programming low level practices
• Traditional Methodologies
• Rigid, known environments
• Static processes
• Structured Methods
• Waterfall project management clones

7
Agile Philosophieshttp//www.agilealliance.org

• Agile Manifesto
• Individuals and interactions
• over processes and tools
• Working software
• over comprehensive documentation
• Customer collaboration
• over contract negotiation
• Responding to change
• over following a plan
• Quote from consortium While there is value in
  the items on the right (sub-topics above), we
  value the items on the left (major topics) more.
  

8
General Agile Principles

• Deliver something useful to customer
• Nothing counts until you deliver software
• Deliver frequently
• Four to eight weeks
• Encourage collaboration
• Active customer involvement is imperative!
  find a champion
• Daily meetings very short
• Provide technical excellence
• Rely on the skills of your people
• Do the simplest thing possible
• Balance flexibility and structure
• Self-correcting teams
• Dont let people suffer
• Install when business needs it technically
  feasible

9
Agile Manifesto - Highlights

• Nothing counts until you deliver software
• Deliver every four to eight weeks
• Direct contact - customers developers
• Daily contact and communication
• Let champions drive projects
• Maintain a constant pace
• Agile processes enhance quality
• Teams self-correct
• Never forget the goal - software

10
Adaptability Scale
Remember Sarbanes-Oxley
11
Project Management Styles

• Command Control
• Defined process
• Plan what you expect make predictions
• Enforce the predictions
• Change Control
• to manage change
• Leadership Collaboration
• Empirical process
• Start with a plan
• Change plan frequently
• Inspection Adaptation
• To control change

12
Waterfall Methodsand clones

• Process
• Sequential
• Analysis, design, construct, test, implement
• Rigid
• Paper intensive
• Good with 1960s mainframes with cards
• When Useful
• Known processes
• Hardware vendor software upgrades
• Little creativity required

13
Many Agile Methods

• What methodology to use ?
• Crystal Crystal Light ( helps with decision )
• Project management
• Scrum ( broadest, variable end-point )
• DSDM Model ( construction oriented )
• Lean Development (domain, 80/20 approach )
• Feature-Driven Development - FDD (most
  structured)
• Adaptive
• Extreme Programming XP
• Programming oriented
• Extreme Programming - XP
• Customer Stories, programs, working conditions
• Agile reporting ( core option )

14
Agile Methodologies

• Agile Software Development Ecosystems
• Reference book web site
• by Jim Highsmith
• http//www.jimhighsmith.com
• Tom DeMarco
• Agility 1
• everything else 0
• Numerous books
• Highlights only

15
Observations - Agile Conference

• Easy to go wrong
• Mis-interpretation, mis-implementation, missed
  point
• No silver bullet
• One size does not fit all
• Beware too much documentation
• Offload legal requirements from developers
• Always on time, on scope, on budget
• Leads to mediocrity ( no risks taken )
• Cultural change
• from - military Command Control
• to cooperative Leadership Collaboration
• Productivity leaps
• 77 function points / month when industry average
  2 pts / mo
• WebFocus fits agile development very well

16
Reasons for Failure

• One Size Methodology Fits All
• Intolerant Methodology
• Embellished
• Do frivolous tasks
• Ignore necessary tasks
• Heavy
• Untried
• Tried Once
• Limited applicability
• Champion
• Failure to Tune
• Distractions
• Dates cant be met

17
Crystal Methodology Theory ( NOT RELATED to
Crystal Reporting )

• Self-adapting "shrink-to-fit," software
  development methodologies
• Different set of policies conventions for each
  project
• Projects are sensitive to people issues
• Crystal is people-centric
• other methodologies may be
• process-centric
• architecture-centric
• tool-centric
• Keys
• Communication intensive
• Masses of paper do not improve communication
• Experiment based
• Requires feedback loops
• Theory of Methodologies

18
Crystal Communication

• Effectiveness of different methods
• Courtesy of Alistair Cockburn
• http//alistair.cockburn.us/crystal/articles/cpanf
  ocisd/characterizingpeopleasnonlinear.html

People are Non-Linear First-Order Components in
Software Development
19
Crystal - Model

• Model
• Risk to end-user, Size, Complexity
• Tables present questions to ask
• To establish processes controls
• When to change
• Weight is costly
• Heavy methods for large teams
• More ceremony for critical deliverables
• Avoid frivolous tasks
• Efficiency is expendable
• Non-bottleneck activities Rarely used items
• Crystal Methodologies
• Reference books web site
• Alistair Cockburn Author
• http//alistair.cockburn.us/crystal

20
SCRUM

• Principles
• Do the right thing (from customers view)
• Deal with reality, not artifacts or fixed plans
• Short iterations
• Usually four weeks
• Be prepared to implement at the end of an
  iteration
• Allow changes
• Keep everything visible
• Concentrate on time remaining - not time spent
• Highlights and pinpoints responsibilities
• Self organization
• Creates sense of obligation and teamwork
• Extensive collaboration
• Changes role of Project Leader
• Certification available

21
SCRUM

• Broad application
• Iterative, lightweight project management method
• Software AND non-software projects
• Based on manufacturing Process Control theory
• Adaptable
• Can be implemented
• Beginning of a project
• Middle of a project
• Product development effort that is in trouble
• More information
• Ken Schwaber and Mike Beedle
• http//www.controlchaos.com/

22
SCRUM Concepts

• Courtesy of Ken Schwaber
• http//www.controlchaos.com/Scrumo.htm

23
SCRUM

• Full visibility
• Priority list
• Items in development cycle ( sprint )
• Short, daily meetings ( 15 minutes )
• Work hours remaining in sprint
• Assignments, who what
• Responsibility Authority
• Anyone can add to priority list
• Business manages priorities
• Developers determine what can be done
• Project Leader removes impediments
• Removes developers from politics

24
SCRUM - Impact

• Backlog
• Scope Creep no such thing
• No request ever lost
• It may never get a high enough priority
• Business owner only one who can remove it
• Return on Investment
• Install option available
• Software completed to date available / immediate
  return
• Option to stop
• Save development costs
• Perfect for fixed price projects
• Reorder deliverables list no cost
• Substitute deliverables with equal effort no
  cost
• Stop Development clause easy to apply
• Big competitive advantage

25
Scrum - Definitions

• Product Backlog
• Prioritized list of items
• Business Product Owner sets priorities
• List of Functional requirements, Nonfunctional
  requirements, Issues
• Contains preliminary estimate
• Changes constantly
• Items in a sprint locked down
• Time-box
• Period of time that can not be exceeded
• Sprint
• Time-box of 30 days
• Sprint backlog defines delivery list
• Product item
• Internal support item (i.e. database, system doc)
• Work remaining burndown graph

26
Scrum - Players

• Scrum
• Product Owner
• Adds items to product backlog list
• Set priorities
• Team
• Determines sprint list
• Develops software
• Stakeholder
• Funded project, will use it, affected by it
• Requests enhancements
• ScrumMaster
• Responsible for Scrum process
• Removes impediments
• Other interested parties
• IT Management
• Manpower allocation
• Budgets Billing
• Senior Business Management
• Best use of corporate resources

27
Scrum - Milestones

• Sprint Planning meeting ( two 4 hour segments )
• Product owner Team identifies backlog items
  to be delivered in 30 day sprint / Team commits
• Team plans details in Sprint Backlog
• Daily Scrum Meeting ( 15 minutes )
• Team Review prior day, identify impediments,
  plan day
• Sprint Review Meeting ( 4 hours )
• Team demonstrates completed functionality- to
  Product Owner and other interested parties
• Sprint Retrospective meeting ( 3 hours )
• Team how to improve next sprint ormake more
  enjoyable

28
SCRUM database - goals

• Software development
• Collect new requests
• Manage product backlog
• Manage Scrum process
• Reduce administrative load on developers
• Reduce or eliminate most status reporting needs
• Support Management
• Provide data for manpower planning
• Provide status to everyone
• Priority in queue
• Currently being worked on
• Remaining effort
• Provide historical information
• Time tracking mechanism for billing
• Answer What did you do for us last year

29
SCRUM Process Flow
30
SCRUM Database Model
31
SCRUM demo app Priority Page
32
DSDM Model

• Principles
• Iterative incremental development
• Changes during development are reversible
• Requirements baselined at high level
• Minimum details to continue
• Testing throughout lifecycle
• Developers AND business users
• More Information
• Developed in UK
• Grew out of RAD approach
• http//www.dsdm.org/

33
Lean Development Model

• Most strategic oriented of Agile methods
• Principles
• Domain instead of point solutions
• 80 today better than 100 tomorrow
• Everything is changeable
• Minimal is essential
• Needs determine the technology
• Best value for money
• More information
• http//www.poppendieck.com/
• http//www.leanconstruction.org/

34
Adaptive Software Model

• Software development
• Complex adaptive system
• Like stock market or flock of birds
• Circumstances generate emergent capabilities
• Total skills exceeds sum of parts
• Only emergent capabilities tames complexity
• Software organizations
• Must create the proper circumstances
• for emergence
• of superior capabilities

35
Adaptive Software Model

• Principles
• Leadership Collaboration
• Speculate, collaborate, learn
• Drop militaristic Command and Control (stifles
  emergence)
• Cycle through phases many times, converging
• Tasks not listed in plan
• Components at end of plan are listed
• Mission directed, feature driven, iterative
• time boxed, risk driven, change tolerant

36
Feature-Driven Development (FDD)

• Process
• One-time
• Quick, high value, low dollar
• Develop overall domain model
• Build feature list
• Plan by feature
• Iterative
• Design by feature
• Two week cycles building components
• More information
• http//www.featuredrivendevelopment.com/
• http//www.nebulon.com/fdd/

37
Extreme Programming - XP

• Project management
• Collect customer stories
• Estimate prioritize
• Small iterative development cycles
• Development
• Test-first development
• Write test cases before coding
• Collective code ownership
• Refactoring
• Pair programming
• Daily standup meetings
• Many processes rules
• More information
• http//www.extremeprogramming.org
• http//www.objectmentor.com/home

38
Agile - Review

• Positives
• Software delivered
• No death marches
• Dangers
• Process itself becomes the goal
• Placebo effect
• Key point is missed
• SCRUM
• Immediately useful
• Becoming broadly popular
• Classes around the world
• Hundreds certified

39
Reporting Population

• Super User
• Technical skills
• Understands their piece of the data
• May be tactical or strategic
• Strategic User
• Executive or analyst
• Needs highly summarized and graphed snapshots
• Limited technical skills
• Tactical User
• Needs up-to-minute detailed data to do their job
• Limited technical skills

40
Normal Report Development Process

• Demand that customer predict the future
• Forecast list of needed reports
• Requires visualization / no data available
• Data may support 20,000,000 variations
• Design signoff in blood
• Early code freeze
• Rigid Change Control
• More signoffs
• Complain about scope creep
• Accuse users of changing their mind
• List all changes as bugs
• Falsely implies a defective product

Requirements List
41
Agile Reporting

• Bounded Chaos
• Random actions inside of the boundary
• Self Building
• Developer builds 90
• User checks options
• Program finishes writing itself
• Program creates report
• Broad Scope
• Can replace millions of fixed reports
• Requirements can be hazy up front

42
Agile Reporting - Definitions

• Report (definition)
• Any output where user can not change data in
  database
• More than printer
• Core Options
• Core on every report, fixed
• Options added to report based on users
  selections columns totals
• Filters
• Limits input data
• Medium
• Web page, paper, excel, pivot table, PDF file,
  lists, graphs, etc
• Measures
• Changes the core - unusual

43
IBI Sample - ggorder2.fex

• As Installed
• Units by Product
• For one date
• Agile Modification
• Add sort1
• Standard front-end
• Add dollar value
• Immaterial to agile

44
Core Option - Web Page

• Shows total sales of each product by state
  sales rep
• Checking no boxes one number only

45
Agile ggorder2.fex

• Core only
• Boxes checked
• None

46
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• Vendor Name
• Product

47
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• Product
• Totals
• Store Name

48
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• State, City, Store Name (totals), Vendor
  Name, Product
• Options
• 362,889 formats
• ONE program
• Risks managed

49
Agile Reporting Core Options

• Core
• Units Ordered
• Options
• State
• Contact
• Product
• Package
• Totals on Contact, State

50
Core Option Executive Demo

• IBI Sales Prototype
• Used Production databases
• Not standard IBI demo
• Specs and Core Option HTML were supplied
• 4 Days development
• Clear to all
• Reports
• Simple list Volumes Dollars (agile core
  option)
• Delta Change since yesterday (agile core
  option)
• Complex Delta Changes of averages
• 3-D Graphs
• Pies with drill downs

51
Exclusions

• Power not demonstrated
• Combined databases
• Reporting Cubes
• Scheduling Mailing
• Super-user interface

52
Risk Mitigation

• End-User reporting
• Normal recommendation
• Risks
• Mitigation for average user
• Super-users

53
Normal Reporting Recommendation

• Reporting is simple, use
• MS Access
• Excel
• Pivot Tables
• WebFocus
• Crystal Reports
• Etc
• Training
• Help
• Manuals
• Limited classes

54
Risks

• Complexity
• Database structure
• Queries
• All tests against production databases
• Overload servers network
• Over-reporting
• Double counting data
• Under-reporting
• Not including data

55
Mitigation

• Target the skill sets
• Developers
• Handle complexity
• Insure accuracy
• Insure efficiency
• Identify possible options
• Set up Core Option reports
• End Users
• Decide on core options during development
• Test during development
• Understand the results

56
Products Languages

• Effort
• Easy
• Varying levels of difficulty
• Brute force vs- fits vendors product well
• Power
• Is agile
• Appears to be agile
• Partly agile
• Beware of pretty vs- power

57
Regular File Structure - WebFocus
58
Core Option Structure - WebFocus

• Build JavaScrips
• Options Template
• Options SQL database
• Template Fields text file
• Hidden Fields fairly static

59
Report ( fex )
DEFINE FILE TRADES Cost/P12.2CB VOLUME
Price Value/P12.2 VOLUME STRIKE_PRICE
Net/P12.2 Value - Cost END TABLE FILE TRADES
SUM VOLUME AS 'Volume' Cost
Value sort1 - add other BY statements
here END
60
The End( References )

• http//www.agilealliance.org - Agile Alliance
• http//www.jimhighsmith.com - Agile overview
• http//alistair.cockburn.us/crystal - Crystal
• http//www.controlchaos.com/Scrumo.htm - Scrum
• http//www.dsdm.org - DSDM
• http//www.poppendieck.com - LEAN
• http//www.leanconstruction.org - LEAN
• http//www.santafe.edu - Complex Adaptive
• http//www.featuredrivendevelopment.com - FDD
• http//www.nebulon.com/fdd - FDD
• http//www.extremeprogramming.org - EX
  (Extreme)
• http//www.objectmentor.com/home - EX (Extreme)

61
Agile Manifesto (1)Follow these principles for
Developing Software

• Satisfy customerswith valuable software
• The highest priority is to satisfy the customer
  through early and continuous delivery of valuable
  software.
• Welcome change requests at any time
• Making changes quickly can be a competitive
  advantage
• Welcome changing requirements, even late in
  development. Agile processes harness change for
  the customer's competitive advantage.

62
Agile Manifesto (2)Follow these principles for
Developing Software

• Deliver working software frequently
• Dont let people suffer
• Deliver working software frequently, from a
  couple of weeks to a couple of months, with a
  preference to the shorter timescale.
• Daily Contact
• Customers developers
• Avoid intermediaries
• Business people and developers must work together
  daily throughout the project.

63
Agile Manifesto (3)Follow these principles for
Developing Software

• Use Champions
• Build projects around motivated individuals. Give
  them the environment and support they need, and
  trust them to get the job done.
• Team Communication
• Avoid long or large meetings
• The most efficient and effective method of
  conveying information to and within a development
  team is face-to-face conversation.

64
Agile Manifesto (4)Follow these principles for
Developing Software

• Working software is primary measure
• Models, documentation, use cases do not count
• Working software is the primary measure of
  progress
• Maintain a constant pace indefinitely
• No Death Marches
• Agile processes promote sustainable development.
  The sponsors, developers, and users should be
  able to maintain a constant pace indefinitely.

65
Agile Manifesto (5)Follow these principles for
Developing Software

• Quality enhances agility
• Late and post-implementation changes easier
• Continuous attention to technical excellence and
  good design enhances agility.
• Self-organizing teams increase quality
• Volunteers produce better results
• Members look out for and help each other
• The best architectures, requirements, and designs
  emerge from self-organizing teams.

66
Agile Manifesto (6)Follow these principles for
Developing Software

• Simplicity
• Do less get more
• The art of maximizing the amount of work not done
  -- is essential.
• Teams self-correct
• Not continuously will drive people nuts
• At regular intervals, the team reflects on how to
  become more effective, then tunes and adjusts its
  behavior accordingly.

67
File Naming Standards - WebFocus
Function Program Name Suffix
Pre-load Scr_Status_h.fex h
Get Core Options Scr_Status.htm
Report Scr_Status.fex
Drill Down Scr_Status_dr1.fex dr()
Report Caster Scr_Status_c1.fex c()
Menu Item Scr_Status_m1.fex m()

• Application prefix Name Suffix
• Drilldown can be called from many reports
• Name is first report written
• Program field names UpperLower_Case