Agile Methodologies and Chaos Revisited

1
Agile MethodologiesandChaos Revisited

• William (Bill) Myers
• Cinergy Corporation
• Information Builders Summit 2004 User Conference
• May 2004

The statements and opinions herein are those of
the individual author. They are provided for
informational purposes only and should not be
relied upon for decision making.
2
(No Transcript)
3
Introduction

• Audience Flow
• General Agile reporting ( 1/3 rd )
• WebFocus Agile reports ( 1/3 rd )
• Agile Methodologies ( 1/3 rd )
• Philosophy Methods
• Management Coding
• Questions
• Follow-up questions at end
• Appendix
• Links
• Additional Details / starts at References slide
• Additional points
• Supplemental Handout

4
What is Agility

• Ability to adapt
• to Changing circumstances
• to Changing requirements
• Impact
• Improves business climate
• Provides competitive edge
• Quick gratification
• Customers Developers
• Faster Return on Investment

5
Agile Methodologies / Perspective

• How to Decide
• Relationships Responsibilities
• 80 / 20 rule
• Feature driven
• Customer stories
• Baseline high level
• Emerge/Converge
• Reduce reporting risks

Note Crystal Methodology IS NOT
RELATED to the reporting product
6
Reporting Population

• Super User
• Technical skills
• Understands their piece of the data
• May be tactical or strategic
• Strategic User
• Executive or analyst
• Needs highly summarized and graphed snapshots
• Limited technical skills
• Tactical User
• Needs up-to-minute detailed data to do their job
• Limited technical skills

7
Normal Report Development Process

• Demand that customer predict the future
• Forecast list of needed reports
• Requires visualization / no data available
• Data may support 20,000,000 variations
• Design signoff in blood
• Early code freeze
• Rigid Change Control
• More signoffs
• Complain about scope creep
• Accuse users of changing their mind
• List all changes as bugs
• Falsely implies a defective product

Requirements List
8
Agile Reporting

• Bounded Chaos
• Random actions inside of the boundary
• Self Building
• Developer builds 90
• User checks options
• Program finishes writing itself
• Program creates report
• Broad Scope
• Can replace millions of fixed reports
• Requirements can be hazy up front

9
Agile Reporting - Definitions

• Report (definition)
• Any output where user can not change data in
  database
• More than printer
• Core Options
• Core on every report, fixed
• Options added to report based on users
  selections columns totals
• Filters
• Limits input data
• Medium
• Web page, paper, excel, pivot table, PDF file,
  lists, graphs, etc
• Measures
• Changes the core - unusual

10
IBI Sample - ggorder2.fex

• As Installed
• Units by Product
• For one date
• Agile Modification
• Add sort1
• Standard front-end
• Add dollar value
• Immaterial to agile

11
Core Option - Web Page

• Shows total sales of each product by state
  sales rep
• Refer to handout
• Checking no boxes one number only

12
Agile ggorder2.fex

• Core only
• Boxes checked
• None

13
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• Vendor Name
• Product

14
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• Product
• Totals
• Store Name

15
Agile ggorder2.fex

• Same fex
• No changes
• Boxes checked
• State, City, Store Name (totals), Vendor
  Name, Product
• Options
• 362,889 formats
• ONE program
• Risks managed

16
Core Option Executive Demo

• IBI Sales Prototype
• Used Production databases
• Not standard IBI demo
• Specs and Core Option HTML were supplied
• 4 Days development
• Clear to all
• Reports
• Simple list Volumes Dollars (core option)
• Delta Change since yesterday (core option)
• Complex Delta Changes of averages
• 3-D Graphs
• Pies with drill downs

17
Exclusions

• Power not demonstrated
• Combined databases
• Reporting Cubes
• Scheduling Mailing
• Super-user interface

18
Risk Mitigation

• End-User reporting
• Normal recommendation
• Risks
• Mitigation for average user
• Super-users

19
Normal Reporting Recommendation

• Reporting is simple, use
• MS Access
• Excel
• Pivot Tables
• WebFocus
• Crystal Reports
• Etc
• Training
• Help
• Manuals
• Limited classes

20
Risks

• Complexity
• Database structure
• Queries
• All tests against production databases
• Overload servers network
• Over-reporting
• Double counting data
• Under-reporting
• Not including data

21
Mitigation

• Target the skill sets
• Developers
• Handle complexity
• Insure accuracy
• Insure efficiency
• Identify possible options
• Set up Core Option reports
• End Users
• Decide on core options during development
• Test during development
• Understand the results

22
Regular File Structure - WebFocus
23
Core Option Structure - WebFocus

• Build JavaScrips
• Options Template
• Options SQL database
• Template Fields text file
• Hidden Fields fairly static

24
File Naming Standards - WebFocus
Function Program Name Suffix
Pre-load Scr_Status_h.fex h
Get Core Options Scr_Status.htm
Report Scr_Status.fex
Drill Down Scr_Status_dr1.fex dr()
Report Caster Scr_Status_c1.fex c()
Menu Item Scr_Status_m1.fex m()

• Application prefix Name Suffix
• Drilldown can be called from many reports
• Name is first report written
• Program field names UpperLower_Case

25
Development Steps

• Build an original report
• Identify the CORE
• Use any BY statements for a sample set of options
• Add options list to database
• Write Pre-Load ( fex )
• Add filters now or later
• Write web page ( htm )
• Add filters now or later
• Add sort1 to original report
• You can add these front ends to existing reports

26
Option Files

• Common to all reports
• INCLUDE files (4)
• DATA files - static (2)
• DATA file field list unique to report (1)
• Structure in handout you add fields in database
• Unique to each report
• HTML web page
• Left side (options) standard except report title
  in heading
• Right side (filters) unique for each report
• Listed in supplemental handout

27
Option Files (list)

• Pre-Load (scr_status_h.fex )
• opt_page_common.fex
• OptionTbls.sql
• opt_hid.dat
• opt_mng.dat
• Web Page (scr_status.htm)
• Sample.htm ( copy JavaScript and certain
  sections of HTML)
• !IBI.FIL.OPT_MNG
• !IBI.FIL.OPTJ_CLR
• !IBI.FIL.OPTJ_BOX
• !IBI.FIL.OPT_HIDN
• Report (scr_status .fex )
• sort1
• opt_parent_drilldown.fex
• Drilldown (scr_status_dr1.fex )
• opt_drill_headings.fex
• opt_drill_where .fex
• wqlist

28
Pre-Load (scr_status_h.fex )
-SET ECHOOFF - Set the report
program name -SET Rpt_Pgm
scr_status - Set default date
-SET riskdate
EDIT(MDYY,'99/99/9999') -SET startmo
EDIT(MDYY,'99') -SET startyr
EDIT(MDYY,'9999') -RUN - Add
standard javascript and html code
-INCLUDE opt_page_common -RUN
- Open web page
-HTMLFORM scr_status
29
Option Filter Web Page
See summit_2004_demo1.htm in references
Amper Variables from Pre-Load fex
Standard Option Files
HOLD File from Pre-Load fex
30
Web Page Java Script section
31
JavaScript - Static
32
JavaScript - Dynamic
33
Web Page HTML Check Boxes
34
Hidden HTML Variable List
35
Option Filter HTML section
ltbody background"/images/Gray_Textured1152.gif"
bgcolor"FFFFFF" text"000080" link"0000FF"
onload"reloadSort('summit_2004_demo')"gt --------
--------------------------------------------------
-------------- lttd aligncenter width"45"
VALIGN"top"gt lttable cellpadding1
cellspacing1 border0gt !IBI.FIL.OPTJ_BOX
lt/tablegt lt/tdgt --------------------------------
---------------------------------------- lttdgt
ltp align"left"gt ltinput type"button"
value"Submit" name"bttn_submit" style"font-
weight bold" onclick"javascriptpresubmit(thi
s.form)"gtlt/Pgt nbsp lt/tdgt ---------------------
--------------------------------------------------
- lt/tablegt ltinput type'hidden' name'IBIAPP_app'
value""gt ltinput type'hidden' name'IBIF_ex'
value"summit_2004_demo"gt !IBI.FIL.OPT_HIDN
lt/FORMgt
36
Report ( fex )
DEFINE FILE TRADES Cost/P12.2CB VOLUME
Price Value/P12.2 VOLUME STRIKE_PRICE
Net/P12.2 Value - Cost END TABLE FILE TRADES
SUM VOLUME AS 'Volume' Cost
Value sort1 - add other BY statements
here END
37
Report with Drilldown ( fex )
TABLE FILE TRADES SUM VOLUME AS 'Volume'
Cost Value sort1 ON TABLE COLUMN-TOTAL
VOLUME Cost Value ON TABLE SET ONLINE-FMT frmt
ON
TABLE SET STYLE
UNITSIN,
PAGESIZE'Letter', LEFTMARGIN0.50,
RIGHTMARGIN0.50,
TOPMARGIN0.50,
BOTTOMMARGIN0.50, SQUEEZEON,
ORIENTATIONPORTRAIT,

TYPEREPORT, GRIDOFF, FONT'TIMES NEW
ROMAN', SIZE10, COLORBLACK,
BACKCOLORNONE, STYLENORMAL,

-SET optdrillfile
'summit_2004_demo_dr1' -SET
coredrillfile 'summit_2004_demo_dr1'
-INCLUDE opt_parent_drilldown.fex ENDSTYLE

END
38
Drilldown ( fex )
TABLE FILE TRADES PRINT Cost Value
Net HEADING "Deals supporting Summary" "
" -INCLUDE opt_drill_headings.fex -
wqlist field values to exclude surrounding
quote marks -SET wqlist'''PRODYEAR'' '' ''
''PRODMONTH'' '' ''' -INCLUDE opt_drill_where.fex
END
39
What is Agile Development?

• Set of policies and conventions
• Controls risk
• Puts project into safety zone
• Agile Methodologies
• Changing environments
• Philosophies
• Products, Personal relationships, Timing,
  Documentation
• Methods
• Approach Principles
• Project management
• Programming low level practices
• Traditional Methodologies
• Rigid, known environments
• Static processes
• Structured Methods
• Waterfall project management clones

40
Agile Philosophieshttp//www.agilealliance.org

• Agile Manifesto
• Individuals and interactions
• over processes and tools
• Working software
• over comprehensive documentation
• Customer collaboration
• over contract negotiation
• Responding to change
• over following a plan
• Quote from consortium While there is value in
  the items on the right (sub-topics above), we
  value the items on the left (major topics) more.
  

41
General Agile Principles

• Deliver something useful to customer
• Nothing counts until you deliver software
• Deliver frequently
• Four to eight weeks
• Encourage collaboration
• Active customer involvement is imperative!
  find a champion
• Daily meetings very short
• Provide technical excellence
• Rely on the skills of your people
• Do the simplest thing possible
• Balance flexibility and structure
• Self-correcting teams
• Dont let people suffer
• Install when business needs it technically
  feasible
• WebFocus is an Agile product

42
Adaptability Scale
Remember Sarbanes-Oxley
43
Project Management Styles

• Command Control
• Defined process
• Plan what you expect make predictions
• Enforce the predictions
• Change Control
• to manage change
• Leadership Collaboration
• Empirical process
• Start with a plan
• Change plan frequently
• Inspection Adaptation
• To control change

44
Waterfall Methodsand clones

• Process
• Sequential
• Analysis, design, construct, test, implement
• Rigid
• Paper intensive
• Good with 1960s mainframes with cards
• When Useful
• Known processes
• Hardware vendor software upgrades
• Little creativity required

45
Many Agile Methods

• What methodology to use ?
• Crystal Crystal Light ( helps with decision )
• Project management
• Scrum ( broadest, variable end-point )
• DSDM Model ( construction oriented )
• Lean Development (domain, 80/20 approach )
• Feature-Driven Development - FDD (most
  structured)
• Adaptive
• Extreme Programming XP
• Programming oriented
• Extreme Programming - XP
• Customer Stories, programs, working conditions
• Agile reporting ( core option )

46
Agile Methodologies

• Agile Software Development Ecosystems
• Reference book web site
• by Jim Highsmith
• http//www.jimhighsmith.com
• Tom DeMarco
• Agility 1
• everything else 0
• Numerous books
• Highlights only

47
Observations - Agile Conference

• Easy to go wrong
• Mis-interpretation, mis-implementation, missed
  point
• No silver bullet
• One size does not fit all
• Beware too much documentation
• Offload legal requirements from developers
• Always on time, on scope, on budget
• Leads to mediocrity ( no risks taken )
• Cultural change
• from - military Command Control
• to cooperative Leadership Collaboration
• Productivity leaps
• 77 function points / month when industry average
  2 pts / mo
• WebFocus fits agile development very well

48
Reasons for Failure

• One Size Methodology Fits All
• Intolerant Methodology
• Embellished
• Do frivolous tasks
• Ignore necessary tasks
• Heavy
• Untried
• Tried Once
• Limited applicability
• Champion
• Failure to Tune
• Distractions
• Dates cant be met

49
Crystal Methodology Theory ( NOT RELATED to
Crystal Reporting )

• Self-adapting "shrink-to-fit," software
  development methodologies
• Different set of policies conventions for each
  project
• Projects are sensitive to people issues
• Crystal is people-centric
• other methodologies may be
• process-centric
• architecture-centric
• tool-centric
• Keys
• Communication intensive
• Masses of paper do not improve communication
• Experiment based
• Requires feedback loops
• Theory of Methodologies

50
Crystal Communication

• Effectiveness of different methods
• Courtesy of Alistair Cockburn
• http//alistair.cockburn.us/crystal/articles/cpanf
  ocisd/characterizingpeopleasnonlinear.html

People are Non-Linear First-Order Components in
Software Development
51
Crystal - Model

• Model
• Risk to end-user, Size, Complexity
• Tables present questions to ask
• To establish processes controls
• When to change
• Weight is costly
• Heavy methods for large teams
• More ceremony for critical deliverables
• Avoid frivolous tasks
• Efficiency is expendable
• Non-bottleneck activities Rarely used items
• Crystal Methodologies
• Reference books web site
• Alistair Cockburn Author
• http//alistair.cockburn.us/crystal

52
SCRUM

• Principles
• Do the right thing (from customers view)
• Deal with reality, not artifacts or fixed plans
• Short iterations
• Usually four weeks
• Be prepared to implement at the end of an
  iteration
• Allow changes
• Keep everything visible
• Concentrate on time remaining - not time spent
• Highlights and pinpoints responsibilities
• Self organization
• Creates sense of obligation and teamwork
• Extensive collaboration
• Changes role of Project Leader
• Certification available

53
SCRUM

• Broad application
• Iterative, lightweight project management method
• Software AND non-software projects
• Based on manufacturing Process Control theory
• Adaptable
• Can be implemented
• Beginning of a project
• Middle of a project
• Product development effort that is in trouble
• More information
• Ken Schwaber and Mike Beedle
• http//www.controlchaos.com/

54
SCRUM Concepts

• Courtesy of Ken Schwaber
• http//www.controlchaos.com/Scrumo.htm

55
SCRUM

• Full visibility
• Priority list
• Items in development cycle ( sprint )
• Short, daily meetings ( 15 minutes )
• Work hours remaining in sprint
• Assignments, who what
• Responsibility Authority
• Anyone can add to priority list
• Business manages priorities
• Developers determine what can be done
• Project Leader removes impediments
• Removes developers from politics

56
SCRUM - Impact

• Backlog
• Scope Creep no such thing
• No request ever lost
• It may never get a high enough priority
• Business owner only one who can remove it
• Return on Investment
• Install option available
• Software completed to date available / immediate
  return
• Option to stop
• Save development costs
• Perfect for fixed price projects
• Reorder deliverables list no cost
• Substitute deliverables with equal effort no
  cost
• Stop Development clause easy to apply
• Big competitive advantage

57
DSDM Model

• Principles
• Iterative incremental development
• Changes during development are reversible
• Requirements baselined at high level
• Minimum details to continue
• Testing throughout lifecycle
• Developers AND business users
• More Information
• Developed in UK
• Grew out of RAD approach
• http//www.dsdm.org/

58
Lean Development Model

• Most strategic oriented of Agile methods
• Principles
• Domain instead of point solutions
• 80 today better than 100 tomorrow
• Everything is changeable
• Minimal is essential
• Needs determine the technology
• Best value for money
• More information
• http//www.poppendieck.com/
• http//www.leanconstruction.org/

59
Adaptive Software Model

• Software development
• Complex adaptive system
• Like stock market or flock of birds
• Circumstances generate emergent capabilities
• Total skills exceeds sum of parts
• Only emergent capabilities tames complexity
• Software organizations
• Must create the proper circumstances
• for emergence
• of superior capabilities

60
Adaptive Software Model

• Principles
• Leadership Collaboration
• Speculate, collaborate, learn
• Drop militaristic Command and Control (stifles
  emergence)
• Cycle through phases many times, converging
• Tasks not listed in plan
• Components at end of plan are listed
• Mission directed, feature driven, iterative
• time boxed, risk driven, change tolerant

61
Feature-Driven Development (FDD)

• Process
• One-time
• Quick, high value, low dollar
• Develop overall domain model
• Build feature list
• Plan by feature
• Iterative
• Design by feature
• Two week cycles building components
• More information
• http//www.featuredrivendevelopment.com/
• http//www.nebulon.com/fdd/

62
Extreme Programming - XP

• Project management
• Collect customer stories
• Estimate prioritize
• Small iterative development cycles
• Development
• Test-first development
• Write test cases before coding
• Collective code ownership
• Refactoring
• Pair programming
• Daily standup meetings
• Many processes rules
• More information
• http//www.extremeprogramming.org
• http//www.objectmentor.com/home

63
Agile - Review

• Positives
• Software delivered
• No death marches
• Dangers
• Process itself becomes the goal
• Placebo effect
• Key point is missed
• SCRUM
• Immediately useful
• Becoming broadly popular
• Classes around the world
• Hundreds certified

64
The End( References )

• http//www.agilealliance.org - Agile Alliance
• http//www.jimhighsmith.com - Agile overview
• http//alistair.cockburn.us/crystal - Crystal
• http//www.controlchaos.com/Scrumo.htm - Scrum
• http//www.dsdm.org - DSDM
• http//www.poppendieck.com - LEAN
• http//www.leanconstruction.org - LEAN
• http//www.santafe.edu - Complex Adaptive
• http//www.featuredrivendevelopment.com - FDD
• http//www.nebulon.com/fdd - FDD
• http//www.extremeprogramming.org - EX
  (Extreme)
• http//www.objectmentor.com/home - EX (Extreme)

65
Agile Manifesto - Highlights

• Nothing counts until you deliver software
• Deliver every four to eight weeks
• Direct contact - customers developers
• Daily contact and communication
• Let champions drive projects
• Maintain a constant pace
• Agile processes enhance quality
• Teams self-correct
• Never forget the goal - software

66
Agile Manifesto (1)Follow these principles for
Developing Software

• Satisfy customerswith valuable software
• The highest priority is to satisfy the customer
  through early and continuous delivery of valuable
  software.
• Welcome change requests at any time
• Making changes quickly can be a competitive
  advantage
• Welcome changing requirements, even late in
  development. Agile processes harness change for
  the customer's competitive advantage.

67
Agile Manifesto (2)Follow these principles for
Developing Software

• Deliver working software frequently
• Dont let people suffer
• Deliver working software frequently, from a
  couple of weeks to a couple of months, with a
  preference to the shorter timescale.
• Daily Contact
• Customers developers
• Avoid intermediaries
• Business people and developers must work together
  daily throughout the project.

68
Agile Manifesto (3)Follow these principles for
Developing Software

• Use Champions
• Build projects around motivated individuals. Give
  them the environment and support they need, and
  trust them to get the job done.
• Team Communication
• Avoid long or large meetings
• The most efficient and effective method of
  conveying information to and within a development
  team is face-to-face conversation.

69
Agile Manifesto (4)Follow these principles for
Developing Software

• Working software is primary measure
• Models, documentation, use cases do not count
• Working software is the primary measure of
  progress
• Maintain a constant pace indefinitely
• No Death Marches
• Agile processes promote sustainable development.
  The sponsors, developers, and users should be
  able to maintain a constant pace indefinitely.

70
Agile Manifesto (5)Follow these principles for
Developing Software

• Quality enhances agility
• Late and post-implementation changes easier
• Continuous attention to technical excellence and
  good design enhances agility.
• Self-organizing teams increase quality
• Volunteers produce better results
• Members look out for and help each other
• The best architectures, requirements, and designs
  emerge from self-organizing teams.

71
Agile Manifesto (6)Follow these principles for
Developing Software

• Simplicity
• Do less get more
• The art of maximizing the amount of work not done
  -- is essential.
• Teams self-correct
• Not continuously will drive people nuts
• At regular intervals, the team reflects on how to
  become more effective, then tunes and adjusts its
  behavior accordingly.

72
Scrum - Definitions

• Product Backlog
• Prioritized list of items
• Business Product Owner sets priorities
• List of Functional requirements, Nonfunctional
  requirements, Issues
• Contains preliminary estimate
• Changes constantly
• Items in a sprint locked down
• Time-box
• Period of time that can not be exceeded
• Sprint
• Time-box of 30 days
• Sprint backlog defines delivery list
• Product item
• Internal support item (i.e. database, system doc)
• Work remaining burndown graph

73
Scrum - Players

• Scrum
• Product Owner
• Adds items to product backlog list
• Set priorities
• Team
• Determines sprint list
• Develops software
• Stakeholder
• Funded project, will use it, affected by it
• Requests enhancements
• ScrumMaster
• Responsible for Scrum process
• Removes impediments
• Other interested parties
• IT Management
• Manpower allocation
• Budgets Billing
• Senior Business Management
• Best use of corporate resources

74
Scrum - Milestones

• Sprint Planning meeting ( two 4 hour segments )
• Product owner Team identifies backlog items
  to be delivered in 30 day sprint / Team commits
• Team plans details in Sprint Backlog
• Daily Scrum Meeting ( 15 minutes )
• Team Review prior day, identify impediments,
  plan day
• Sprint Review Meeting ( 4 hours )
• Team demonstrates completed functionality- to
  Product Owner and other interested parties
• Sprint Retrospective meeting ( 3 hours )
• Team how to improve next sprint ormake more
  enjoyable

75
SCRUM demo app Priority Page
76
SCRUM database - goals

• Software development
• Collect new requests
• Manage product backlog
• Manage Scrum process
• Reduce administrative load on developers
• Reduce or eliminate most status reporting needs
• Support Management
• Provide data for manpower planning
• Provide status to everyone
• Priority in queue
• Currently being worked on
• Remaining effort
• Provide historical information
• Time tracking mechanism for billing
• Answer What did you do for us last year

77
Agile Reporting Core Options

• Core
• Units Ordered
• Options
• State
• Contact
• Product
• Package
• Totals on Contact, State