Custom Telemetry The Hackystat telemetry system includes a

1
Telemetry-based analysis of software engineering
product and process data

• Philip Johnson
• Collaborative Software Development Laboratory
• Information and Computer Sciences
• University of Hawaii
• johnson_at_hawaii.edu

2
Review

• For your projects, you have added tools and
  sensors to support collection and analysis of the
  following kinds of process and product metrics
• Metric Hackystat Sensor
• Active Time Eclipse Hackystat Sensor
• Review Time/Issues Eclipse Jupiter Sensor
• Builds Ant Build Sensor
• Size Ant LOCC Sensor
• Unit tests Ant JUnit Sensor
• Test coverage Ant JBlanket Sensor
• CVS commits/churn CVS Sensor

3
Daily Project Details

• The Daily Project Details analysis provides you
  with a summary of these process and product
  metrics for a single day on a single project

4
What is DPD good for?

• Use the DPD analysis (and it's associated alert)
  to help you accomplish the following
• Verify that sensors are working.
• Example If you worked yesterday, but the daily
  email doesn't indicate that, then check to see if
  something is wrong with the sensors.
• Detect sudden changes warranting attention.
• Example If coverage suddenly drops, check if
  somone committed code without corresponding unit
  tests.

5
What is DPD not good for?

• Very gradual changes/deterioration in
  process/product metrics.
• Example Gradual coverage decrease over six
  months.
• Process/product metrics that do not occur on
  daily basis
• Example Weekly reviews.
• Interactions between process/product metrics
• Example Increased review time leads to increased
  coverage and reduced build failure.

6
Software Project Telemetry

• An approach to monitoring trends in your process
  and product metrics.
• Allows you to detect gradual changes in metric
  data over days, weeks, or months.
• Allows monitoring of metric data that occurs less
  frequently (such as review data)
• Allows discovery of co-varying metrics.

7
Telemetry Example (Invocation)
8
Telemetry Example (Chart 1)
9
Telemetry Example (Chart 2)
10
Normal Invocation

• Select a report type
• Normally "Chart", but you can get Table, CSV, or
  XML output (for export to other tools).
• Select a project
• Telemetry analyses are focused on one project.
• Select a time interval and grain size
• Days, Weeks, or Months
• Select a predefined telemetry report type
• "Shrink-wrapped" analyses available for 413.
• Also possible to define your own (advanced)

11
TestSizeAndCoverage Telemetry Report

• Two Charts
• Number of Test Classes, Total number classes.
• Test coverage
• Process issues
• Is code size and coverage data getting collected
  regularly?
• Do the test classes change as the system classes
  change?
• How do these changes impact on coverage?
• How might testing be improved in future?

12
Example TestSizeAndCoverage
13
Example TestSizeAndCoverage
14
Development Progress Telemetry Report

• Four charts
• Cumulative, group-wide active time
• Cumulative, per-member active time
• Source code size (lines of code)
• Source code size (number of classes)
• Process issues
• Are code size and active time data being
  collected regularly?
• Is progress being reflected in additional code or
  not?
• Is Active Time distributed across all members?
• What is the relationship between changes in LOC
  and changes in classes?
• What does this data indicate about the progress
  of development?

15
Development Progress Example
16
Development Progress Example
17
Development Progress Example
18
Development Progress Example
19
Review Telemetry Report

• Two Charts
• Total review active time
• Per-member review active time
• (other charts coming soon)
• Process issues
• Is review data being collected?
• Is review being performed regularly?
• Are all team members devoting time to review?
• How can review be improved?
• Note that the 'week' grain size is best for this
  analysis.

20
Review Example
21
Review Example
22
Daily Process Stability Telemetry Report

• Eight charts
• Project-level
• Active Time
• Builds (pass/fail)
• Unit tests (pass/fail)
• CVS commits
• CVS Code churn
• Per-member
• Active Time
• Churn
• Commits

• Process issues
• Is system being worked on, evaluated, and
  committed regularly?
• Are members contributing regularly and equally?
• How can the daily progress on the system be made
  more efficient and effective?

23
Daily Process Stability Example
24
Daily Process Stability Example
25
Daily Process Stability Example
26
Daily Process Stability Example
27
Daily Process Stability Example
28
Daily Process Stability Example
29
Daily Process Stability Example
30
Daily Process Stability Example
31
Points to ponder

• From the preceding charts, what can you conclude
  about the development process and products of
  this team at this point in time?
• (Exclude Review data, from another project)
• What can you not conclude about the project from
  these charts?
• What would you recommend to this team?
• What would you want to ask this team?
• What other data would you like to see collected?

32
Custom Telemetry

• The Hackystat telemetry system includes a
  "Telemetry Definition Language" that supports the
  definition of telemetry "streams", "charts" and
  "reports".
• See the "Telemetry Management" command in the
  Preferences page for more information.
• You can define your own custom reports, and
  generate addition data streams (for example,
  LOC/hour, defect density, etc.)