(Ab-) using Round Robin Databases with Perl

1
(Ab-) using Round Robin Databases with Perl
Mike Schilli, Yahoo! 06/18/2008
2
Overview

• What are Round Robin Databases?
• RRD and Perl which module to use?
• Practical Examples

3
Round Robin Databases

• Archives of fixed size for unlimited data
• Overwrite old spots if full

4
Round Robin Archives
start
5
Round Robin Archives

• Overwrite if full

6
Round Robin Archives

• RRDs with multiple archives with different
  granularity

7
Data Flow with rrdtool
Data Source
Archive
8
Data Source Conversions
9
Steps and Heartbeat
10
Steps and Heartbeat
11
Steps and Heartbeat
12
Generous Heartbeat
13
Generous Heartbeat
14
Steps and Heartbeat

• Long story short
• Heartbeat gt Step
• if youre roughly feeding in step intervals.
• Example
• Step300 (5 min)
• Heartbeat600

15
Feeding PDPs to the Archive
16
Feeding PDPs to the Archive

• Consolidation Functions (MAX, AVERAGE, etc.)
• xfiles factor (allowed percentage of missing
  values)

17
(No Transcript)
18
Simple Example

• Web Orders
• 000 100
• 100 110
• 200 120
• 300 130
• 400 140
• 500 150

19
Simple Example
20
Simple Example

• rrdtool 'create' 'myrrdfile.rrd' '--start'
  '1211871600' '--step' '3600' 'DSordersGAUGE7200
  UU' 'RRAMAX0.51100'
• rrdtool 'update' 'myrrdfile.rrd' '1211875200100'
• rrdtool 'update' 'myrrdfile.rrd' '1211878800110'
• rrdtool 'update' 'myrrdfile.rrd' '1211882400120'
• rrdtool 'update' 'myrrdfile.rrd' '1211886000130'
• rrdtool 'update' 'myrrdfile.rrd' '1211889600140'
• rrdtool 'update' 'myrrdfile.rrd' '1211893200150'
• rrdtool 'graph' 'mygraph.png' '--vertical-label'
  'Orders per Hour' '--end' '1211893200' '--start'
  '1211871600' 'DEFdraw1myrrdfile.rrdordersMAX'
  'AREAdraw10000FFWeb Orders

21
RRDTool

• Arcane syntax
• Perl API RRDs.pm
• Pretty close to RRDTool syntax
• Calls rrdtool under the hood

22
To the Rescue RRDToolOO

• CPAN Module RRDToolOO
• Takes the drudgery out of the RRD syntax
• Does what you want by default
• Translates between Perl and RRD (watch it with
  debug on)
• Named parameters instead of fixed order

23
Simple Example

• !/usr/local/bin/perl -w
• use strict
• use RRDToolOO
• use DateTime
• my rrd RRDToolOO-gtnew(
• file gt "myrrdfile.rrd" )

24
Simple Example

• my start_dt DateTime-gttoday(
• time_zone gt "local" )
• my dt start_dt-gtclone()

25
Simple Example

• rrd-gtcreate(
• step gt 3600,
• start gt start_dt-gtepoch(),
• data_source gt name gt "orders",
• type gt "GAUGE",
• ,
• archive gt rows gt 100
• )

26
Simple Example

• for my hour (0..5)
• dt-gtadd(hours gt 1)
• rrd-gtupdate(
• time gt dt-gtepoch(),
• value gt 100 hour10,
• )

27
Simple Example

• rrd-gtgraph(
• image gt "mygraph.png",
• vertical_label gt 'Orders per Hour',
• start gt start_dt-gtepoch(),
• end gt dt-gtepoch(),
• draw gt
• type gt "area",
• color gt '0000FF',
• legend gt "Web Orders",
• )

28
Simple Example
29
RRDTool Commands

• rrdtool 'create' 'myrrdfile.rrd' '--start'
  '1211871600' '--step' '3600' 'DSordersGAUGE7200
  UU' 'RRAMAX0.51100'
• rrdtool 'update' 'myrrdfile.rrd' '1211875200100'
• rrdtool 'update' 'myrrdfile.rrd' '1211878800110'
• rrdtool 'update' 'myrrdfile.rrd' '1211882400120'
• rrdtool 'update' 'myrrdfile.rrd' '1211886000130'
• rrdtool 'update' 'myrrdfile.rrd' '1211889600140'
• rrdtool 'update' 'myrrdfile.rrd' '1211893200150'
• rrdtool 'graph' 'mygraph.png' '--vertical-label'
  'Orders per Hour' '--end' '1211893200' '--start'
  '1211871600' 'DEFdraw1myrrdfile.rrdordersMAX'
  'AREAdraw10000FFWeb Orders

30
Where RRDtool shines

• Graph time-based data
• Dont worry about scaling or axis text
• Stacked graphs

31
Where RRDtool sucks

• Cant add data for past events
• Cant add data twice at the same timestamp

32
Pitfalls

• Make sure that the first update happens after RRD
  start time
• Updates need to happen at increasing timestamps
• On large step sizes (1 year), limit the number of
  rows to fit within lt 2030.

33
Debugging with Log4perl

• use LogLog4perl qw(easy)
• LogLog4perl-gt
• easy_init( DEBUG )

34
Log4perl Output

• rrdtool 'create' 'myrrdfile.rrd' '--start'
  '1211871600' '--step' '3600' 'DSordersGAUGE7200
  UU' 'RRAMAX0.51100'
• rrdtool 'update' 'myrrdfile.rrd' '1211875200100'
• rrdtool 'update' 'myrrdfile.rrd' '1211878800110'
• rrdtool 'update' 'myrrdfile.rrd' '1211882400120'
• rrdtool 'update' 'myrrdfile.rrd' '1211886000130'
• rrdtool 'update' 'myrrdfile.rrd' '1211889600140'
• rrdtool 'update' 'myrrdfile.rrd' '1211893200150'
• rrdtool 'graph' 'mygraph.png' '--vertical-label'
  'Orders per Hour' '--end' '1211893200' '--start'
  '1211871600' 'DEFdraw1myrrdfile.rrdordersMAX'
  'AREAdraw10000FFWeb Orders

35
Example Stock Portfolio
36
Example Measure Power Usage

• Mastech MAS-345
• Serial Interface (RS-232)

37
Example Measure Power Usage
38
Example Measure Power Usage
39
Aberrant Behaviour
40
The End

• Thanks!

41
References

• RRDToolOO on CPAN
• RRDSimple on CPAN
• Portfolio Rendering article http//perlmeister.co
  m/talks/lm-200805.pdf
• RRDTool article http//www.linux-magazine.com/iss
  ue/44/Perl_RDDtool.pdf
• RRDTool article (in German) http//www.linux-maga
  zin.de/Artikel/ausgabe/2004/06/perl/perl.html

42

• Bonus Slides

43
Aberrant Behaviour

• Holt Winter Algorithm
• Data Smoothing/Prediction

44
Aberrant Behaviour

• Holt Winter Algorithm
• RRAs
• HWPREDICT
• SEASONAL
• DEVSEASONAL
• DEVPREDICT
• FAILURES
• RRAHWPREDICTrowsalphabetaseasonal_period

45
Aberrant Behaviour

• alpha (baseline) 0..1
• 1 recent observations carry greater weight
• 0 past observations carry greater weight
• beta (slope) 0..1
• gamma (seasonal) 0..1
• failures defaults to 7

46
Aberrant Behaviour

• Example
• alpha 0.50
• beta 0.50
• gamma 0.01
• seasonal_period 2
• threshold 7
• window_length 9