A Heartbeat Mechanism and its Application in Gigascope

1
A Heartbeat Mechanism and its Application in
Gigascope

• Johnson, Muthukrishnan, Shkapenyuk, Spatscheck
• Presented by
• Joseph Frate and John Russo

2
Introduction

• Unblocking aggregation, join and union operators
• Limit scope of output tuples that an input tuple
  can affect.
• Two techniques
• Define queries over window of input stream
• Applicable to continuous query systems for
  monitoring
• Timestamp mechanism
• Applicable to data reduction applications

3
Gigascope

• A high-performance streaming database for network
  monitoring
• Some fields of input stream behave as timestamps
• Locality of input tuple
• Aggregation must have timestamp as one of group
  by fields
• Join query relates timestamp fields of both
  inputs
• Merge operator
• Union operator
• Preserves timestamp property of one of the fields

4
Timestamps and Gigascope

• Effective as long as progress is made in all
  input streams
• Join or merge operators can stall if one input
  stream stalls, causing systems failure

5
Example

• Sites monitored by Gigascope have multiple
  gigabit connections to the Internet.
• One or more is a backup link through which
  traffic can be diverted if primary link fails
• All links are monitored simultaneously
• Since primary link has gigabit traffic and backup
  link has no traffic, merge operator will quickly
  overflow.
• Presence of tuples carries timestamps, absence
  does not

6
Focus of Paper

• Authors propose use of heartbeats or punctuation
  to unblock operators
• Heartbeats originate at source query operators
  and propagate throughout query
• Timestamp punctuations are generated at source
  query nodes and inferred at every other operator
• Punctuated heartbeats unblock operators that
  would otherwise be blocked
• Focus is on multi-stream operators (joins and
  merges)
• Significantly reduces memory load for join and
  merge operators

7
Related Work

• Heartbeat
• Widely used for fault-tolerance in distributed
  systems
• Remote nodes send periodic heartbeats to let
  other nodes know that they are alive
• Heartbeats are also used in distributed DSMS
• Stream Punctuations
• Embed special marks in stream to indicate end of
  subset of data
• Previous work in heartbeat mechanisms with
  streaming data have focused on enforcing ordering
  of timestamps of tuples before query processing

8
Gigascope Architecture

• Stream-only database
• No continuous queries, thus no windows to unblock
  queries
• Streams are labeled with timestampness, such as
  monotone increasing. Used by query planner to
  unblock blocking operators

9
Gigascope Architecture (Continued)

• Aggregation query
• One of the group by attributes must have
  timestampness.
• When this attribute changes, all groups and
  aggregates are flushed to the operators output
• We can define this as an epoch. Flush occurs at
  the end of each epoch.
• Example here time is labeled as monotone
  increasing
• select tb, srcIP, count() from TCP
• group by time/60 as tb, srcOP
• In this query, we count the packets from each
  source IP address during 60 second epochs

10
Gigascope Architecture (Continued)

• Merge Operator
• Union of two streams A and B
• A and B must have same schema
• Both streams must have a timestamp field, such as
  t, on which to merge
• If tuples on A have a larger value of t than
  those on B, tuples on A are buffered until B
  catches up

11
Gigascope Architecture (Continued)

• Join Query
• Must contain a join predicate such as A.trB.ts
  or A.tr/3 B.ts2
• Relates timestamp field from A with B
• Input streams are buffered so that streams match
  up on the timestamp.

12
Two-Level Query Architecture

• Low level used for data reduction
• High level performs complex processing
• Controls high streaming rates
• Data streams from NICs are placed in a ring
  buffer.
• These are called source streams

13
Two-Level Query Architecture (continued)

• Since volumes are too large to provide copy to
  each query, Gigascope creates a subquery
• For example
• Query Q to be executed over source stream S
• Gigascope creates a subquery q which directly
  accesses S
• Q is transformed into Q0 which is executed over
  the stream output of q

14
Two-Level Query Architecture (continued)

• Low-level subqueries are called LFTAs
• Fast, lightweight data reduction queries
• Objective is to quickly process high volume data
  stream in order to minimize buffer requirements
• Expensive processing is performed on output of
  low level queries
• Smaller volume
• Easily buffered

15
Two-Level Query Architecture (continued)

• Much of subquery processing can be performed on
  the NIC itself
• Low-level aggregation uses a fixed-size hash
  table for maintaining groups in group by
• If a collision occurs in hash table, old group is
  ejected as a tuple and new group replaces it in
  its slot
• Similar methodology as subaggregates and
  superaggregates in data cube computations
• Higher level queries complete aggregation

16
Two-Level Query Architecture (continued)

• Traffic shaping policies are implemented in order
  to spread out processing load
• Aggregation operator uses a slow flush to emit
  tuples when aggregation epoch changes

17
Heartbeats to Unblock Streaming Operators

• Gigascope heartbeats are produced by low-level
  query operators
• Propagated throughout query DAG
• Incur same queuing delays as tuples
• System performance monitoring
• Aids in detecting failed nodes
• Stream punctuation mechanism is implemented
• Injects special temporal update tuples into
  operator output streams
• Informs receiving operator of end of subset of
  data
• Authors first attempted on-demand generation of
  temporal update tuples
• Authors settled upon approach of using heartbeats
  to carry temporal update tuples

18
Schema of Temporal Update Tuple

• Identical to regular tuple
• All attributes marked as temporal are initialized
  with values that will not violate the ordering
  properties
• i.e. timebucket is marked as temporal increasing
• Temporal update tuple with timebuckett is
  received by an operator
• All future tuples will have timebucket t
• Non-temporal attributes are ignored by receiving
  operators
• Goal is to generate temporal attribute values
  aggressively and set them to highest possible
  value

19
Heartbeat Generation at LFTA LevelLow-level
Streaming Operators

• Read data directly from source data streams
  (packets sniffed from NICs)
• Use filtering, projection and aggregation to
  reduce amount of data in a stream before passing
  it to higher-level nodes
• Two types of streaming operators selection and
  aggregation
• Multi-query optimization through prefilters

20
Heartbeat Generation at LFTA LevelLow-level
Streaming Operators

• Normal mode of operation is to block waiting for
  new tuples to be posted to NICs ring buffer.
• Once a tuple is in the buffer, it is processed
• Wait is periodically interrupted to generate a
  punctuated heartbeat

21
Heartbeats in select LFTAs

• Selection, projection and transformation is
  performed by LFTAs on packets arriving from data
  stream source.
• If predicate is satisfied, output tuple is
  generated according to projection list.
• A few additions
• Modify accept_tuple to save all temporal
  attribute values referenced in select clause
• Whenever a request is received to generate a
  temporal update tuple, use maximum of saved value
  of temporal attributes and a value saved by
  prefilter to infer value of temporal update tuple

22
Heartbeats in Aggregation LFTAs

• Group by and aggregation functionality is
  implemented using direct-mapped hash table
• Collision results in ejected tuple being sent to
  output stream
• High-level aggregation node completes aggregation
  of partial results produced by LFTA
• Flushing occurs whenever incoming tuple advances
  the epoch.
• Slow-flush is used to avoid overflow of buffers
• Gradually emits tuples as new tuples arrive
• Last seen temporal values in input stream are
  saved, similar to select.
• This value is used to generate temporal update
  tuples
• Also maintains value of last temporal attribute
  of last tuple flushed

23
Heartbeats in Aggregation LFTAs

• Whenever a request for a heartbeat is received,
  uses following formula in order to insure that
  heartbeat does not violate temporal attribute
  ordering properties
• if we have unflushed tuples
• use the value of last flushed tuple
• else
• use max of saved value of the temporal
• attributes and the value saved by the
• prefilter

24
Using System Time for Temporal Values

• When a link has no traffic for a long time,
  heartbeat is generated using inference from
  system time.
• Skew between system clock and buffering has to be
  accounted for when setting up Gigascope system

25
High Level Query Nodes (HFTA)

• Second level of query execution (two levels)
• LFTA for low-level filtering and sub-queries
• HFTA for complex query processing
• Selection
• Multiple types of aggregation
• Stream merge
• Inner and outer join
• Data from multiple streams (HFTAs, LFTAs)

26
HFTA Execution

• Normal mode of execution
• Block while waiting for new tuples to arrive
• Determines which operator to apply to that stream
• Process the incoming tuple for that operator
• Route output (if operator required)
• Also regularly receives temporal update tuples
• Operators interpret temporal update tuples
• Operators use these tuples to unblock themselves

27
HFTA Heartbeats in selection

• Mostly identical to selection LFTAs
• Unpack field values referenced in query predicate
• Check if predicate is satisfied
• If satisfied, generate projection list
• Difference can receive temporal update tuples
  (in addition to regular data tuples)
• Temporal update tuple is received
• Operator updates the saved values of all temporal
  attributes referenced in query Select
• Generates new temporal update tuple
• Rest of normal tuple processing is bypassed

28
HFTA Heartbeats in aggregation and sampling

• Non-blocking operator
• Aggregates data within a time window (epoch)
• Epoch defined by values of temporal group-by
  attributes
• Required to keep all groups and corresponding
  aggregates until end of epoch
• Then flushes to output stream
• Slow flush to avoid overflow
• One output tuple for each input tuple
• Change in epoch ? immediate flush

29
HFTA Heartbeats in stream merge

• Union of two streams R and S in a way that
  preserves the ordering properties of the temporal
  attributes
• R and S must have same schema
• Both must have a temporal field to merge on
• Stream buffers until the other catches up (if
  difference in time fields)

30
HFTA Heartbeats in join

• Relate two data streams by timestamp
• e.g., R.tr 2 S.ts
• INNER, LEFT, RIGHT, and FULL OUTER
• Minimum timestamps for R and S maintained
• Buffers input streams to ensure they match up by
  timestamp predicate
• May include temporal attributes

31
Other heartbeat applications

• Initial goal of heartbeats was to collect
  statistics about nodes in a distributed setting
• Main focus of paper is to use heartbeats to carry
  stream punctuation
• Found that there are other uses for heartbeats

32
Fault tolerance

• Heartbeat mechanism used to detect node failures
  in distributed systems
• Nodes must ping to indicate still alive
• Gigascope slightly different
• Heartbeats periodically generated by low-level
  queries
• Propagated upward through query execution DAG
• Constant flow of heartbeat tuples indicates if
  low-level sub-query is responding or not
  responding
• No heartbeat over specified amount of time ?
  query has failed and recovery is initiated

33
System performance analysis

• Heartbeat message subject to same routing as
  other data in network
• Heartbeat visit all nodes
• Statistics gathered on every node visit
• Full trace of all nodes
• Can be used to identify poor-performing nodes

34
Distributed query optimization

• Automated tools to utilize collected statistics
• Re-optimize query execution plans (eliminate
  identified bottlenecks)
• Other statistics
• Predicate selectivities
• Data arrival rates
• Tuple processing costs

35
Performance evaluation

• Experiments conducted on live network feed
• Queries monitor three network interfaces
• main1 and main2 DAG4.3GE Gigabit Ethernet
  interfaces (see bulk of traffic)
• control 100Mbit interface
• Approximately 100,000 packets per second
• 400 Mbits/sec
• Dual-processor 2.8 GHz P4, 4GB RAM, FreeBSD 4.10
• Focus is to unblock streaming operators
• high-rate main links
• low-rate backup links (control)

36
Evaluation Unblocking stream merge using
heartbeats

• Effect of heartbeats on memory usage of queries
  that use stream merge operator
• Used the following query
• SELECT tb, protocol, srcIP, destIP, srcPort,
  destPort, count()
• FROM DataProtocol
• GROUP BY time/10 as tb, protocol, srcIP,
  destIP, srcPort, destPort

37
Evaluation Unblocking stream merge using
heartbeats

• Query computes the number of packets observed in
  different flows in 10 second time buckets
• Query planner merges three streams

38
Evaluation Unblocking stream merge using
heartbeats

• When control link has no traffic, large amount of
  data buffered (from other streams)
• Varied heartbeat interval
• From 1 second to 30 seconds
• 5 second increments

39
Evaluation Unblocking stream merge using
heartbeats

• Heartbeats successfully unblock the stream merge
  operators
• Higher intervals ? more state maintained
• Also results in more data being flushed when an
  epoch advances (could cause query failure when
  system lacks traffic sharing, such as slow flush)

40
Evaluation Unblocking join operators using
heartbeats

• How effectively do heartbeats
• Unblock join queries
• Reduce overall query memory requirements

41
Evaluation Unblocking join operators using
heartbeats (continued)

• Query flow1
• SELECT tb,protocol,srcIP,destIP,
  srcPort,destPort,count() as cnt
• FROM main0_and_control.DataProtocol
• GROUP BY time/10 as tb,protocol,srcIP, destIP,
  srcPort, destPort
• Query flow2
• SELECT tb,protocol,srcIP,destIP,
  srcPort,destPort,count() as cnt
• FROM main1.DataProtocol
• GROUP BY time/10 as tb,protocol,srcIP, destIP,
  srcPort, destPort

42
Evaluation Unblocking join operators using
heartbeats (continued)

• Query full_flow
• SELECT flow1.tb,flow1.protocol, flow1.srcIP,
  flow1.destIP,
• flow1.srcPort,flow1.destPort,
• flow1.cnt, flow2.cnt
• OUTER_JOIN FROM flow1, flow2
• WHERE flow1.srcIPflow2.srcIP and
• flow1.destIPflow2.destIP and
• flow1.srcPortflow2.srcPort and
• flow1.destPortflow2.destPort and
• flow1.protocolflow2.protocol and
• flow1.tb flow2.tb

43
Evaluation Unblocking join operators using
heartbeats (continued)
44
Evaluation Unblocking join operators using
heartbeats (continued)

• Two sub-queries compute the flows aggregated in
  10 second time-buckets
• main1control
• main2
• Full query results combined using full outer join
  for final output

45
Evaluation Unblocking join operators using
heartbeats (continued)

• Varied interval of generated heartbeats
• from 1 second to 60 seconds
• 10 second increments

46
Evaluation Unblocking join operators using
heartbeats (continued)

• Less memory usage with greater heartbeat interval
  (linear growth)
• Avoid accumulating large state of blocking
  operators
• Decrease bursty-ness of output (since less data
  is dumped at the end of the heartbeat interval
  when the interval is low)

47
Evaluation CPU overhead of heartbeat generation

• Measure CPU overhead from heartbeats
• Measure average CPU load of a merge query
• SELET tb, protocol, srcIP, destIP, srcPort,
  destPort, count()
• FROM DataProtocol
• GROUP BY time/10 as tb, protocol, srcIP, destIP,
  srcPort, destPort
• Run on two high-rate interfaces
• Compared 1 second heartbeat interval vs.
  identical system with heartbeats disabled

48
Evaluation CPU overhead of heartbeat generation
(continued)

• Both monitored links have moderately high load,
  so merge operators are naturally unblocked
• Heartbeats disabled 37.3 CPU load
• Heartbeats enabled 37.5 CPU load
• Insignificant difference
• Explained by variations in traffic load
• Conclusion overhead of heartbeat mechanism is
  immeasurably small

49
Conclusion

• Heartbeats regularly generated
• Propagated to all nodes
• Attach temporal update tuples to unblock
  operators
• Heartbeat mechanism can be used for other
  applications in distributed setting
• Detecting node failure
• Performance monitoring
• Query optimization
• Performance evaluation
• Capable of working at multiple Gigabit line
  speeds
• Significantly decrease query memory utilization