Communication-Efficient Distributed Monitoring of Thresholded Counts

1
Communication-Efficient Distributed Monitoring
of Thresholded Counts

• Ram Keralapura, UC-Davis
• Graham Cormode, Bell Labs
• Jai Ramamirtham, Bell Labs

2
Introduction

• Monitoring is critical to managing distributed
  networked systems
• Main challenges
• Continuous
• Distributed
• Resource-constrained environments

3
Thresholded Counts

• New fundamental class of problems
• Tracking counts for an event beyond a given
  threshold value with user-specified accuracy
• Motivating scenarios
• Total of connections to a server when it
  exceeds the normal operational condition (ex,
  DDoS attacks)
• Total traffic to a particular destination prefix
  when it exceeds the pre-defined limit
• Tracking the total number of cars on a highway

4
Thresholded Counts (contd)

• Two key properties
• Threshold value
• User specified tracking accuracy

5
System Architecture

• remote sites (or monitors) and a coordinator
  site (or central node)

Non-continuous updates
Local thresholds at remote sites
Counts can be positive, negative, or fractional
Ignore network delays and losses
6
Approach

• Every remote monitor , maintains a set of local
  thresholds
• Local count at monitor , should always lie
  between two neighboring thresholds
• Global estimate at the central node

7
Approach (contd)

• Maximum error in the global estimate should
  satisfy
• Two methods to set local thresholds
• Static thresholding
• Adaptive thresholding

8
Static Thresholding

• Problem For given values and , we have to
  determine such that,

9
Uniform
Central Node
Monitor-1
Monitor-2
Monitor-3
Blended threshold assignment
Proportional
Monitor-3
Central Node
Monitor-1
Monitor-2
10
Static Thresholding (contd)

• Blended threshold assignment
• ? uniform threshold assignment
• ? proportional threshold assignment
• Complexity

11
Adaptive Thresholding

• Every monitor maintains only two threshold
  values and
• Problem For given values and , and a
  threshold violation from monitor , determine
  for all the monitors such that,

12
Slack
Central Node
Monitor-1
Monitor-2
Monitor-3
Basic Adaptive Algorithm
Central Node
Monitor-1
Monitor-2
Monitor-3
13
Experimental Setup

• Built a simulator with monitoring nodes and a
  central node
• Implemented all the static and adaptive
  algorithms
• Data set Public traces from NLANR

14
Count Accuracy
15
Validating the Theoretical Model
16
Comparing Costs Static and Adaptive Cases
17
Related Work

• Top-k monitoring Babcock et al
• Heavy-hitter definition
• Adaptive filters for continuous queries Olston
  et al
• Distributed continuous queries but does not
  address the thresholded counts problem
• Distributed triggers Jain et al
• Simplified version of the thresholded counts
  problem
• Randomized algorithms with statistical guarantees
• Geometric approach for threshold functions
  Sharfman et al
• Focus is mainly on non-linear functions

18
Summary

• We defined a fundamental class of problems called
  Thresholded Counts
• We proposed algorithms to address the problem
  static and adaptive
• Analyzed the complexities of these algorithms and
  provided proofs
• Using experiments, we showed the effectiveness of
  our algorithms

19
Future Work

• Building the monitoring system for real networks
  to explore the practical aspects of our framework
• Sensor networks
• IP network monitoring
• Address scalability issues
• For example, hierarchical monitoring architecture
• Extend for different query types with thresholded
  nature
• For example, arithmetic combinations

20
Thank you!! Questions??
Contact rkeralapura_at_ucdavis.edu