Adaptive Overload Control for Busy Internet Servers

1
Adaptive Overload Control for Busy Internet
Servers

• Matt Welsh and David Culler
• USITS 2003
• Presented by Bhuvan Urgaonkar

2
Internet Services Today

• Massive concurrency demands
• Yahoo 1.2 billion pageviews/day
• AOL web caches 10 billion hits/day
• Load spikes are inevitable
• Peak load is orders of magnitude greater than
  average
• Traffic on September 11, 2001 overloaded many
  news sites
• Load spikes occur exactly when the service is
  most valuable!
• In this regime, overprovisioning is infeasible
• Increasingly dynamic
• Days of the static web are over
• Majority of services based on dynamic content
• E-commerce, stock trading, driving directions
  etc.

3
Problem Statement

• Supporting massive concurrency is hard
• Threads/processes dont scale very well
• Static resource containment is inflexible
• How to set a priori resource limits for widely
  varying loads?
• Load management demands a feedback loop
• Replication alone does not solve the load
  management problem
• Individual nodes may still face huge variations
  in demand

4
Proposal The Staged Event-Driven Architecture

• SEDA A new architecture for Internet services
• A general-purpose framework for high concurrency
  and load conditioning
• Decomposes applications into stages separated by
  queues
• Enable load conditioning
• Event queues allow inspection of request streams
• Can perform prioritization or filtering during
  heavy load
• Apply control for graceful degradation
• Perform load shedding or degrade service under
  overload

5
Staged Event-Driven Architecture

• Decompose service into stages separated by queues
• Each stage performs a subset of request
  processing
• Stages internally event-driven, typically
  nonblocking
• Queues introduce execution boundary for isolation
• Each stage contains a thread pool to drive stage
  execution
• Dynamic control grows/shrinks thread pools with
  demand

6
Per-stage admission control

• Admission control done at each stage
• Failure to enqueue a request gt backpressure on
  preceding stages
• Application has flexibility to respond as
  appropriate
• Less conservative than single AC

7
Response time controller

• 90th percentile response time over some interval
  passed to the controller
• AIMD heuristic used to determine token bucket
  rate
• Exact scheduling mechanisms unspecified
• Future work Automatic tuning of parameters

8
Overload management

• Class based differentiation
• Segragate request processing for each class into
  its own set of stages
• Or, have a common set of stages but make the
  admission controller aware of the classes
• Service degradation
• SEDA signals occurrence of overload to
  applications
• If application wants it may degrade service

9
Arashi A SEDA-based email service

• A web-based email service
• Managing folders, deleting/refiling mails,
  search etc
• Client workload emulates several simultaneous
  users, user behavior derived from traces of the
  UCB CS IMAP server

10
Controller operation
11
Overload control with increased user load
12
Increased user load (contd)
13
Overload control under a massive load spike
14
Per-stage AC Vs Single AC
15
Advantages of SEDA

• Exposure of the request stream
• Request level performance made available to
  application
• Focused, application-specific admission control
• Fine-grained admission control at each stage
• Application can provide own admission control
  policy
• Modularity and performance isolation
• Inter-stage communication via event passing
  enables code modularity

16
Shortcomings

• Biggest shortcoming Heuristic based
• May work for some applications, fail for others
• Not completely self-managed
• Response time targets supplied by administrator
• Controller parameters set manually
• Limited to apps based on the SEDA approach
• Evaluation of overheads missing
• Exact scheduling mechanisms missing

17
Some thoughts/directions

• Formal ways to reason about the goodness of
  resource management policies
• Also, the distinction between transient and
  drastic/persistent overloads
• Policy issues Revenue maximization and
  predictable application performance
• Designing Service Level Agreements
• Mechanisms to implement them
• Application modeling and workload prediction

18
Overload control a big picture
Unavoidable overload
Avoidable overload
Underload

• Detection of overloads
• Formal and rigorous ways of defining the
  goodness of
• self-managing techniques
• UO and AO involve different actions (e.g.
  admission
• control versus reallocation). Are they
  fundamentally
• different?

19
Knowing where you are!

• Distinguish avoidable overloads from unavoidable
  overloads
• Need accurate application models, workload
  predictors
• Challenges multi-tiered applications, multiple
  resources, dynamically changing appl behavior
• Simple models based on networks of queues?
• How good would they prove?

Performance Goal
MODEL
Resource allocations
Workload (predicted)
20
Workload prediction a simple example

• A static application model
• Find cpu and nw usage distributions by offline
  profiling
• Use the 99th percentiles as cpu, nw requirements
• When the application runs for real
• We dont get to see what the tail would have been
• So resort to some prediction techniques
• E.g., a web server
• record requests N
• record requests serviced M
• extrapolate to predict the cpu, nw requirements
  of N requests

21
Service-level agreements

• We may want
• Workload Response time
• w1 r1
• w2 r2
• wN rN
• Is this possible to achieve? Maybe not.
• How about
• Response time Revenue/request
• r1 1
• rN N