Probabilistic (Average-Case) Analysis and Randomized Algorithms

1
Probabilistic (Average-Case) Analysis and
Randomized Algorithms

• Two different but similar analyses
• Probabilistic analysis of a deterministic
  algorithm
• Analysis of a Randomized algorithm
• Tools from probability theory
• Indicator variables
• Linearity of expectation
• Example Problems/Algorithms
• Hiring problem (Chapter 5)
• Biased coin
• Sorting/Quicksort (Chapter 7)
• Hat Check Problem
• Online Hiring Problem

2
Probabilistic (Average-Case) Analysis

• Algorithm is deterministic for a fixed input, it
  will run the same every time
• Analysis Technique
• Assume a probability distribution for your inputs
• Analyze item of interest over the distribution
• Caveats
• Specific inputs may have much worse performance
• If the distribution is wrong, analysis may give
  misleading picture

3
Randomized Algorithm

• Randomize the algorithm for a fixed input, it
  will run differently depending on the result of
  random coin tosses
• Randomization examples/techniques
• Randomize the order that candidates arrive
• Randomly select a pivot element
• Randomly select from a collection of
  deterministic algorithms
• Key points
• Works well with high probability on every input
• May fail on every input with low probability

4
Key Analysis Tools

• Indicator variables
• Suppose we want to study random variable X that
  represents a composite of many random events
• Define a collection of indicator variables Xi
  that focus on individual events typically X S
  Xi
• Linearity of expectations
• Let X, Y, and Z be random variables s.t. X Y
  Z
• Then EX EYZ EY EZ
• Recurrence Relations

5
Hiring Problem

• Input
• A sequence of n candidates for a position
• Each has a distinct quality rating that we can
  determine in an interview
• Algorithm
• Current 0
• For k 1 to n
• If candidate(k) is better than Current, hire(k)
  and Current k
• Cost
• Number of hires
• Worst-case cost is n

6
Analyze Hiring Problem

• Assume a probability distribution
• Each of the n! permutations is equally likely
• Analyze item of interest over probability
  distribution
• Define random variables
• Let X random variable corresponding to of
  hires
• Let Xi indicator variable that ith
  interviewed candidate is hired
• Value 0 if not hired, 1 if hired
• X Si 1 to n Xi
• EXi ?
• Explain why
• EX ESi 1 to n Xi Si 1 to n EXi ?
• Key observation linearity of expectations

7
Alternative analysis

• Analyze item of interest over probability
  distribution
• Let Xi indicator random variable that the ith
  best candidate is hired
• 0 if not hired, 1 if hired
• Questions
• Relationship of X to Xi?
• EXi ?
• Si 1 to n EXi ?

8
Questions

• What is the probability you will hire n times?
• What is the probability you will hire exactly
  twice?
• Biased Coin
• Suppose you want to output 0 with probability ½
  and 1 with probability ½
• You have a coin that outputs 1 with probability p
  and 0 with probability 1-p for some unknown 0 lt p
  lt 1
• Can you use this coin to output 0 and 1 fairly?
• What is the expected running time to produce the
  fair output as a function of p?

9
Quicksort Algorithm

• Overview
• Choose a pivot element
• for example the last element
• Partition elements to be sorted based on
  partition element
• Recursively sort smaller and larger elements

10
Quicksort Walkthrough

• 17 12 6 23 19 8 5 10
• 6 8 5 10 17 12 23 19
• 5 6 8 17 12 19 23
• 6 8 12 17 23
• 6 17
• 5 6 8 10 12 17 19 23

11
Formal Average-case Analysis

• Let X denote the random variable that represents
  the total number of comparisons performed
• Let indicator variable Xij the event that the
  ith smallest element and jth smallest element are
  compared
• 0 if not compared, 1 if compared
• X Si1 to n-1 Sji1 to n Xij
• EX Si1 to n-1 Sji1 to n EXij

12
Computing EXij

• EXij probability that i and j are compared
• Observation
• All comparisons are between a pivot element and
  another element
• If an item k is chosen as pivot where i lt k lt j,
  then items i and j will not be compared
• EXij
• Items i or j must be chosen as a pivot before any
  items in interval (i..j)

13
Computing EX

• EX Si1 to n-1 Sji1 to n EXij
• Si1 to n-1 Sji1 to n 2/(j-i1)
• Si1 to n-1 Sk1 to n-i 2/(k1)
• Si1 to n-1 2 Hn-i1
• Si1 to n-1 2 Hn
• 2 (n-1)Hn

14
Alternative Average-case Analysis

• Let T(n) denote the average time required to sort
  an n-element array
• Average assuming each permutation equally
  likely
• Write a recurrence relation for T(n)
• T(n)
• Using induction, we can then prove that T(n)
  O(n log n)
• See homework 2, problem 6 for more details

15
Avoiding the worst-case

• Understanding quicksorts worst-case
• Methods for avoiding it
• Pivot strategies
• Randomization

16
Understanding the worst case
A B D F H J K A B D F H J A B D F
H A B D F A B D A B A The worst case
occur is a likely case for many applications.
17
Pivot Strategies

• Use the middle Element of the sub-array as the
  pivot.
• Use the median element of (first, middle, last)
  to make sure to avoid any kind of pre-sorting.
• What is the worst-case performance for these
  pivot selection mechanisms?

18
Randomized Quicksort

• Make chance of worst-case run time equally small
  for all inputs
• Methods
• Choose pivot element randomly from range
  low..high
• Initially permute the array

19
Hat Check Problem

• N people give their hats to a hat-check person at
  a restaurant
• The hat-check person returns the hats to the
  people randomly
• What is the expected number of people who get
  their own hat back?

20
Online Hiring Problem

• Input
• A sequence of n candidates for a position
• Each has a distinct quality rating that we can
  determine in an interview
• We know total ranking of interviewed candidates,
  but not with respect to candidates left to
  interview
• We can hire only once
• Online Algorithm Hire(k,n)
• Current 0
• For j 1 to k
• If candidate(j) is better than Current, Current
  j
• For j k1 to n
• If candidate(j) is better than Current, hire(j)
  and return
• Questions
• What is probability we hire the best qualified
  candidate given k?
• What is best value of k to maximize above
  probability?

21
Online Hiring Analysis

• Let S be the probability that we successfully
  hire the best qualified candidate
• Let Si be the probability that we successfully
  hire the best qualified candidate AND this
  candidate was the ith one interviewed
• What needs to happen for Si to be true?
• Best candidate is in position i Bi
• No candidate in positions k1 through i-1 are
  hired Oi
• These two quantities are independent, so we can
  multiply their probabilities to get Si

22
Computing S

• Bi 1/n
• Oi k/(i-1)
• Si k/(n(i-1))
• S Sigtk Si k/n Sigtk 1/(i-1) is probability of
  success
• k/n (Hn Hk) roughly k/n (ln n ln k)
• Maximized when k n/e
• Leads to probability of success of 1/e