Probabilistic Algorithm Analysis

1
COMP 482 / ELEC 420

• Probabilistic Algorithm Analysis

2
Math Background Review Beyond

• Asymptotic concepts useful math
• Recurrences
• Probabilistic analysis

3
Your To-Do List

• Read CLRS 5.
• Assignment 2.

4
Motivation

• Not all algorithmic analysis is based on
  deterministic recurrences.
• Some require probabilistic analysis.
• Review some probability
• Extended example
• Review probability
• Review/learn basic technique

5
Conditional Probability
6
Random Variables

• What is the probability of getting exactly 1
  tail when flipping two normal coins?
• X Tails, x1
• S TwoNormalCoinsFlipped HH, HT, TH, TT
• PrXx ?s ? S X(s)x Prs
• PrTails1
• Scoins ? TwoNormalCoinsFlipped
  Tails(coins)1 Prcoins
• PrHT PrTH
• ¼ ¼
• ½

7
Random Variables

• Random variables X,Y independent
• ?
• PrXx ? Yy PrXx ? PrYy
• Expected values of random variables
• EX ?x (x ? PrXx)
• EXY EXEY linearity of expectation
• Ea?X a?EX
• EX?Y EX?EY if X,Y independent

8
Hiring Problem (aka Secretary Problem)

• Goal Hire the most qualified person for a job.
• Algorithm well use
• Decide how many people (n) to interview.
• Hire the first candidate.
• For each interviewee, in turn
• Interview.
• Hire, if better than current employee.
• Costs
• (n people interviewed ? ci each) (m people
  hired ? ch each)
• O(ci ? n ch ? m)
• By definition, n ? m. Assume ci ? ch.

9
Hiring Problem Worst-Case

• mn
• In other words,
• Hire every interviewee.
• Interviewees (unluckily) arranged in ascending
  order of quality.
• Cost O(cin chn) O(chn)

10
Hiring Problem Average-Case

• What is m likely to be?
• Wrong answer m can be 1n, so use the mean
  mn/2.
• Instead, average cost over all possible
  interviewee orders.
• Number each interviewee 1n.
• Consider each permutation of the set 1,,n.
• To compute this average, use probabilistic
  techniques to avoid listing every possible
  permutation.

11
Averaging via Expected Values
ETotalHired E?i1..n Hired?i
?i1..n EHired?i
EHired?i 1?Pri hired 0?Pri not
hired Pri hired 1/i Each has equal
chance to be the best of first i.
?i1..n 1/i ln n O(1)
Expected algorithm cost O(ci n ch log n)
12
Input Order

• Lets say the candidates come from an employment
  agency.
• Algorithms cost is greatly affected by the
  agency.
• Agency could send best applicants first.
• Typical of well-run agencies.
• Waste of money to interview later applicants.
• Agency could send worst applicants first.
• Maximizes their fees.
• We can randomize order, to minimize chance of
  worst case.
• Protects against poor input distributions.
  (Accidental or malicious.)
• Can still produce worst case, but only with low
  probability.

13
Randomization
Randomizing the order of an array A Let n
elements in A For i1 to n, swap Ai with
Arandom(in)

• Simple algorithm, with an interesting proof that
  it computes a uniform random permutation.

The key use a loop invariant Before ith
iteration, for any (i-1)-permutation of elements
from 1n, A1i-1 contains the permutation with
probability ((n-i1)!)/(n!).
14
Randomization
Proving loop invariant Before ith iteration, for
any (i-1)-permutation of elements from
1n, A1i-1 contains the permutation with
probability ((n-i1)!)/(n!).
Let n elements in A For i1 to n, swap Ai
with Arandom(in)

• Base case, i1
• A1..0 contains the 0-permutation (no data) with
  probability (n!)/(n!)1.
• Holds trivially.

15
Randomization
Proving loop invariant Before ith iteration, for
any (i-1)-permutation of elements from
1n, A1i-1 contains the permutation with
probability ((n-i1)!)/(n!).
Let n elements in A For i1 to n, swap Ai
with Arandom(in)

• Inductive case, show holds for i1
• For an arbitrary i-permutation, the permutation
  is obtained ?
• Event1 its first i-1 elements, an
  (i-1)-permutation, were obtained in the first i-1
  iterations, and
• Event2 the ith iteration chooses that
  i-permutations ith element.

PrEvent1 ? Event2 PrEvent1 ? PrEvent2
Event1
16
Randomization
Proving loop invariant Before ith iteration, for
any (i-1)-permutation of elements from
1n, A1i-1 contains the permutation with
probability ((n-i1)!)/(n!).
Let n elements in A For i1 to n, swap Ai
with Arandom(in)

• Inductive case, show holds for i1
• For an arbitrary i-permutation, the permutation
  is obtained ?
• Event1 its first i-1 elements, an
  (i-1)-permutation, were obtained in the first i-1
  iterations, and
• Event2 the ith iteration chooses that
  i-permutations ith element.

PrEvent1 ? Event2 PrEvent1 ? PrEvent2
Event1
(n-i1)!/n! ? 1/(n-i1)
(n-i)!/n! Algebra, cancelling term
(n-(i1)1)!/n! Algebra
17
Randomization
Proving loop invariant Before ith iteration, for
any (i-1)-permutation of elements from
1n, A1i-1 contains the permutation with
probability ((n-i1)!)/(n!).
Let n elements in A For i1 to n, swap Ai
with Arandom(in)

• At termination (in1)
• Invariant ? any n-permutation of elements 1n
  occurs in A1..n with probability 1/(n!).
• I.e., uniform random distribution.

18
On-Line Hiring Problem

• Dont want to interview all candidates.
• Instead, interview some until find one good
  enough. More realistic.
• Algorithm
• For each interviewee,
• Interview
• Hire and stop loop, if good enough.

Cost O(cin ch) O(cin)
19
On-Line Hiring Problem

• One possible algorithm
• Interview k candidates, and hire the first one
  after that with a better score than previously
  found.

BestScore -? For i1 to k, BestScore
max(BestScore, Score(i)) For ik1 to n,
If Score(i) gt BestScore, Then Hire i
Quit Hire n

• Strategy to find best k
• Fix k, compute the probability of getting best
  candidate.
• Find k to maximize probability.

20
On-Line Hiring Problem

• Define S succeed in finding best candidate
• Si succeed in finding best candidate best
  is i

Since we never hire from the first k, PrS1
PrSk 0 PrS ?ik1n PrSi
Since Sis disjoint, PrS ?i1...n PrSi
Need to determine PrSi for i?k1n.
Si candidate i is best candidates k1i-1
not chosen candidate i is best best
among 1i-1 is among 1k (not k1i-1)
PrSi ?
1/n ? k/(i-1) k/(n?(i-1))
21
On-Line Hiring Problem

• Define S succeed in finding best candidate
• Si succeed in finding best candidate best
  is i

PrSi 0 for i?1k PrSi k/(n?(i-1)) for
i?k1n PrS ?ik1n PrSi
?ik1n k/(n?(i-1)) k/n ? ?ik1n 1/(i-1)
k/n ? ?jkn-1 1/j Renaming ji1
? What now? ?
22
On-Line Hiring Problem

• PrS ? k/n ? (ln n - ln k) Find k to maximize
  PrS.

Differentiate w.r.t. k, and set to zero 1/n ?
(ln n - ln k - 1) 0 ln k ln n - 1 ln k ln
n/e k n/e
Plug in PrS ? (n/e)/n ? (ln n - ln (n/e))
1/e ? (ln n - ln n ln e) 1/e
? 1 1/e ? 0.368
Algorithm has better than 1/3 chance of getting
best candidate.