Network reliability and QoS measurements

1
Network reliability and QoS measurements

• Henning Schulzrinne
• Columbia University
• Samsung, Seoul
• March 2004

2
Overview

• The IRT Lab at Columbia University
• Application Internet multimedia
• Quality of service
• scheduling and admission control ? thousands of
  papers
• network signaling
• end-system performance ? embedded end systems
  PCs
• QoS ? network application reliability

3
Laboratory overview

• 11 PhDs
• 3 at IBM, Lucent, Telcordia
• 5 MS
• Visitors (Ericsson, Fujitsu, Mitsubishi, Nokia,
  U. Coimbra, U. Oulu, )
• China, Finland, Greece, India, Japan, Portugal,
  Spain, Sweden, US, Taiwan

4
IRT topics

• Internet multimedia protocols and systems
• Internet telephony and radio (SIP, RTSP, RTP)
• Content distribution networks
• Internet-scale event distribution
• Service creation
• Ubiquitous, context-aware computing and
  communications
• Protocols and services for wireless ad-hoc
  networks
• Service discovery
• Quality of service
• Pricing for adaptive services
• Scalable resource reservation protocols (CASP,
  BGRP, YESSIR)
• End-system evaluation
• Network measurements
• Service reliability

5
Internet multimedia

• Internet telephony replacing the existing
  circuit-switched system with Internet-based
  systems
• Signaling and services
• Quality of service philosophies
• end systems adapt and compensate
• end systems use FEC, LBR, PLC
• jitter ? playout delay compensation
• network offers guarantees ? difficult
  architecturally, business, not necessarily
  technically
• we pursue both

6
Assessment of VoIP Service Availability

• Wenyu Jiang
• Henning Schulzrinne
• IRT Lab, Dept. of Computer Science
• Columbia University

7
Overview

• (on-going work, preliminary results, still
  looking for measurement sites, )
• Service availability
• Measurement setup
• Measurement results
• call success probability
• overall network loss
• network outages
• outage induced call abortion probability

8
Service availability

• Users do not care about QoS
• at least not about packet loss, jitter, delay
• FEC and PLC can deal with losses up to 5-8
• rather, its service availability ? how likely is
  it that I can place a call and not get
  interrupted?
• availability MTBF / (MTBF MTTR)
• MTBF mean time between failures
• MTTR mean time to repair
• availability successful calls / first call
  attempts
• equipment availability 99.999 (5 nines) ? 5
  minutes/year
• ATT 99.98 availability (1997)
• IP frame relay SLA 99.9
• UK mobile phone survey 97.1-98.8

9
Availability PSTN metrics

• PSTN metrics (Worldbank study)
• fault rate
• should be less than 0.2 per main line
• fault clearance ( MTTR)
• next business day
• call completion rate
• during network busy hour
• varies from about 60 - 75
• dial tone delay

10
Example PSTN statistics
Source Worldbank
11
Measurement setup
Node name Location Connectivity Network
columbia Columbia University, NY gt OC3 I2
wustl Washington U., St. Louis I2
unm Univ. of New Mexico I2
epfl EPFL, Lausanne, CH I2
hut Helsinki University of Technology I2
rr NYC cable modem ISP
rrqueens Queens, NY cable modem ISP
njcable New Jersey cable modem ISP
newport New Jersey ADSL ISP
sanjose San Jose, California cable modem ISP
suna Kitakyushu, Japan 3 Mb/s ISP
sh Shanghai, China cable modem ISP
Shanghaihome Shanghai, China cable modem ISP
Shanghaioffice Shanghai, China ADSL ISP
12
Measurement setup

• Active measurements
• call duration 3 or 7 minutes
• UDP packets
• 36 bytes alternating with 72 bytes (FEC)
• 40 ms spacing
• September 10 to December 6, 2002
• 13,500 call hours

13
Call success probability

• 62,027 calls succeeded, 292 failed ? 99.53
  availability
• roughly constant across I2, I2, commercial ISPs

All 99.53
Internet2 99.52
Internet2 99.56
Commercial 99.51
Domestic (US) 99.45
International 99.58
Domestic commercial 99.39
International commercial 99.59
14
Overall network loss

• PSTN once connected, call usually of good
  quality
• exception mobile phones
• compute periods of time below loss threshold
• 5 causes degradation for many codecs
• others acceptable till 20

loss 0 5 10 20
All 82.3 97.48 99.16 99.75
ISP 78.6 96.72 99.04 99.74
I2 97.7 99.67 99.77 99.79
I2 86.8 98.41 99.32 99.76
US 83.6 96.95 99.27 99.79
Int. 81.7 97.73 99.11 99.73
US ISP 73.6 95.03 98.92 99.79
Int. ISP 81.2 97.60 99.10 99.71
15
Network Outages

• sustained packet losses
• arbitrarily defined at 8 packets
• far beyond any recoverable loss (FEC,
  interpolation)
• 23 outages
• make up significant part of 0.25 unavailability
• symmetric A?B ?? B?A?
• spatially correlated A?B ? ? A?X?
• not correlated across networks (e.g., I2 and
  commercial)

16
Network outages
17
Network outages
no. of outages symmetric duration (mean) duration (median) total (all, hm) outages gt 1000 packets
all 10,753 30 145 25 1720 1058
I2 819 14.5 360 25 317 233
I2 2,708 10 259 26 747 537
ISP 8,045 37 107 24 933 458
US 1,777 18 269 20 518 353
Int. 8,976 33 121 26 1202 642
18
Outage-induced call abortion proability

• Long interruption ? user likely to abandon call
• from E.855 survey Pholding e-t/17.26 (t in
  seconds)
• ? half the users will abandon call after 12s
• 2,566 have at least one outage
• 946 of 2,566 expected to be dropped ? 1.53 of
  all calls

all 1.53
I2 1.16
I2 1.15
ISP 1.82
US 0.99
Int. 1.78
US ISP 0.86
Int. ISP 2.30
19
Conclusion

• Availability in space is (mostly) solved ?
  availability in time restricts usability for new
  applications
• initial investigation into service availability
  for VoIP
• need to define metrics for, say, web access
• unify packet loss and no Internet dial tone
• far less than 5 nines
• working on identifying fault sources and
  locations
• looking for additional measurement sites

20
Quality and Performance Evaluation of VoIP
End-points

• Wenyu Jiang
• Henning Schulzrinne
• Columbia University

21
Motivations

• The quality of VoIP depends on both the network
  and the end-points
• Extensive QoS literature on network performance,
  e.g., IntServ, DiffServ
• Focus is on limiting network loss delay
• Little is known about the behavior of VoIP
  end-points

22
Performance Metrics for VoIP End-points

• Mouth-to-ear (M2E) delay
• compare network delay
• Clock skew
• whether it causes any voice glitches
• amount of clock drift
• Silence suppression behavior
• whether the voice is clipped (depends much on
  hangover time)
• robustness to non-speech input, e.g., music
• Robustness to packet loss
• voice quality under packet loss
• Acoustic echo cancellation
• Jitter adaptation delay gt max(jitter)?

23
Measurement Approach

• Capture both original and output audio
• Use adelay program to measure M2E delay
• auto correlation
• no clock synchronization needed
• Assume a LAN environment by default
• Serve as a baseline of reference, or lower bound

24
VoIP End-points Tested

• Hardware End-points
• Cisco, 3Com and Pingtel IP phones
• Mediatrix 1-line SIP/PSTN Gateway
• Software clients
• Microsoft Messenger, NetMeeting (Win2K, WinXP)
• Net2Phone (NT, Win2K, Win98)
• Sipc/RAT (Solaris, Ultra-10)
• Robust Audio Tool (RAT) from UCL as media client
• Operating parameters
• In most cases, codec is G.711 ?-law, packet
  interval is 20ms

25
IP Phone Hardware

• DSP for audio coding, AEC
• ?C for protocol processing
• embedded OS (Linux, Windriver, ) with web
  browser
• Ethernet interface, maybe with hub

26
Example M2E Delay Plot

• 3Com to Cisco, shown with gaps gt 1sec
• Delay adjustments correlate with gaps, despite
  3Com phone has no silence suppression

27
Visual Illustration of M2E Delay Drop, Snapshot 1

• 3Com to Cisco 1-1 case
• Left/upper channel is original audio
• Highlighted section shows M2E delay (59ms)

28
Snapshot 2

• M2E delay drops to 49ms, at time of 416

29
Snapshot 3

• Presence of a gap during the delay change

30
Effect of RTP Marker Bits on Delay Adjustments

• Cisco phone sends M-bits, whereas Pingtel phone
  does not
• Presence of M-bits results in more adjustments

31
Sender Characteristics

• Certain senders may introduce delay spikes,
  despite operating on a LAN

32
Average M2E Delays for IP phones and sipc

• Averaging the M2E delay allows more compact
  presentation of end-point behaviors
• Receiver (especially RAT) plays an important role
  in M2E delay

33
Average M2E Delays for PC Software Clients

• Messenger 2000 wins the day
• Its delay as receiver (68ms) is even lower than
  Messenger XP, on the same hardware
• It also results in slightly lower delay as sender
• NetMeeting is a lot worse (gt 400ms)
• Messengers delay performance is similar to or
  better than a GSM mobile phone.

A B A?B B?A
MgrXP (pc) MgrXP (notebook) 109ms 120ms
Mgr2K (pc) MgrXP (notebook) 96.8ms 68.5ms
NM2K (pc) NM2K (notebook) 401ms 421ms
Mobile (GSM) PSTN (local number) 115ms 109ms
34
Delay Behaviors for PC Clients, contd.

• Net2Phones delay is also high
• 200-500ms
• V1.5 reduces PC-gtPSTN delay
• PC-to-PC calls have fairly high delays

A B A?B B?A
N2P v1.1 NT P-2 (pc2) PSTN (local number) 292ms 372ms
N2P v1.5 NT P-2 (pc2) PSTN (local number) 201ms 373ms
N2P v1.5 W2K K7 (pc) PSTN (local number) 196ms 401ms
N2P v1.5 W2K K7 (pc) N2P v1.5 W98 P-3 (notebook2) 525ms 350ms
35
Effect of Clock Skew Cisco to 3Com, Experiment
1-1

• Symptom of playout buffer underflow
• Waveforms are dropped
• Occurred at point of delay adjustment
• Bugs in software?

36
Clock Skew Rates

• Mostly symmetric between two devices
• RAT (Sun Ultra-10) has unusually high drift
  rates, gt 300 ppm (parts per million)
• High clock skews confirmed in many (but not all)
  PCs and workstations

Drift Rates (in ppm) 3Com Cisco Mediatrix Pingtel RAT
3Com -8.3 55.4 43.3 41.2 -333
Cisco -55.2 -0.4 -11.8 -12.1 -381
Mediatrix -43.1 11.7 1.3 -0.8
Pingtel -40.9 12.7 2.8 -3.5 -380
RAT 343 403 376 12.3
37
Drift Rates for PC Clients

• Drift Rates not always symmetric!
• But appears to be consistent between Messenger
  2K/XP and Net2Phone on the same PC
• Existence of 2 clocking circuits in sound card?

A B A?B B?A
MgrXP (pc) MgrXP (notebook) 172 87.7
Mgr2K (pc) MgrXP (notebook) 165 85.6
NM2K (pc) NM2K (notebook) ? -33?
Net2Phone NT (pc2) PSTN 290 -287
Net2Phone 2K (pc) PSTN 166 82
Mobile (GSM) PSTN 0 0
38
Packet Loss Concealment

• Common PLC methods
• Silence substitution (worst)
• Packet repetition, with optional fading
• Extrapolation (one-sided)
• Interpolation (two-sided), best quality
• Use deterministic bursty loss pattern
• 3/100 means 3 consecutive losses out of every 100
  packets
• Easier to locate packet losses
• Tested 1/100, 3/100, 1/20, 5/100, etc.

39
PLC Behaviors

• Loss tolerance (at 20ms interval)
• By measuring loss-induced gaps in output audio
• 3Com and Pingtel phones 2 packet losses
• Cisco phone 3 packet losses
• Level of audio distortion by packet loss
• Inaudible at 1/100 for all 3 phones
• Inaudible at 3/100 and 1/20 for Cisco phone, yet
  audible to very audible for the other two.
• Cisco phone is the most robust
• Probably uses interpolation

40
Effect of PLC on Delay

• No affirmative effect on M2E delay
• E.g., sipc to Pingtel

41
Silence Suppression

• Why?
• Saves bandwidth
• May reduce processing power (e.g., in
  conferencing mixer)
• Facilitates per-talkspurt delay adjustment
• Key parameters
• Silence detection threshold
• Hangover time, to delay silence suppression and
  avoid end clipping of speech
• Usually 200ms is long enough Brady 68

42
Hangover Time

• Measured by feeding ON-OFF waveforms and monitor
  RTP packets
• Cisco phones is the longest (2.3-2.36 sec), then
  Messenger (1.06-1.08 sec), then NetMeeting
  (0.56-0.58 sec)
• A long hangover time is not necessarily bad, as
  it reduces voice clipping
• Indeed, no unnatural gaps are found
• Does waste a bit more bandwidth

43
Robustness of Silence Detectors to Music

• On-hold music is often used in customer support
  centers
• Need to ensure music is played without any
  interruption due to silence suppression
• Tested with a 2.5-min long soundtrack
• Messenger starts to generate many unwanted gaps
  at input level of -24dB
• Cisco phone is more robust, but still fails from
  input level of -41.4dB

44
Acoustic Echo Cancellation

• Important for hands-free/conferencing (business)
  applications
• Primary metric Echo Return Loss (ERL)
• Measured by LAN-sniffing RTP packets
• Most IP phones support AEC
• ERL depends slightly on input level and
  speaker-phone volume
• Usually gt 40 dB (good AEC performance)

IP Phone 3Com Cisco ipDialog Pingtel Snom-100
ERL (dB) 40-45 53-? 49-54 33-42 ?-5 (no AEC)
45
M2E Delay under Jitter

• Delay properties under the LAN environment serves
  as a baseline of reference
• When operating over the Internet
• Fixed portion of delay adds to M2E delay as a
  constant
• Variable portion (jitter) has a more complex
  effect

• Initial test
• Used typical cable modem delay traces
• Tested RAT to Cisco
• No audible distortion due to late loss
• Added delay is normal

46
M2E Delay under Jitter, contd.

• Cisco phone generally within expectation
• Can follow network delay change timely
• Takes longer (10-20sec) to adapt to decreasing
  delay
• Does not overshoot playout delay
• More end-points to be examined

Artificial Trace
Real Trace with Spikes
47
Conclusions

• Average M2E Adelay
• Low (mostly lt 80ms) for hardware IP phones
• Software clients lowest for Messenger 2000
  (68.5ms)
• Application (receiver) most vital in determining
  delay
• Poor implementation easily undoes good network
  QoS
• Clock skew high on SW clients (RAT, Net2Phone)
• Packet loss concealment quality
• Acceptable in all 3 IP phones tested, w. Cisco
  more robust
• Silence detector behavior
• Long hangover time, works well for speech input
• But may falsely predict music as silence
• Acoustic Echo Cancellation good on most IP
  phones
• Playout delay behavior good based on initial
  tests

48
Future Work

• Further tests with more end-points on how jitter
  influences M2E delay
• Measure the sensitivity (threshold) of various
  silence detectors
• Investigate the non-symmetric clock drift
  phenomena
• Additional experiments as more brands of VoIP
  end-points become available