Developing for Optimal VoIP Service Quality

1
(No Transcript)
2
Developing for Optimal VoIP Service Quality

• Understanding Quality From Users Perspective

Arun Bhardwaj Director, Business Development
Keynote Systems, Inc. arun.bhardwaj_at_keynote.com
3
Agenda

• VoIP Market View
• Unique Approach to Voice Service Quality
  Measurement
• Comparative Analysis of Various Voice
  Technologies
• Often Ignored Factors Affective Quality
• Service Quality Trends in the VoIP Industry

4
Unique Nature of VoIP
5
Voice Communication Landscape
6
Why is Voice Quality Essential?
VoIP Growth
Barriers to VoIP Adoption
Subscribers Revenue
2004 1.3million 200million
2005 4.5million 1billion
7
Competitive Research Study Objectives

• Rank the relative performance of PSTN,
  PacketCable, VoIP hard phone, and VoIP soft phone
  service providers.
• Identify industry trends in service level
  performance since the last Keynote study.
• Identify the range of performance between the
  best voice service providers and the worst.
• Examine peak and prime-time performance
  variations.
• Identify the strengths and weaknesses of each
  service provider and voice service technology.

8
Measurement Topology
9
Monitoring Scope
10
Audio Characteristics Analysis
Last Mile Impairments Measuring Within Network
is Not Enough
All Calls All Calls Calls with MOS lt 3.0 Calls with MOS lt 3.0
of calls Percentage of calls Percentage
Hiss 0 0.0 0 0.0
Static 539 2.9 407 71.3
Hum 7 0.0 0 0.0
Frequency Clipping 0 0.0 0 0.0
Front Clipping 5 0.0 1 0.2
Back Clipping 422 2.3 405 70.9
Other Clipping 607 3.3 520 91.1
Holdover 1,099 5.6 32 5.6
Total 18,456 571
11
Keynote Voice Perspective Agent Technology
Caller agent compares received and reference
audio samples
Responder Agent Accepts calls sends audio sample

New York Cable/DSL/Sprint Caller Responder
Caller Agent Initiates calls requests audio
sample
San Francisco Cable/DSL/Sprint Caller Responder
12
Measured Parameters
- Average Mean Opinion Score (MOS) - Calls gt
Acceptable MOS - MOS Geographic Variability
Audio Clarity
Voice Service Quality
Responsiveness
Reliability
- Average Audio Delay - Calls gt Acceptable
Delay - Audio Delay Geo Variability
- Service Availability - Dropped Calls - Average
Answer Time
Holistic Customer Experience !!!
13
True End-to-End Monitoring Methodology
14
Competitive Research Report
VoIP Phone
Digital Cable Phone
Soft Phone
PSTN
15
Case Study Invisible Annoyance
Low MOS score for gt 90 of calls
Customer Problem
16
Case Study Invisible Annoyance
Low MOS score for gt 90 of calls
Customer Problem
Analyzed Audio Characteristics of all calls for
the problem period using Voice Perspective
Keynote Analysis
Diagnosis
Hum problem and hardware ATA model type showed
strong correlation
VoIP Perspective Agent ATA Model of Calls with Hum
New York ATT Model B 96.9
New York Sprint Model B 87.6
New York Time Warner Cable Model B 97.6
New York UUNet Model B 97.7
New York Verizon DSL Model B 92.3
San Francisco ATT Model A 0.0
San Francisco Comcast Cable Model B 97.1
San Francisco SBC DSL Model B 97.4
San Francisco Sprint Model A 0.0
San Francisco UUNet Model A 0.1
17
Case Study Invisible Annoyance
Low MOS score for gt 90 of calls
Customer Problem
Analyzed Audio Characteristics of all calls for
the problem period using Voice Perspective
Keynote Analysis
Diagnosis
Hum problem and hardware ATA model type showed
strong correlation
The problem was in a specific telephone adapter
model type
Audio Clarity Ranking improved by TWO places
after replacing adapters Increased customer
satisfaction (Mean Opinion Score increased by 0.3)
18
Data Collection Period and Size

• Data collected from August 1st August 31st,
  2006.
• Long distance and local VoIP to PSTN calls
  between New York and San Francisco every 30
  minutes.
• Call placed on every VoIP provider and network
  combination
• Total of over 125,000 phone calls were placed
  during the one month period

19
Study Results Overview
20
Summary of Results

• Key performance indicators such as Service
  Availability and Average MOS improved for most
  providers.
• Average one-way audio delay between 150 and 250
  ms Best 62 ms
  Worst 335 ms.
• Average Mean Opinion Score range 3.0 to 4.0.
  Best
  4.24 Worst 2.64.
• Calls on most providers have clipping or audio
  holdover causing service degradation.

21
Summary of Results

• Primetime Versus Non-primetime Performance
• Higher variation in audio delay than in Mean
  Opinion Score.
• DSL connections offered less audio delay variance
  
• Cable modem connection delivered more consistent
  MOS

22
Reliability Overview Service Types

• PacketCable service providers are more reliable
  than PSTN, VoIP Hard Phone, and VoIP Soft Phone
  service providers.

23
Audio Quality Overview Service Types

• PacketCable service providers had better overall
  audio quality than PSTN, VoIP Hard Phone, and
  VoIP Soft Phone service providers.

24
Trends Service Availability

• Of the eleven service providers measured in
  previous studies, seven had a better Service
  Availability percentage in August, 2006 than in
  any previous Keynote study.

25
Trends Average MOS

• Of the eleven service providers measured in
  previous studies, seven had a higher Average MOS
  in August, 2006 than in any previous Keynote
  study.

26
Audio Delay and MOS Trends
27
Performance Ranges Audio Delay

• Only four of the fourteen providers measured in
  August had an Average Audio Delay below 150 ms.
• The best Average Audio Delay was 62 ms the worst
  was 335 ms.

28
Performance Ranges Average MOS

• Only four of the fourteen providers measured an
  Average MOS above the 4.0 toll quality
  threshold.
• The best Average MOS was a 4.24 the worst was a
  2.64.

29
Codecs Used

• The most commonly used codec is ITU-T G.711 PCMU.
• Every VoIP Hard Phone provider with an Average
  MOS over 4.0 used the ITU-T G.711 PCMU codec.
• ITU-T G.721 and ITU-T G.729 are still in use by a
  few VoIP service providers.

Note Codec used cannot be determined for
PacketCable providers and some VoIP Soft Phone
software clients with proprietary signaling
protocols. There is no codec used in the
customer premises equipment for analog PSTN
service.
30
Analog Telephone Adaptors and Software Clients
31
Summary
32
Industry Trends

• Most providers measured in previous studies are
  improving their reliability
• PacketCable providers now exceed the overall
  reliability of PSTN service.
• VoIP providers as an industry need to improve in
  service availability.
• As a whole, the industry standards in
  responsiveness and audio clarity continue to
  improve, and PacketCable service providers lead
  the other voice technologies.

33
How to Improve VoIP Quality

• Watch the competition Ensure that your service
  not only performs well all the time, but also
  performs better than or at par with your
  competition.
• Focus on end user experience Measure VoIP
  performance as close as possible to the end-user
  experience. Actual waveform analysis of call
  audio brings the measurement perspective as close
  as possible to what your customers are
  experiencing.
• Measure service holistically Small things can
  ruin the best service experience. Focus on
  measuring every aspect of your VoIP call
  experience, and use the insight gained from the
  measurements to tune your network infrastructure
  to ensure few outages and excellent call quality.

34
Public Agents Based Contact Center Monitoring
Public (Caller) Agent Infrastructure
NY
ABC Enterprise
SF
Contact Center
VoIP Network
KR
CHI
PSTN Network
IP-PSTN GW
DAL
KR
FL
Keynote Responder (Terminates VoIP or PSTN Calls)
35
Appendix B

• Measurement technology

36
Ranking Methodology Audio Quality

• The Audio Quality index ranking is based on
  Keynote extensions of the Apdex standard to
  represent user satisfaction with audio quality
• Mean Opinion Score (MOS) T, F 4.0,
  3.1
• Audio Delay (ms) T, F
  150, 400
• Each call is determined to be in the Satisfied,
  Tolerating, or Frustrated performance ranges for
  MOS and audio delay, based upon industry standard
  thresholds.
• See http//www.apdex.org/
• Thresholds based on Telecommunications
  Industry Association Technical Services Bulletin
  116 Voice Quality Recommendations for IP
  Telephony.
• Thresholds based on International
  Telecommunications Unions standard ITU-T G.114
  One-way transmission time.