QoS Measurement and Management for Multimedia Services

1
QoS Measurement and Management for Multimedia
Services

• Thesis Proposal
• Wenyu Jiang
• April 29, 2002

2
Topics Covered

• Objective QoS metrics for real-time multimedia
• Subjective/Perceived quality
• Objective perceptual quality estimation
  algorithms
• Quality enhancement for real-time multimedia
• IP telephony deployment
• VoIP quality in the current Internet

3
Backgrounds and Motivations

• The Internet is still best-effort
• Needs QoS monitoring
• What to measure/monitor?
• Loss, delay, jitter
• Must map to perceived quality
• What to do if quality is not good?
• End-to-End FEC, LBR
• Network provisioning voice traffic aggregation
• IP telephony service deployment
• Current ITSPs are not doing well
• Lack of study on localized deployment
• What is the status of the current Internet?

4
How Real-time Multimedia Works

• A/D conversion Encoding Packet transmission
  Decoding Playout D/A conversion
• Dominant QoS factors
• Loss ? clipping/distortion in audio
• Delay ? lower interactivity
• Jitter ? late loss

5
Delay and Loss Measurement

• Solutions for clock synchronization
• Telephone-based synchronization
• RTT-based, assume symmetric delays
• GPS-based
• Dealing with Clock drift
• De-skewing by linear regression
• One-way vs. round-trip measurement
• Internet load often asymmetric
• One-way loss and delay are more relevant to
  real-time multimedia

6
Loss and Delay Models

• Loss Models
• Gilbert model
• Extended Gilbert model
• Others
• Delay Models
• More difficult to construct
• No universal distribution function
• Temporal correlation between delays

7
Perceived Quality Estimation
MOS Grade Score
Excellent 5
Good 4
Fair 3
Poor 2
Bad 1

• Mean Opinion Score (MOS)
• Requires human listeners
• Labor and time intensive
• Reflective of real quality
• Objective perceptual quality estimation
  algorithms
• PESQ, PSQM/PSQM, MNB, EMBSD
• Speech recognition based (new)

8
Network Provisioning for VoIP

• Silence suppression
• Saves bandwidth?statistical multiplexing
• The on/off patterns in human voice depend on the
  voice codec or the silence detector
• Voice traffic aggregation
• Multiplexing by token bucket filtering
• The on/off patterns in human voice directly
  affects aggregation performance
• Past study assumes exponential distribution

9
IP Telephony Deployment

• Localized deployment
• More practical than a grand-scale Internet
  deployment
• Can still interoperate with an IP telephony
  carrier
• Issues
• PSTN interoperability
• Security
• Scalability
• Billing

10
Research Objectives

• Objective QoS metrics
• Modeling
• Their relationship to perceived quality
• Objective perceptual quality estimation
  algorithms vs. perceived quality (MOS)
• Quality improvement measures
• End-to-End FEC vs. LBR
• Network-based voice traffic aggregation
• IP telephony deployment issues
• VoIP quality measurement over the Internet

11
Completed Work QoS Measurement Tools

• UDP packet trace generator
• Clock synchronization and de-skewing tool
• Loss and delay modeling tools
• By examining a packet trace
• Outputs Gilbert and extended Gilbert model
  parameters
• Outputs conditional delay CCDF
• Playout simulator
• Simulates several common playout algorithms
• FEC is also supported

12
Completed Work Comparison of Loss Models

• Loss burst distribution
• Roughly, but not exactly exponential
• Inter-loss distance
• Clustering between adjacent loss bursts

13
Loss Model Comparison, contd.

• Loss burstiness on FEC performance
• FEC less efficient under bursty loss
• Final loss pattern (after playout, FEC)
• Generally also bursty

14
Mapping from Loss Model to Perceived Quality

• Random vs. bursty loss
• Bursty ? lower MOS
• Effect of loss burstiness
• Sometimes very bursty loss does not lead to lower
  quality

15
A New Delay Model

• Conditional CCDF (C3DF)
• Allows estimation of burstiness in the late
  losses introduced by (fixed) playout algorithm

16
Objective vs. Subjective MOS

• Algorithms PESQ, PSQM, PSQM, MNB, EMBSD

Using Original Linear 16 samples as reference
signal
Using G.729 no loss clip as reference signal
17
Objective MOS Correlation, contd.

• Second test set
• Stronger saturation effect observed for MNB1
  and MNB2, but not for PESQ

Linear-16 reference signal
G.729 reference signal
18
Auditory Distance vs. MOS

• EMBSD and PSQM appear to have the largest
  spread, i.e., least correlation w. MOS
• PSQM seems to be similar to MNB in terms of
  correlation

19
Auditory Distance vs. MOS, contd.

• Second test set
• Similar behaviors observed

Linear-16 reference signal
G.729 reference signal
20
Analysis of Objective MOS Correlation

• Quantitative metric
• Correlation coefficient ?
• But it does not tell everything!

Algorithm Test Set 1 Test Set 1 Test Set 2 Test Set 2
Algorithm ?l16 ?g729 ?l16 ?g729
MNB1 0.897 0.885 0.767 0.798
MNB2 0.910 0.935 0.844 0.870
PESQ 0.888 0.902 0.892 0.910
21
Speech Recognition Performance as a MOS predictor

• Evaluation of automatic speech recognition (ASR)
  based MOS prediction
• IBM ViaVoice Linux version
• Codec used G.729
• Performance metric
• absolute word recognition ratio
• relative word recognition ratio

22
Recognition Ratio vs. MOS

• Both MOS and Rabs decrease w.r.t loss
• Then, eliminate middle variable p

23
Speaker Dependency Check

• Absolute performance is speaker-dependent
• But relative word recognition ratio is not

24
Speech Intelligibility Results

• Human listeners are asked to do transcription
• Human recognition result curves are less smooth
  than MOS curves.

25
Analysis of Voice On-Off Patterns

• Past study finds spurt gap distributions to be
  exponential
• Modern voice codecs and silence detectors have
  different behaviors

26
Voice Traffic Aggregation

• Simulation environment
• DiffServ token bucket filter
• Exponential, CDF and trace-based model
  simulations
• N voice sources
• Token buffer size B (packets)
• R ratio of reserved vs. peak bandwidth
• Key performance figure
• Probability of out-of-profile packet

27
Aggregation Simulation Results

• Results based on G.729 VAD
• CDF model resembles trace model in most cases
• Exponential (traditional) model
• Under-predicts out-of-profile packet probability
• The under-prediction ratio increases as token
  buffer size B increases

28
Simulation Results, contd.

• Results based on NeVoT SD (default parameters
  high threshold, long hangover)
• Similar behavior, although the gap between
  exponential and CDF model is smaller for NeVoT
  case

29
Comparisons of FEC and LBR

• Forward error correction
• Bit-exact recovery
• No decoder state drift upon recovery
• Low bit-rate redundancy (LBR)
• Just the opposite to FEC
• Design of an optimal LBR algorithm
• State repair via redundant codec
• Optimal packet alignment
• MOS quality verified to be better than the rat
  LBR
• Allows a more fair comparison with FEC

30
MOS Quality of FEC vs. LBR

• FEC shows a substantial and consistent advantage
  over LBR
• This is true for all LBR configurations we tested
• Main codec is G.729 except for AMR LBR

DoD-CELP LBR
DoD-LPC LBR
31
MOS of FEC vs. LBR, contd.

• AMR LBR narrowest gap with FEC
• (Not shown here) FEC out-performs LBR under
  random loss as well

G.723.1 LBR
AMR LBR
32
Optimizing FEC Quality

• Packet interval? ? loss burstiness? ? FEC
  efficiency ?
• Result FEC MOS performance also improves

33
Optimizing Conversational MOS for FEC

• A larger packet interval ? more delay
• Trade-off between quality and delay
• The E-model
• Considers both delay and loss (and many other
  transmission quality factors)
• Optimizing FEC MOS with the E-model

34
Optimizing FEC MOS, contd.

• Validating E-model based prediction with real MOS
  test results

35
Localized IP Telephony Deployment Architecture

• Component based and distributed architecture
• Allows easy integration of all SIP-compliant
  devices and programs

36
Deployment Issues

• PSTN interoperability
• T1 configuration and PBX integration
• T1 line type (Channelized vs. ISDN PRI)
• Line coding and framing (layer 2)
• Trunk type Direct-inward-dialing (DID)
• Access permission on the PBX side
• SIP/PSTN gateway configuration
• Dial-peer locates the proper SIP server or PSTN
  trunk
• Dial-plan (translating calls from/to PSTN)

37
Deployment Issues, contd.

• Security
• Issue gateway has no authentication feature
• Solution
• Use gateways access control lists to block
  direct calls
• SIP proxy server handles authentication using
  record-route
• Allows easier change in authentication module
  (software-based)
• Certain users can only make certain gateway calls
  
• Scalability
• SIP server (DNS SRV scaling)
• Gateway voice-mail server conference server
• Billing
• Initial implementation via transaction logging

38
On-going Research

• Measurement of the current Internet
• How well can it support VoIP?
• Or, how easy can VoIP applications adapt to
  (unfavorable) network conditions?
• How fast does network condition change?
• Can network redundancy help improve VoIP quality?
• Physical redundancy (access links)
• Virtual redundancy (overlay networking)

39
Conclusions

• Completed research relating to many aspects of
  real-time multimedia, in particular VoIP
• On-going work calls for
• A comprehensive measurement of the Internet
• Analysis of the to-be measurement data
• An answer to the question how good is it today,
  and, how much better can we do?