Adaptive Multimedia Streaming

1
Adaptive Multimedia Streaming

• Oussama Layaida, Daniel Hagimont, Slim Ben
  Atallah
• Projet SARDES INRIA/Rhone-Alpes
• Journées Académiques Microsoft
• 19-21 April 2004

2
Road Map

• Problem statement
• State of the art
• Approach and Prototype implementation
• Experimental results
• Conclusion

3
Problem statement

• Distributed multimedia applications are very
  resource sensitive
• Network resources bandwidth,
• Processing resources CPU, memory
• Displaying resources Screen size, graphical
  card,
• Resources heterogeneity
• Fluctuating resources
• QoS management

4
Problem Statement (1) Resources heterogeneity
Video On Demand
TV braodcast

• Resource characteristics may vary between several
  launches
• Network (bandwidth, latency, loss rate, etc.)
• Hardware (screen size, video card, memory, CPU,
  storage, etc.)
• Software (encoding format, protocol, etc.)

10Mb/s
30Kb/s
10Mb/s
9600b/s
256 Kb/s

• PC/ADSL
• Hardware
• Software

Workstation/ Ethernet

• PDA/Wireless
• -Network
• Hardware
• Software

• mobile/GSM
• Network
• Hardware
• Software

5
Problem Statement (2) Fluctuating resources
Video On Demand
TV braodcast

• Resource availability may vary at run time
• Network capabilities (Internet, Wifi, )
• Terminal capabilities (CPU, Memory Battery)
• Software capabilities (Unexpected encoding
  format)

10Mb/s
30Kb/s
10Mb/s
9600b/s
256 Kb/s
PC/ADSL
Workstation/ Ethernet
PDA/Wireless
mobile/GSM
6
QoS management

• How to provide QoS management
• Take into account launch time and run time
  resources variation
• Best tradeoff for the user
• Limit modification of the application

7
State of the art

• Middleware-based approaches
• Resource reservation BLAKE98
• Extended component (object) models ATI03
  PYR03
• QoS Management in network or system layers
• Application-based Approaches (ZIN99)
• Server-based SisalemSchulzrinne98 Needs
  server code modification
• Client-based McCanne96 Needs client code
  modification
• Network-based Rejai00 Fox97 Smith98

8
State of the art (1) Server-based approach

• Insert adaptation machinery at each end server
• Client notifies its state (loss rate, jitter,
  etc.)
• Server adapts its transmission rate (e.g encoding
  parameters) according to conditions of receivers
• Ref. SCH98
• Do not scale to multipoint communication
• A single stream cannot fit requirements of
  heterogeneous clients

30, 256 Kb/s or 10 Mb/s ?
9
State of the art (2) Client-based approach

• Using layered encoding and transmission.
• Split data into multiple layers
• Receivers choose how many layers they wish to
  receive
• Ref. MCC96
• Address only adaptation to network capacities
• Do not consider host and software capacities
• Not always possible to split content into layers
  (e.g. Special encoders modifying software
  client)

10
State of the art (3) Network-based approach

• Proxies
• Computation added inside the network, between
  clients and servers
• Dynamically adapt rate and content to fit clients
  requirements
• Existing work
• Generally to cache and transcode discrete data
  (e.g. Web pages, Images, etc.) Ref. IBM00
• Protocol conversion (Multicast to unicast, or to
  webcast, etc) Ref.Joh01
• Encoding format specific (MJPEG to H.261, MPEG
  filtering) Ref.AMR95LEI02
• Provides only partial solutions, lack
  reconfiguration features,

11
State of the art (4) synthesis

• Server-based and client-based approaches are
  intrusive
• Existing network-based approaches lack
  flexibility for
• Configuring (at launch time)
• Reconfiguring (at run time)

12
Approach overview (1)
Configuration
Reconfiguration
Media Streams
Server
Client
Adaptation
Proxy

• Adaptation processing incoming streams and
  delivering them
• Configuration instantiation of an adaptation
  process according to server and client properties
  at launch time
• Reconfiguration whenever the conditions change,
  the adaptation may be modified at run time

13
Approach overview (2)
APSL Specifications
Specification level
APSL Compiler
Component-architecture
Configuration policies
Reconfiguration Manager
Configuration Manager
Resources states
Run time
Launch time
Resources Probes
Control level
Qos values
Reconfiguration actions
Configuration
Streaming Execution level
Media Session
Multimedia Components
14
Adaptation Proxy Specification Language (APSL)

• XML-based language which allows description of
• Incoming and outgoing streams
• The adaptation process to apply on streams
• An acyclic graph of interconnected components
• Attributes of these components
• The reconfiguration policy
• Probes and events to monitor execution
• Conditions to trigger reconfigurations
• Actions to perform reconfigurations

15
An APSL specification MJPEG-MPEG1 Transcoder

• rialpes.fr5008" /
• m.inrialpes.fr5012" /
• /
• height"144" width"176" /
• frame-rate"25" quality"80" fmt"32"/
• id"EVT1" op"exceeds" value"10" unit""/
• id"EVT2" op"exceeds" value"50" unit""/
• priority"1" status"active"
• ref"id(E)_at_quality" value"5"/

16
Multimedia components

• Basic building blocs of adaptation processes
• Encoding, decoding, sending, receiving, effects
• One or more in (or out) put streams
• Configuration interface
• Networking components for protocols (TCP, UDP,
  RTP, HTTP )
• Processing components which transform streams
  (decoders, encoders, resizers )

17
Configuration manager

• Build an adaptation process from an APSL
  specification
• Check validity of the APSL specification
• Create a composite component (Media session) to
  encapsulate the whole architecture
• Create basic components
• Interconnect them
• Start the media session

18
Reconfiguration manager

• Implements the reconfiguration policies with
  additional components
• Probes catch the defined events
• Actuators implement reconfiguration actions
• 2 kinds of reconfigurations
• Parameter reconfigurations
• Modification of components attributes
• Quality factor, size
• Architectural reconfiguration
• Modification of the component architecture of the
  adaptation process
• Component addition or replacement

19
Configuration and reconfiguration
Probe
EVT2
Reconfiguration Manager
Actuator (C2)
Actuator (C1)
EVT1
IMediaSession 
IEncodeur 
IRTPOut 
setQuality setFrameRate
getLossRate getRTT .
removeComponent addComponent ConnectComponent Disc
onnectComponents
H.261 Encoder
RTP receiver
MJPEG Decoder
Resizer
RGB to YUV
RTP sender
MPEG Encoder
Media Session
20
Prototype implementation

• APSL
• C XML parser which translates APSL
  specifications into component-based adaptation
  processes
• Multimedia DirectShow components
• Programming model that relies on COM technology.
• A component, called  Filter , is a COM object
  encapsulating a basic task on multimedia data
  (decoding, encoding, etc.).
• Filters are connected throw their pins, and
  accept typed data (Type, format, Sample
  size.etc.).
• Additional components developed through
  inheriting from DirectShow base classes
• Represent 13.000 lines of C
• 7000 lines new components
• 6000 lines (re)configuration of the proxy

21
Experimental results

• Experimental environment
• Configuring media sessions on the proxy
• deployment time (cost of configuration)
• performance of adaptations on the proxy
• Impact of adaptations
• on client performance
• on energy consumption
• Reconfiguring media sessions
• reconfiguration time (cost of reconfiguration)
• Impact of reconfiguration
• on client performance

22
Experimental environment

• ServerProxy PC (Pentium 4 2GHz 256 Mb)
  running Windows 2000
• Terminal PDA Compaq Ipaq H3970 (PXA250 400MHz
  64 Mb) running Pocket PC 2003
• Wirless network 802.11b
• LAN Ethernet 100

TranscodingProxy
MPEG Files on HTTP server
802.11b WaveLan
Ethernet 100 Mbps
HTTP RTP
Wireless PDA
H.261 Live video server using RTP
Video streaming applications H.26x, MPEG
clients.
23
Configuring media sessions

• Objectives
• Measure deployment time (cost of configuration)
• Measure CPU consumption for video adaptation
  (ability for the proxy to support the load at
  runtime)
• Considered 5 scenarios

24
Configuring media sessions

• Scenario1 A proxy configuration receiving
  multicast video streams which are forwarded to a
  unicast client
• Scenario 2 A HTTP-to-RTP stream conversion proxy
  with a transcoding from MPEG to H.261

25
Configuring media sessions

• Scenario 3 An H.263 to H.261 video transcoding
  proxy
• Scenario 4 An H.263 to H.261 video transcoding
  proxy with 2 receivers

H.261 Encoder
RGB to YUV
RTP sender
RTP receiver
H.263 Decoder
Duplicator
Resizer
RGB to YUV
RTP sender
H.261 Encoder
26
Configuring media sessions

• Scenario 5 A video conferencing server allowing
  mixing of 4 incoming video streams

27
Configuring media sessions

• CPU consumption
• A PC can easily support the load

• Configuration time
• Acceptable
• Can benefit from already-loaded dll

28
Impact of adaptations on client performance

• Evaluation with the videoconferencing application
  (scenario 5) using the VVP videoconferencing tool
• 2 configurations
• Without adaptation
• each participant sends his video in multicast
• each participant selects the video to display
• With adaptation
• A proxy which mixes all streams into one

29
Impact on client performance

• Without adaptation data bit rate around 900 Kbps
  and frame rate under 10
• With adaptation data bit rate around 200 Kbps
  and frame rate reach the original value (25
  frames/sec)

30
Impact on energy consumption

• Transcoding a video stream can save energy
• Used notification of PDAs battery driver when
  battery level changes by 10

31
Reconfiguring media sessions

• Reconfiguration replacement of the encoder
  component at runtime
• MPEG - H.261
• Reconfiguration time 43 ms
• Initialization of the new component 28 ms
  (before application interruption)
• Replacement of the component 15 ms (the time the
  application is paused)

32
Impact of reconfiguration on QoS

• Before reconfiguration packet loss around 30
  and frame rate under 20 fps
• During reconfiguration frame rate decreases to
  0, blackout time of 500 ms (even if proxy is
  paused only during 15 ms)
• After reconfiguration packet loss under 10 and
  frame rate at 25 fps

33
Conclusions

• Lessons learned
• Proxy-based adaptations allow managing QoS
  transparently wrt applications
• Adaptations are intrinsically dynamic they have
  to be configurable and reconfigurable
• A component-based framework provides the right
  features
• Microsoft Directshow provides the best tradeoff
  between flexibility and performance
• Future work
• Preventing unstable reconfiguration behaviors
  (e.g. ping-pong effect)
• Studying consistency of reconfiguration policies
• Distributed reconfigurations, in particular
  reconfigurations involving both the proxy and the
  PDA

34
Bibliography

• AMR95 E. Amir, S. McCanne, H. Zhang. An
  Application Level Video Gateway. November 1995.
• CC/PP W3C, Client Capacities/Preferences
  Profiles A user side framework for content
  negociation, http//www.w3.org/.
• DX8.1 Directx 8.1 Programmers guide,
  http//msdn.microsoft.com/directx/.
• IBM00 IBM Inc. Internet Transcoding for
  Universal Access. Septembre 2000.
• http//www.research .ibm.com/networked_data_syst
  ems/transcoding/index.html.
• Joh01 M. Johanson, A RTP to HTTP video gateway.
  In World Wide Web pages 499-503, 2001.
• LEI02 Z.Lei and N.D.Georganas, H.263 Video
  Transcoding for Spatial Resolution Downscaling,
  April 2002.
• MCC96 S. McCanne, V. Jacobson, M. Vetterli,
  Receiver-driven Layered Multicast, August 1996.
• MAY99 K. Mayer-Patel and L. Rowe, Exploiting
  spatial parallelism for software-only video
  effects, 1999 .
• OOI01 Wei Tsang Ooi and Robbert van Renesse,
  Distributing media transformation over multiple
  media gateways, October 2001.
• SCH98 D. Sisalem, H, Schulzrinne, The
  loss-delay adjustment algorithm a TCP-friendly
  adaptation scheme, July 1998.
• ATI03 Atighetchi M and all. Building
  Auto-Adaptive Distributed Applications The
  QuO-APOD Experience. The 23th International
  Conference on Distributed Computing Systems
  Workshops, 3rd International workshop on
  Distributed Auto-Adaptiveand Reconfigurable
  Systems (ICDCS-DARS03). May 2003 .
• BLAK98 Blake S, Black D, Carlson M, Davies D,
  Wang Z, and Weiss W. An Architecture for
  Differentiated Services.IETF, RFC 2475, 1998.
• PYR03 Pyralli I, Schmidt D. C, Cytron R. K.
  Techniques for Enhancing Real-time CORBA Quality
  of Service. In Proc. of IEEE Special Issue on
  Real-Time Systems, 91(7), 2003.
• ZIN99 Zinky J., Schulzerinne H. Comparison of
  Adaptive Internet Multimedia Applications.
  Transactions on Communications, Vol. E82-B. June
  1999.