Reliable and Scalable Internet Telephony

1
Reliable and Scalable Internet Telephony

• by Kundan Singh
• Advisor Henning Schulzrinne
• Computer Science Department,
• Columbia University, New York
• Feb 14, 2005

2
Agenda for the presentation

• What is Internet telephony?
• What is the problem?
• Why is it important?
• Results so far
• Difference with related work

30 slides
3
Internet telephony

• Multimedia calls over the Internet
• Signaling
• SIP Session Initiation Protocol
• Where is bob_at_example.com located?
• Media
• Audio/video codecs
• RTP Real-time Transport Protocol
• Elements (devices)
• End system, server (proxy/registrar), gateway,
  MCU,

4
Internet telephony infrastructureCINEMA
Columbia InterNet Extensible Multimedia
Architecture
CINEMA servers
Telephone switch
rtspd media server
Local/long distance 1-212-5551212
sipconf Conference server
Quicktime
RTSP
PSTN
RTSP clients
Department PBX
sipum Unified messaging
sipd Proxy, redirect, Registrar server
Internal Telephone Extn 7040
713x
SQL database
cgi
SIP/PSTN Gateway
vxml
Web based configuration
H.323
siph323 SIP-H.323 translator
NetMeeting
5
CINEMAMy contribution in design and
implementation
CINEMA Applications
RTSP media server
SIP proxy server
SIP/H.323 gateway
SIP/RTP conferencing
SIP/RTSP unified messaging
SIP/VoiceXML browser
rtspd
sipd
sip323
sipconf
sipum
sipvxml
Xerces-C
Flite Xerces-C
OpenH323
CINEMA Libraries
MySQL PWLib Resparse
and web-based GUI
C/C 58K out of 187 KLOC Tcl 30 KLOC
6
My research background
P2P VoIP using SIP
SIP Failover/load sharing
Multimedia collaboration
CINEMA user interface
Interactive voice response
Enterprise VoIP infrastructure
Mobile NAT
Libsip (SIP library)
SIP conferencing
SIP-RTSP voice mail
SIP-H.323 translator
H.323 client gateway
PhD_at_columbia
Reliability and scalability
MS_at_columbia
VoIP infrastructure
Work_at_motorola India
Undergrad_at_BITS India
1997 1999 2000 2001 2002
2003 2004 2005
7
Telephone reliability(PSTN Public Switched
Telephone Network)
bearer network
telephone switch(SSP)
8
Internet telephony(SIP Session Initiation
Protocol)
alice_at_yahoo.com
bob_at_example.com
yahoo.com
example.com
192.1.2.4
129.1.2.3
DB
9
SIP network architectureScalability requirement
depends on role
Cybercafe
ISP
IP network
IP phones
GW
ISP
MG
MG
SIP/MGC
GW
SIP/PSTN
SIP/MGC
Carrier network
MG
GW
PBX
T1 PRI/BRI
PSTN phones
PSTN
10
Reliability and scalabilityfor call routing,
registration, conferencing, voicemails

• Requirements
• Reliable
• Mean Time Between Failures (MTBF), Mean Time To
  Recover (MTTR), percentage availability
• Scalable
• Registration rate, call rate, requests/s
• Server and network components
• Proposed solutions
• Server redundancy
• Apply existing web-redundancy designs
• Evaluate quantitatively
• Peer-to-peer
• Novel P2P-SIP architecture
• Evaluate quantitatively

11
Server redundancyThe problem failure or overload
12
Server redundancyKnown techniques

• Client-based
• Cisco phones primary and backup proxy
• DNS
• NAPTR, SRV
• IP address takeover
• Database redundancy

13
High availabilityFailover in our test bed -
CINEMA
Web scripts
Web scripts
D2
D1
Slave/ master
Master/ slave
replication
P2
P1
phone.cs.columbia.edu
sip2.cs.columbia.edu
REGISTER
_sip._udp SRV 0 0 5060 phone.cs.columbia.edu
SRV 1 0 5060 sip2.cs.columbia.edu
proxy1 phone.cs backup sip2.cs
14
High availabilityMore issues

• Client re-sends INVITE to P2
• Immediately on ICMP error
• Or after 10s otherwise
• sipd has in-memory cache
• Refresh registration much before expiry
• Cisco phone registers to P1 and P2
• Web access gets delayed information

15
High availabilityMeasurements on failover
Slave/ master
Master/ slave
D2
D1
DNS

• Call setup latency
• Client retry timeout (T1), DNS TTL
• User unavailability
• None (refresh double register)
• Registration refresh interval (Tr), cache refresh
  interval (Tc), client retry timeout (T2), DB
  replication delay, DNS TTL
• Web access latency
• servers
• Tradeoff reliability vs capacity

Caller
P2
P1
T1
Callee
D2
P2
P1
D1
Tc
Td
Tc
A
Tr
T2
A
Tc
16
ScalabilityLoad sharing redundant proxies and
databases

• REGISTER
• Write to D1 D2
• INVITE
• Read from D1 or D2
• Database write/ synchronization traffic becomes
  bottleneck

P1
D1
P2
D2
P3
17
ScalabilityLoad sharing divide the user space

• Proxy and database on the same host
• Stateless proxy can become overloaded
• Use many
• Hashing
• Static vs dynamic

P1
D1
a-h
P2
D2
i-q
P3
D3
r-z
18
ScalabilityComparison of the two designs
P1
P1
a-h
D1
D1
P2
P2
i-q
D2
D2
P3
P3
D2
r-z
Total time per DB

• ((tr/D)1)TN
• (A/D) B

• ((tr1)/D)TN
• (A/D) (B/D)

D number of database servers N number of
writes (REGISTER) r reads/writes
(INVREG)/REG T write latency t read
latency/write latency
19
Reliability and scalabilityTwo stage
architecture for CINEMA
a_at_example.com
a.example.com _sip._udp SRV 0 0 a1.example.com
SRV 1 0 a2.example.com
a1
s1
a2
sipbob_at_example.com
s2
sipbob_at_b.example.com
b_at_example.com
b.example.com _sip._udp SRV 0 0 b1.example.com
SRV 1 0 b2.example.com
s3
b1
b2
ex
example.com _sip._udp SRV 0 40 s1.example.com
SRV 0 40 s2.example.com SRV 0 20
s3.example.com SRV 1 0 ex.backup.com
Request-rate f(stateless, groups) Bottleneck
CPU, memory, bandwidth? Failover latency ?
20
Reliability and scalabilityFuture work analysis
and measurement
Rp Mp
a1
Rs Ms
P11
s1
a2
S3
? ?R ?P REGISTER INVITE, etc
B2
s2
?/B
?r, ?p
s3
b1
?s
b2
ex

• When is stateless proxy stage needed
• What are the optimal values for S,B,P
• for required scalability (1-10 million BHCA) and
  reliability (99.999)
• using commodity hardware

21
Server-based vs peer-to-peer

• Server-based
• Cost maintenance, configuration
• Central points of failures
• Controlled infrastructure (e.g., DNS)
• Peer-to-peer
• Robust no central dependency
• Self organizing, no configuration
• Scalability ?

22
We propose P2P-SIP

• Unlike server-based SIP architecture
• Unlike proprietary Skype architecture
• Robust and efficient lookup using DHT
• Interoperability
• DHT algorithm uses SIP communication
• Hybrid architecture
• Lookup in SIPP2P
• Unlike file-sharing applications
• Data storage, caching, delay, reliability
• Disadvantages
• Lookup delay and security

23
P2P-SIPBackground DHT (Chord)
Key node
81 9 14
82 10 14
84 12 14
88 16 21
81624 32
83240 42
1
54
8
58
10
14
47
21

• Identifier circle
• Keys assigned to successor
• Evenly distributed keys and nodes
• Finger table logN
• ith finger points to first node that succeeds n
  by at least 2i-1
• Stabilization for join/leave

42
38
32
38
24
30
24
P2P-SIPDesign Alternatives
servers
1
54
10
38
24
30
clients
Use DHT in server farm
Use DHT for all clients But some are resource
limited
Use DHT among super-nodes
25
P2P-SIPNode architecture registrar, proxy, user
agent
Signup, Find buddies
IM, call
On reset
Signout, transfer
On startup
Leave
Find
Join
REG, INVITE, MESSAGE
Multicast REG
Peer found/ Detect NAT
REG

• DHT communication using SIP REGISTER
• Known node sip15_at_192.2.1.3
• Unknown node sip17_at_sippeer.net
• User sipalice_at_example.com

26
P2P-SIPProblems

• Mapping node identifier to SIP URI
• Node startup
• Discovery, join, maintenance
• Node shutdown or failure
• Adaptor for existing phones
• NAT/firewall traversal
• Offline messages
• Multi-party conferencing
• Security

27
P2P-SIPImplementation
31

• sippeer C, Linux, Chord
• Node join and form the DHT
• Node failure is detected and DHT updated
• Registrations transferred on node shutdown
• Co-located sipc can use sippeer service

29
31
25
26
15
28
P2P-SIPFuture work scalability evaluation

• messages depends on
• Keep-alive and finger table refresh rate
• Call arrival distribution
• User registration refresh interval
• Node join, leave, failure rates
• Mrs rf(log(N))2 c.log(N) (k/t)log(N)
  ?(log(N))2/N
• nodes f(capacity,rates)
• CPU, memory, bandwidth
• Verify by measurement and profiling

29
P2P-SIPFuture work reliability and call setup
latency evaluation

• User availability depends on
• Super-node failure distribution
• Node keep-alive and finger refresh rate
• User registration refresh rate
• Replicate user registration
• Measure effect of each
• Call setup latency
• Same as DHT lookup latency O(log(N))
• Calls to known locations (buddies) is direct
• DHT optimization can further reduce latency
• User availability and retransmission timers
• Measure effect of each

30
Summary and future work

• Internet telephony infrastructure
• Server redundancy
• Two stage architecture evaluation
• Server-less/peer-to-peer VoIP
• Quantitative evaluation
• Multi-domain deployment
• PSTN interworking

31
PublicationsConference, workshop, technical
report, magazine

• H. Schulzrinne, K. Singh and X. Wu, "Programmable
  Conference Server", Columbia University Technical
  Report CUCS-040-04, NY, Oct 2004.
• K. Singh and H. Schulzrinne, "Peer-to-peer
  Internet Telephony using SIP", New York Metro
  Area Networking Workshop, CUNY, NY, Sep 2004.
• K. Singh and H. Schulzrinne, "Peer-to-peer
  Internet Telephony using SIP", Columbia
  University Technical Report CUCS-044-04, NY, Oct
  2004.
• K. Singh and H. Schulzrinne, "Failover and Load
  Sharing in SIP Telephony", Columbia University
  Technical Report CUCS-011-04, NY, May 2004.
• K. Singh, Xiaotao Wu, J. Lennox and H.
  Schulzrinne, "Comprehensive Multi-platform
  Collaboration", MMCN 2004 - SPIE Conference on
  Multimedia Computing and Networking, Santa Clara,
  CA, Jan 2004.
• K. Singh, Xiaotao Wu, J. Lennox and H.
  Schulzrinne, "Comprehensive Multi-platform
  Collaboration", Columbia University Technical
  Report CUCS-027-03, NY, Nov 2003.
• M. Buddhikot, A. Hari, K. Singh and S. Miller,
  "MobileNAT A new Technique for Mobility across
  Heterogeneous Address Spaces", WMASH 2003 - ACM
  International Workshop on Wireless Mobile
  Applications and Services on WLAN Hotspots, San
  Diego, CA, Sep 2003.
• K. Singh, A. Nambi and H. Schulzrinne,
  "Integrating VoiceXML with SIP services", ICC
  2003 - Global Services and Infrastructure for
  Next Generation Networks, Anchorage, Alaska, May
  2003.
• K. Singh, A. Nambi and H. Schulzrinne,
  "Integrating VoiceXML with SIP services", Second
  New York Metro Area Networking Workshop, Columbia
  University, NY, Sep 2002.
• K. Singh, W. Jiang, J. Lennox, S. Narayanan and
  H. Schulzrinne, "CINEMA Columbia InterNet
  Extensible Multimedia Architecture", Columbia
  University Technical Report CUCS-011-02, NY, May
  2002.
• W. Jiang, J. Lennox, H. Schulzrinne and K. Singh,
  "Towards Junking the PBX Deploying IP
  Telephony", NOSSDAV 2001.
• W. Jiang, J. Lennox, S. Narayanan, H.
  Schulzrinne, K. Singh and X. Wu, "Integrating
  Internet Telephony Services", IEEE Internet
  Computing (magazine), May/June 2002 (Vol. 6, No.
  3).
• K. Singh, Gautam Nair and H. Schulzrinne,
  "Centralized Conferencing using SIP", 2nd
  IP-Telephony Workshop (IPTel'2001), April 2001.
• K. Singh and H. Schulzrinne, "Unified Messaging
  using SIP and RTSP", IP Telecom Services Workshop
  2000, Atlanta, Georgia, U.S.A, Sept 2000.
• K. Singh and H. Schulzrinne, "Unified Messaging
  using SIP and RTSP", Columbia University
  Technical Report CUCS-020-00, NY, Oct 2000.
• K. Singh, H.Schulzrinne, "Interworking Between
  SIP/SDP and H.323", 1st IP-Telephony Workshop
  (IPTel'2000), April 2000.
• K. Singh and H. Schulzrinne, "Interworking
  Between SIP/SDP and H.323", Columbia University
  Technical Report CUCS-015-00, NY, May 2000.

32
Backup slides
33
MobileNATArchitecture

• Two IP addresses
• Virtual IP (fixed host-id)
• Actual IP (routable changes)
• DHCP, NAT, mobility manager

CN
128.59.16.149
V135.180.32.4
Anchor node (AN)
MN
MN
135.180.32.6
A135.180.54.7
34
MobileNATComparison with other work
MIP CIP Hawaii HMIP (RR) IDMP TeleMIP MIP LR MIP RO SIP IPv6 IPv6 Mobile NAT Virtual NAT
MIP messaging Y N Y Y Y - - N Y N N N
Inter-tunnel Y Y Y Y Y N Y N O O O N
Intra-tunnel - N N Y Y - - - O O O N
Paging O Y Y Y Y - - N Y UD UD N
Host ID HA HA CoA CoA LCoA - - SIP HA CoA CoA virtual
signaling Y Data Y Y Y Y Y Y Y DHCP/MM DHCP/MM Y
CN modify? N N N N N Y Y - N N N Y
MN modify? Y Y Y Y Y Y Y - Y Y Y Y
Router modify? FA Y Y FA FA - - - O N N N
NAT support Y1 Y Y Y Y IN IN Y IN Y Y IN
Non-mobile IP nodes Y N Y Y Y - - - Y Y Y IN
Triangular route Y Y Y Y Y N N N N N/Y N/Y N
Y yes N no - N/A O optional
INindependent UD Under Development 1 We
assume Mobile IP with UDP tunneling for NAT
35
Related workIP telephony and multimedia
communication

• Unlike low cost VoIP Vonage, ATT
• We provide enterprise infrastructure
• There are enterprise IPtel Cisco, Nortel
• But redundancy architecture, interoperability,
  distributed components model differ
• Collaboration CSCW, SIGGROUP
• Unlike web-centric, or application specific
• We provide standard-based multimedia
  collaboration platform
• Multimedia conferencing Mbone, H.323
• Ours is SIP-based infrastructure, reuse existing
  tools and protocols such as RTSP, media server

36
Related workComprehensive multi-platform
collaboration

• Goal Alternate between synchronous and
  asynchronous communication, and access from
  different devices and clients.
• Synchronous (tightly coupled)
• Video conference, IM, screen sharing, floor
  control,
• Asynchronous (loosely coupled)
• File sharing, message board,
• Messaging and notifications
• Personalized view
• Per-user calendar, access control, address book

• We try to incorporate
• Long lived groups
• Design teams, committees, college classes
• Asymmetric events
• Lecture and lecture series
• Short-lived spontaneous interaction
• Current practice
• Email, teleconference
• Vendor specific tools, platform dependence
• Application specific
• E.g., collaborative software development

37
Multi-party collaborationWhat is done, and what
is left.

• Sipconf conference server
• Audio, video, IM, screen, shared browsing, floor
  control
• No XCON yet use web interface
• Small to medium size conferences
• Cascaded conference mixer
• participants, audio delay
• Failover
• State sharing between servers

38
Related workAvailability for (web) servers

• Availability f(reliability,maintainability)
• Reliability time to failure pdf
• Maintainability time to recover pdf
• Existing work on failover
• TCP connection migration
• IP address takeover
• MAC address takeover
• Reliable server pooling
• Requires new protocol support in clients
• Reliability analysis tools (www.relexsoftware.com)
  
• Availability in the face of (DoS) attacks

39
Related workScalability for (web) servers

• Existing work
• Connection dispatcher
• Content/session-based redirection
• DNS-based load sharing
• HTTP vs SIP
• UDPTCP, signaling not bandwidth intensive, no
  caching of response, read/write ratio is
  comparable for DB
• SIP scalability bottleneck
• Signaling (chapter 4), real-time media data,
  gateway
• 302 redirect to less loaded server, REFER session
  to another location, signal upstream to reduce

40
Related workSIPStone SIP server performance
metric

• Steady state rate for
• successful registration, forwarding and
  unsuccessful call attempts measured using 15 min
  test runs.
• Measure requests/s with given delay constraint.
• Performancef(user,DNS,UDP/TCP,g(request),L)
  where gtype and arrival pdf (request/s),
  Llogging?
• For register, outbound proxy, redirect, proxy480,
  proxy200.
• Parameters
• Measurement interval, transaction response time,
  register/s, calls/s, transaction failure
  probabilitylt5,
• Shortcomings
• does not consider forking, scripting, Via header,
  packet size, different call rates, SSL. Is there
  linear combination of results?

41
Related work3GPP (release 5)s IP Multimedia
core network Subsystem uses SIP

• Proxy-CSCF (call session control function)
• First contact in visited network. 911 lookup.
  Dialplan.
• Interrogating-CSCF
• First contact in operators network.
• Locate S-CSCF for register
• Serving-CSCF
• User policy and privileges, session control
  service
• Registrar
• Connection to PSTN
• MGCF and MGW

42
Related work Skype From the KaZaA community

• Host cache of some super nodes
• Bootstrap IP addresses
• Auto-detect NAT/firewall settings
• Similar to STUN and TURN
• Protocol among super nodes ??
• Allows searching a user (e.g., kun)
• History of known buddies
• All communication is encrypted
• Promote to super node
• Based on availability, capacity
• Conferencing
• Problems
• Proprietary, single service, centralized login

43
Related workP2P

• P2P networks
• Unstructured (Kazaa, Gnutella,)
• Structured (DHT Chord, CAN,)
• Skype and related systems
• Flooding based chat, groove, Magi
• P2P-SIP telephony
• Proprietary NimX, Peerio, damaka
• File sharing SIPShare

44
Why we chose Chord?

• Chord can be replaced by another
• As long as it can map to SIP
• High node join/leave rates
• Provable probabilistic guarantees
• Easy to implement
• X proximity based routing
• X security, malicious nodes

45
Related workJXTA vs Chord in P2P-SIP

• JXTA
• Protocol for communication (peers, groups, pipes,
  etc.)
• Stems from unstructured P2P
• P2P-SIP
• Instead of SIP, JXTA can also be used
• Separate search (JXTA) from signaling (SIP)

46
P2P-SIPNode Startup
columbia.edu

• SIP
• REGISTER with SIP registrar
• DHT
• Discover peers multicast REGISTER
• Join DHT using node-keyHash(ip)
• REGISTER with DHT using user-keyHash(alice_at_columb
  ia.edu)
• Dialing out
• Call, instant message, etc.
• INVITE siphgs10_at_columbia.edu
• MESSAGE sipalice_at_example.com
• Last seen, SIP NAPTR/SRV, DHT

REGISTER
alice_at_columbia.edu
Detect peers
REGISTER alice42
58
42
12
14
REGISTER bob12
32
47
P2P-SIPNode Leaves

• Graceful leave
• Un-REGISTER
• Transfer registrations
• Failure
• Attached nodes detect and re-REGISTER
• New REGISTER goes to new super-nodes
• Super-nodes adjust DHT accordingly

REGISTER key42
REGISTER
OPTIONS
DHT
42
42
48
P2P-SIPAdvanced services

• Offline messages
• INVITE or MESSAGE fails gt Responsible node
  stores voicemail, instant message.
• Conferencing
• Mixer, full mesh, multicast

49
P2P-SIPSecurity open issues (threats,
solutions, issues)

• More threats than server-based
• Privacy, confidentiality
• Malicious node
• Dont forward all calls, log call history (spy),
• free riding, motivation to become super-node
• Existing solutions
• Focus on file-sharing (non-real time)
• Centralized components (boot-strap, CA)
• Assume co-operating peers
• works for server farm in DHT
• Collusion
• Hide security algorithm (e.g., yahoo, skype)
• Chord
• Recommendations, design principles,

50
Server-based vs peer-to-peer
Reliability, failover latency DNS-based. Depends on client retry timeout, DB replication latency, registration refresh interval DHT self organization and periodic registration refresh. Depends on client timeout, registration refresh interval.
Scalability, number of users Depends on number of servers in the two stages. Depends on refresh rate, join/leave rate, uptime
Call setup latency One or two steps. O(log(N)) steps.
Security TLS, digest authentication, S/MIME Additionally needs a reputation system, working around spy nodes
Maintenance, configuration Administrator DNS, database, middle-box Automatic one time bootstrap node addresses
PSTN interoperability Gateways, TRIP, ENUM Interact with server-based infrastructure or co-locate peer node with the gateway
51
My contribution in CINEMASip-h323 signaling
translator

• Background ITU-Ts H.323
• Binary ASN.1 PER, collection of protocols (H.245,
  H.225.0, Q.931, RAS, H.450.x)
• H.323 gatekeeper similar but not same as SIP
  server
• Problems in interworking
• Multi-stage dialing in H.323v1
• Fast start in v2 is optional
• User registration
• Both SIP and H.323 users should be reachable
• Session description is more complex
• End system should select the codecs
• Security and QoS end-to-end or not?
• Solution
• List different scenarios
• No modification in SIP or H.323
• Direct RTP traffic if possible
• Implementation

52
My contribution in CINEMASipum Unified
messaging using SIP and RTSP

• Problem
• Existing systems have voicemail with PBX or
  phone, or send voice attachments in email
• Downloading the whole message is not desirable
• Solution
• Using existing standards (RTSP, SIP) and tools
  (web, media player)
• Distributed components for different
  architectures (PBX, phone, service provider)
• Many ways to retrieve your message (RTSP, SIP,
  phone, web)
• Message deletion issues
• Call reclaiming
• Implementation

53
My contribution in CINEMASipconf Centralized
conferencing using SIP/RTP

• Problem
• Multicast is not available and ad hoc conference
  is useful for small number of users
• Heterogeneous clients (some have video also or
  different audio codecs)
• Solution
• Audio mixer, video forwarder
• IM, VNC screen sharing, shared web browsing
• Playout delay adjustments
• Web based configuration, floor control
• G.711 A/Mu, G.721, DVI, ADPCM, G.722,
• Modular libconf, libmedia, rtplib
• Implementation and performance evaluation

54
My contribution in CINEMASipvxml SIP-based
VoiceXML browser

• Background
• VoiceXML for touch tone-based service programming
• Backend scripts (CGI) or servlets
• Problem
• Then existing solutions were PSTN based
• Solution
• First SIP-VoiceXML implementation
• SIP interface (works with PSTN via a gateway)
• Example cgi scripts
• Calling card service
• Joining a conference (Ajay)
• Accessing voice mail (Ajay)
• Email by phone (Pimrampai)
• Auto attendant (Sean)

55
My contribution in CINEMAlibsip SIP user
agent library in C

• All the applications (sipum, sipconf, siph323,
  sipvxml) use a common underlying library
• Similar API for H.323 defined using wrapper
  around openH323
• Unlike JAIN-SIP or SIP servlet, libsip is more
  high level with facility to access low level
  features
• Dialog, call, endpoint, registration are defined
  as objects (JAIN-SIP 1.1 added dialog as object)
• Uses underlying transaction and parsing library
  shared with sipd
• Test user agent (sipua) is used as tools, e.g.,
  for sipconf testing
• Documentation is at http//www.cs.columbia.edu/kn
  s10/software/siplib

56
My contribution in CINEMAGUI web-based user
interface

• Configuration, user profile, etc., stored in SQL
  DB
• Front end as web-based GUI
• CGI scripts in Tcl
• About 100 pages for various configuration
• User friendly (beginner vs advanced, context
  help)
• Asynchronous collaboration
• Voicemail, file sharing, IM archive, groups,
  address book, calendar
• Undergone three iterations
• See current version at http//phone.cs.columbia.ed
  u/cinema/