Development updates and standards

1
Development updates and standards

• Ingrid Melve, UNINETT, Norway
• http//www.desire.org/
• DESIRE II project

2
The grand picture

• Web servers
• Web clients
• Stuff in between

3
The stuff in between

• Cables, switches, routers, firewalls
• Non-caching proxies
• Caching proxies or mirrors
• Buffered copies of web objects

4
Content delivery

• Mirrors, traditional FTP
• Content delivery nodes
• Finding mirror is currently witchcraft and not
  standardized
• Initiated (and paid) by content distributor

5
Proxies

• Squid and many more
• Finding proxy by configuration of browser
• manual in detail
• PAC-file (manual URL)
• intercepting traffic
• Initiated (and paid) by end user

6
Network topology

• Caching proxies need to be in the normal traffic
  flow path
• Routers and switches may intercept traffic
• Routing policy is important
• If you have too much bandwidth, consider other
  fun things than web caching

7
Distributed relationships
8
Distributed relationships

• Inter-replica
• Cooperation and communication between replica
  origin servers
• Client to replica
• Cooperation and communication between clients
  (user agents proxies) and replica origin server
• Client to proxy
• Configuration, cooperation and communication
  between user agents and a caching proxy
• Inter-cache
• Cooperation and communication between caching
  proxies
• Network element to proxy cache
• Cooperating and communication between
  cachingproxy and network elements.

9
Inter-Replica

• Replica origin servers
• Object replication
• demand driven
• batch driven
• Replication of services
• CGI-scripts with consistent performance
• Interoperability of server software
• Supplied by information owner

10
Client to Replica

• Optimal replica proximity
• Finding an object
• Finding the closest copy
• Finding the right copy
• Content Delivery (Akamai, Sandpiper etc)
• Examples
• URL redirection
• DNS redirection
• URN resolution
• FTP mirror tracker

11
Client to Proxy

• Finding the Proxy
• Manual configuration
• PAC, Proxy Auto-Configuration script
• WPAD, Web Proxy Auto-Discovery
• Caching proxy directives
• HTTP/1.1 Cache-control

12
Inter-proxy communication

• Query for objects
• ICP, Internet Cache Protocol
• HTCP, HyperText Cache Protocol
• Inform about object location in cache mesh
• CacheDigest
• Load balance
• various proprietary solutions
• HTTP/1.1 have extended support for caching
  proxies

13
Network element to proxy array

• Redirection
• SOCKS firewall (application)
• NECP, Network Element Control Protocol
  (network/transport)
• WCCP, Web Cache Coordination Protocol router
  (network/transport)
• Proprietary protocols switch (network/link)
• Transparent redirection hijacking web traffic
• Added functionality
• Load balancing
• Redundancy
• Fallback

14
Alphabet soup

• HTTP
• ICP
• HTCP
• Cache Digest
• WPAD
• WCCP
• NECP
• Experimental
• CARP
• ICP extensions

15
HTTP

• Hyper Text Transfer Protocol
• Draft Standard in IETF (this is above Proposed
  Standard)
• Complex in HTTP/1.1 version
• The big workhorse protocol of the Internet

16
HTTP headers

• Expires
• Last-Modified
• Cache-Control
• Too many web servers do the wrong thing
• lack of knowledge
• mis-configuration
• no understanding of caching
• fear of loosing control if caching is allowed

17
Cache-control

• max-age seconds that an object is valid
• s-maxage seconds that an object is valid in a
  shared cache
• public cacheable in shared caches
• no-cache must resubmit request to origin server
• must-revalidate must obey any freshness
  information
• proxy-revalidate aoplies to proxy cache

18
ICP

• Internet Caching Protocol
• Squid uses version 2
• Lots of UDP traffic
• UDP loss translates to traffic availability
• Introduces extra latency, as you wait for UDP
  round-trip
• Access control separate from HTTP

19
Cache Digest

• Hashed content listing of your cache
• Transfer via HTTP
• access control mechanism
• flow control
• Implemented in Squid
• False negatives cause problems

20
HTCP

• Hyper Text Caching Protocol
• ICP with HTTP header information
• Lots of UDP traffic
• Informational RFC

21
PAC

• Netscape Automatic Proxy Configuration
• JavaScript with proxy configuration for client
• Client needs URL of the script

22
WPAD

• Web Proxy Auto Discovery
• Mechanism, not protocol
• Finds URL for local PAC file
• Uses DHCP as starting point

23
WCCP

• Web Cache Control Protocol
• Cisco proprietary protocol
• Published as Internet-Draft
• Routers speak this with cache clusters

24
NECP

• Network Element Control Protocol
• Published as draft-cerpa--necp-02-txt
• Switches and routers speak this with cache
  clusters
• Automatic failover
• Cut-through for direct access

25
Streaming

• Streaming proxies may be efficient even without
  cache
• Streaming challenges
• many protocols
• emerging products
• incompatible solutions
• Streaming protocols
• RTSP (QuickTime)
• MMS (Microsoft Media)
• Real
• Caching valuable for on-demand services, not for
  realtime applications

26
Proxying things

• LDAP proxies are available, but no caching so far
• Streaming proxies exist, few do caching
• Caching FTP objects can be very efficient
• Caching NNTP could work, but you are probably
  better off with a NNTP only solution
• Caching SSL is not a bright idea

27
Summing up

• Replication and caching
• Competing technologies
• Complementary technologies
• Ongoing work, rapid development
• Replication is in its infancy
• Proxy caching is in puberty
• Proxy caching commercial products available
• Traffic flow is altered at the application layer
• Middleware development to be followed

28
References

• http//www.uninett.no/prosjekt/desire/workshop/re
  ferences.html
• Internet Web Replication and Caching Taxonomy
  draft-ietf-wrec-taxonomy-03.txt