XML, REST, and SOAP at Yahoo

1
XML, REST, and SOAP at Yahoo!

• Parand Tony Darugar
• darugar_at_yahoo-inc.com
• Nov 16th, 2005

2
Yahoo! Developer Network
Committed to being open http//developer.yahoo.com

• Flickr
• Best photo sharing community
• Maps
• Music
• RSS Feeds
• Shopping
• Comparison shopping

• Search APIs
• Web Search
• News, Local
• Audio, Image, Video
• Content Analysis
• My Web (bookmarks tags FOAF)
• Upcoming.org
• Community-driven calendar
• Travel
• Widgets (Konfabulator)

3
Flickr API

• Photo sharing, tagging, community
• Fully programmable
• most complex Y! API implementation
• SOAP / REST / XML-RPC
• Geotagging maps pictures to locations
• About 80 API calls
• Samples in various languages available

4
Flickr Mashups

• Flickr
• Spell
• Mappr

5
Upcoming.org APIs

• A community-driven event calendar where users
  interact with their friends around events.
• 22 REST calls to work with
• events
• metros
• venues
• categories
• watchlists
• Users
• RSS feeds for Metros, Groups, and user specific
  events as well as parameterizes search.

6
Yahoo! Search APIs

• Content analysis Term and concept extraction
• Local search Maps Mashup
• Vertical search (e.g. Web / News)
• Advanced search embed Yahoo! Search in your
  applications

7
Other Mashups

• Pirate Maps!
• http//justin.everett-church.com/
• Yahoo!
• Wikipedia
• People Maps
• http//hackdiary.com/

8
What is Yahoo Search Marketing?
9
What is Yahoo Search Marketing?
10
YSM XML Usage

• XML very heavily used almost every application
  will use XML in some way
• REST heavily used externally accessible
  services available for 5 years, very heavy
  traffic
• SOAP heavily used the default method for
  connecting internal systems

11
Why XML? Internal Use

• Clean definition of interfaces between teams
• Teams negotiate XML interfaces between their
  systems, define performance, security
  requirements, develop independently
• International
• cross-market, multi-language
• Standard
• Easy to learn (lots of books, resources)
• Expertise widely available
• Tools available

12
Why XML? External Use

• Its the obvious choice for external APIs.
• Same reasons
• Clean definition of interfaces
• XML schemas published for the interfaces, along
  with volumes of documentation
• International
• cross-market, multi-language
• Standard
• Easy to learn (lots of books, resources)
• Expertise widely available
• Tools available

13
XML Usage

• Definition language is XML Schema
• Developers are expected to know how to process
  XML, schema
• Parsers best tool for the job is selected
• XML Beans is getting increased usage
• XML is ever-present

14
REST Usage

• Rich set of external APIs, available for 5 years
• Rich set of third party tools developed against
  these APIs
• Very heavy usage
• Many lessons learned

15
REST Lessons Learned

• Scaling
• You can scale the implementation while keeping
  the interface stable good
• Interface is golden
• Invest in creating good interfaces
• Youll end up supporting them for a long time!
• Versioning and staging
• Staging environment is key for successful
  adoption
• Versioning is complicated
• Documentation and support
• Invest in documentation or pay for it in support
• Be prepared for abuse
• If someone can game the system to their benefit,
  they will

16
SOAP Usage

• Default API between internal systems
• Driving to an SOA
• WS-I profiles are used
• Internal guidelines and team to ensure consistent
  API definitions
• Web Services Management Tier is used to separate
  infrastructure from service implementation

17
Web Services Management Tier

• Infrastructure services provided via proxy tier,
  separate from the SOAP implementation
• Auth Auth
• Licensing and Rate Limiting
• Monitoring and SLAs
• Logging and Events
• Management and Directory

18
WS Management Tier Requirements

• Invisible require no code changes to services
• Fast negligible latency
• Scalable must handle all SOAP traffic
• Reliable infrastructure can never be down
• Extensible must integrate with
• Auth Auth (ID Management system)
• Logging / Monitoring system
• Build/QA/Deployment practices

19
REST vs. SOAP

• Source of a lot of healthy debate
• REST has much higher adoption rates in the wild
• Existing HTTP infrastructure can be immediately
  repurposed for REST
• Security
• Abuse prevention
• Caching
• Etc
• SOAP offers richer set of built-in capabilities,
  standards, tools
• SOAP is much more complicated

20
Parands Pet Peeves and Pontifications

• XML Namespaces
• Protect the developer from XML, aka automated
  object-XML mapping
• SOAP is slow not true
• SOAP actually is slow with Java!
• No WS-Security adoption by the major public API
  providers
• Clean interfaces SOA is a good thing
• XML documents Very dynamic or fairly static?

21
XML Namespaces

• Single biggest source of headaches when dealing
  with XML
• XML actually is simple without namespaces
• See my rants in other places.

22
Heard from big-time financial CIO

• We need to protect the developer from XML
• The idea that there must be a layer of
  protection, abstraction between the developer and
  XML
• Automated XML-Object mapping
• Why?
• XML processing is too complicated. Parsing, SAX,
  and DOM are too raw a set of interfaces for
  developers to deal with. XPath (or something like
  it) should be the default access mechanism.

23
Speed of SOAP

• The slowness of SOAP is often cited as a reason
  to use alternative technologies, yet
• We know SOAP can be fast. With C/C based
  implementations, we see thousands of requests per
  second per server with low latency, yet
• SOAP with Java is very slow, providing about 100
  requests per second per server with relatively
  high latency
• Somebody fix SOAP on Java!
• SOAP is an excellent enabler of horizontal scale
  the service provider can typically be scaled by
  throwing more hardware at it.

24
WS-Security Non-adoption

• None of the big public API providers are using
  WS-Security everybody is rolling their own!
• Why?
• WS-Security is complicated
• Encrypting selected parts of the document is an
  expensive, high latency operation
• Use case is rare
• Auth and Auth is primarily whats needed
• SSL works well today, is well understood
• Lets create a WS-Security AuthAuth subset to
  help foster its adoption

25
Clean Interfaces SOA

• Is code sharing a good thing?
• Only if its managed!
• Share code haphazardly and you end up with lots
  of hidden dependencies.
• XML interfaces and SOAs provide excellent
  protection against this (as do other well known
  practices around packaging and library
  management).

26
XML Very Dynamic or Fairly Static?

• XML has two high level use case scenarios
• Content, publishing oriented
• Eg. OpenDocument, DocBook,
• Structure of documents highly dynamic, varied
• Service, API oriented
• Eg. REST, SOAP
• Structure of exchanged documents fairly static
• Both are valid, but lead to different
  optimizations.