Interoperability Standards

1
Interoperability Standards Searching Multiple
Repositories

• Ralph LeVan/OCLC
• Ray Denenberg/Library of Congress

2
The Problem

• How do I provide a common interface for my users?
• How do I combine results from multiple sources?

3
How do I provide a common interface for my users?

• How do I convert my queries into the Content
  Providers (CPs) queries?
• How do I ask for 10 records?
• How do I ask for more records?
• How do I interpret their response?

4
How do I convert my queries into the CPs queries?

• My user said authortwain and titlehuck finn
• Google expects twain huck finn
• Z39.50 twain/1100342 huck finn/1441 and
• Lucene creatortwain and titlePhrasehuck finn

5
How do I ask for 10 records?

• Amazon wont let you
• RedLightGreen MAXRECORDSn
• British Library recordsn

6
How do I ask for more records?

• Amazon pagen
• RedLightGreen STARTINDEXn
• British Library startn

7
How do I interpret their response?

• How many records did I retrieve?
• Did something go wrong?
• How do I convert the CPs records into something
  my users will recognize?

8
How many records did I retrieve?

• Amazon
• 663?5FencodingUTF8keywordspratchettrhi3Aaps
  2Ck3Apratchett2Ci3Astripbookspage1"Books
  nbsp(334)
• RedLightGreen
• Viewing 1-10 of 239 results
• British Library
• 190lts

9
Did Something Go Wrong?

• RedLightGreen
• We didn't find any matches
  for dog and.
• British Library
• Nothing found due to an error
• Too many hits. Refine your
  request.

10
How do I convert the records?

• Amazon
• width"100" cellpadding"0" cellspacing"0"
• id"Td0"
• cellspacing"0"
• cellspacing"0"
• border"0" cellpadding"0" cellspacing"0"
• 21/sr8-1/qid1142436987/refpd_bbs_1/002-2019116-
  8269663?5FencodingUTF8"ges-amazon.com/images/P/0060815221.01._PIsitb-st-a
  rrow,TopLeft,-1,-14_SCTHUMBZZZ_.jpg" width"55"
  alt"Thud! (Discworld, Book 32)" height"82"
  border"0" /
• cellspacing"0" border"0"
• 21/sr8-1/qid1142436987/refpd_bbs_1/002-2019116-
  8269663?5FencodingUTF8"class"srTitle"Thud! (Discworld, Book
  32)
• by Terry Pratchett (Hardco
  ver
• - Sep 13, 2005)
• class"aliasName"Books href"/gp/search/refsr_nr_seeall_1/002-2019116-82
  69663?5FencodingUTF8keywordspratchettrhi3Aa
  ps2Ck3Apratchett2Ci3Astripbooks"See all 334
  items
• href"http//www.amazon.com/gp/product/0060815221/
  sr8-1/qid1142436987/refpd_bbs_1/002-2019116-826
  9663?5FencodingUTF8"Buy new
  nbsp24.95
  15.72

11
Converting Records Cont.

• RedLightGreen
• class"titleText"information about this title." href"ucw.servlets.
  UCWController?ACTIONEDITIONampWORKID21537371a
  mpLANGUAGEENGampMATERIALbooksampFROMRSLT3
  ampFROMWORK1amplangenglish"Hogfather
  , by Terry Pratchett
  3 editions published
  between 1996 and 1998 in English.
  Primary
  Subject Discworld Imaginary Place -
  Fiction
  " height"3" width"10" alt"A title's position
  in a search result is based on relevancy (how
  closely your search terms match the description)
  xAand availability (how many libraries have a
  copy of the title)."/s/white.gif" height"3" width"1"/src"/ucwprod/web/images/green.gif" height"3"
  width"10" alt"A title's position in a search
  result is based on relevancy (how closely your
  search terms match the description) xAand
  availability (how many libraries have a copy of
  the title)."/gif" height"3" width"1"/src"/ucwprod/web/images/green.gif" height"3"
  width"10" alt"A title's position in a search
  result is based on relevancy (how closely your
  search terms match the description) xAand
  availability (how many libraries have a copy of
  the title)."/gif" height"3" width"1"/src"/ucwprod/web/images/green.gif" height"3"
  width"10" alt"A title's position in a search
  result is based on relevancy (how closely your
  search terms match the description) xAand
  availability (how many libraries have a copy of
  the title)."/gif" height"3" width"1"/src"/ucwprod/web/images/gray.gif" height"3"
  width"10" alt"A title's position in a search
  result is based on relevancy (how closely your
  search terms match the description) xAand
  availability (how many libraries have a copy of
  the title)."/gif" height"3" width"1"/eborder"0" cellpadding"0" cellspacing"0"
  width"100"colspan"2"f" height"1"/xmlns"http//www.w3.org/TR/REC-html40"
  border"0" cellpadding"3" cellspacing"0"
  width"100"align"right" class"highlightcell"class"titleText"2.

12
Converting Records Cont.

• British Library
• Thud! / Terry Pratchett.
• http//catalogue.bl.uk/F/-?funcdirect-do
  c-setdoc_number013220851l_baseBLL01fromA9Ope
  nSearch
• Pratchett, Terry. London
  Doubleday, 2005. . ISBN 0385608675 (hbk.)
  17.99 . (Added 20050614 )

13
How do I combine results from multiple sources?

• Things you might want the server to do for you
• Common Record Format
• Common Sort Order
• Common Rank Order

14
Functional Matrix
15
The Old Solutions

• Screen Scraping
• Private APIs
• Z39.50

16
Screen Scraping

• A query has to be generated and embedded in a CP
  specific URL
• Code has to be written to examine the HTML
  returned by a CP
• Prone to breakage
• Web sites change formatting frequently
• Every site is unique
• Separate code to be maintained for every site

17
Private APIs

• Often only a slight improvement over screen
  scraping
• Provides documentation on how to construct the
  URL
• Might provide documentation on how to construct
  the query
• Might guarantee a stable response format
• Still requires unique code for each site

18
Z39.50

• Guarantees a standard request and response
• But
• Not HTTP or HTML
• Binary encoding over raw TCP/IP
• Complicated
• 11 services
• 7 extended services
• Easy to be compliant and not interoperable
• Unfriendly
• The response to a protocol error was to drop the
  connection

19
Why Use A Standard API?

• Defined requests and responses
• Reusable code across sites
• Open Source code

20
The New Solutions

• OpenSearch 1.1
• MXG
• Levels 0-2
• SRU

21
OpenSearch 1.1

• From Wikipedia
• OpenSearch is a collection of technologies that
  allow publishing of search results in a format
  suitable for syndication. It is a way for search
  engines to publish their search results in a
  standard and accessible format

22
OpenSearch 1.1 (cont.)

• Defines a Description Record with information
  about the CP
• ShortName and LongName
• Description
• Tags
• URL template
• Example
• http//herbie.bl.uk9080/opensearch.xml

23
OpenSearch 1.1 (cont.)

• URL Template
• Server Indicates how to specify OpenSearch
  request parameters
• Parameters not specified in the template are
  unavailable
• The only mandatory parameter is searchTerms
• template"http//herbie.bl.uk9080/cgi-bin/OSxml1.
  cgi/?qsearchTermsstartstartIndex?records
  count?formatrss" /

24
OpenSearch 1.1 (cont.)

• Request Parameters
• searchTerms
• count
• startIndex
• startPage
• language
• outputEncoding
• inputEncoding

25
OpenSearch 1.1 (cont.)

• Uses RSS 2.0 with a few extra elements for the
  response
• RSS define title, description and link elements
• OpenSearch adds the totalResults, startIndex,
  itemsPerPage, link and Query elements
• http//herbie.bl.uk9080/cgi-bin/OSxml1.cgi/?qlev
  anformatrss

26
Functional Matrix
Key ?Full Support ?Limited Support
27
Cool Feature

• The RSS mechanism in OpenSearch provides the
  ability to have persistent and periodic queries!

28
NISO MetaSearch XML GatewayMXG

• MXG has been designed to provide a low
  implementation barrier to content providers that
  want to make their databases available to
  metasearch engines.  Interoperability across
  content providers was explicitly not a goal of
  MXG

29
MXG Levels of Support

• Level 0 Requests are simple URLs using any
  query grammar and responses are XML records
• Level 1 Adds a description record for the
  database
• Level 2 Support a limited subset of a standard
  query grammar CQL

30
MXG Request

• Version (mandatory)
• Query (mandatory)
• StartRecord
• MaximumRecords
• http//alcme.oclc.org/MXG/search/ORPubs?version1.
  1query"levan"startRecord1maximumRecords10

31
MXG Response
xmlns"http//www.loc.gov/zing/srw/"
1.1 10mberOfRecords
1sition
1.1 quotstuff
quot t
32
MXG Response Records

infosrw/schema/1/dc-v1.1
xmling
1n
33
MXG Response recordData

• ict" xmlnsdc"http//purl.org
  /dc/elements/1.1/"
  xmlnssrw_dc"infosrw/schema/1/dc-v1.1"
  rrl1234
  Dog and
  Cat

34
MXG Error Messages

• ing/srw/diagnostic/"
• infosrw/diagnostic/1/51
• 66ntqk
• http//www.loc.gov/z3950/agency/zing/srw/diagnosti
  cs-list.html

35
Functional Matrix
Key ?Full Support ?Limited Support
36
MXG Level 1
Add a description record for the
database http//www.loc.gov/z3950/agency/zing/srw/
explain.html http//alcme.oclc.org/MXG/search/ORP
ubs
37
Functional Matrix
Key ?Full Support ?Limited Support
38
MXG Level 2
Support a limited subset of a standard query
grammar CQL Supports indexes and
Booleans http//www.loc.gov/z3950/agency/zing/cql/
http//alcme.oclc.org/srw/search/ORPublications?v
ersion1.1querydc.authorlevanmaximumRecords1
39
Functional Matrix
Key ?Full Support ?Limited Support
40
SRU

• MXG Level 2 Plus
• Full Query Grammar (CQL)
• Full Sort Specification

41
CQL Common Query Language

• Loosely based on CCL Search
• Boolean Proximity Operators
• Index Sets Indexes
• String Indexes vs. Keyword Indexes
• Truncation Characters , ?
• Relations , all, any, exact, within
• Example
• dc.titleharry potter or bib1.isbn123-456-78x

42
Sort

• sortKeys parameter with the following comma
  separated values specified
• Xpath (path to the element to be sorted on)
• Schema (that the xpath comes from)
• Ascending (value is 1true or 0false,
  defaulttrue)
• CaseSensitive (value is 1true or 0false,
  defaultfalse)
• missingValue (values are omit, abort, highValue
  or lowValue, defaulthighValue)
• e.g. sortKeystitle,onix,0

43
Functional Matrix
Key ?Full Support ?Limited Support
44
Cool Feature

• Combining SRU response data and echoed data with
  javascript and stylesheets allows for thin,
  browser based, clients
• http//alcme.oclc.org/MXG/search/ORPubs?version1.
  1query"levan"startRecord1maximumRecords10

45
Functional Matrix
Key ?Full Support ?Limited Support