Title: OAI Protocol for Metadata Harvesting
1OAI Protocol for Metadata Harvesting Its
Usefulness to STM Publishers
- Timothy W. Cole (t-cole3_at_uiuc.edu)Mathematics
Librarian Professor of Library Administration - University of Illinois at Urbana-Champaign
- 2005 Allen Press Emerging Trends SeminarNational
Press Club, Washington, D.C. - 13 April 2005
2OAI Protocol for Metadata Harvesting
- Harvesting approachto interoperabilityat
metadata level - Divides world intoMetadata Providers Service
Providers - Builds on HTTP,XML, community metadata
standards - http//www.openarchives.org/
3Metadata Harvesting Model
Metadata Content Repositories
End-Users
OAI Service Provider
Retrieval
Content
Metadata(e.g. XML)
OAIProvider
Search
OAI Harvester
AggregatedMetadata
OAIProvider
Metadata(e.g. SQL)
Content
4Metadata Harvesting Model (cont.)
- OAI Service Provider (harvester) is middleman
between content provider and end-user for
selected metadata-based transactions e.g., - Resource discovery
- Value-added link mediation
- Transactions involving full content still
conducted directly between end-users and content
provider e.g., - Delivery of complete article in desired format
- OAI-PMH is not synonymous with Open Access
5How OAI-PMH Works
- OAI VERBS
- Identify
- ListMetadataFormats
- ListSets
- ListIdentifiers
- ListRecords
- GetRecord
6Protocol Details
- OAI Transaction OAI request (HTTP)
corresponding OAI response (XML) - Transactions initiated by harvester
- Optional flow control mechanisms to manage
provider load - OAI Item Identifiers persistent unique
- Item (Metadata) Date Stamps support selective
harvesting - OAI supports multiple metadata formats
- Distinguishes between an ITEM (complete metadata)
a RECORD (disseminated item of metadata in
given format)
7Reliance on HTTP XML
- OAI-PMH is a REpresentational State Transfer
(REST) protocol (unlike RPC, SOAP) - OAI requests and responses are sent via the HTTP
protocol - OAI requests encoded as HTTP GET or POST
operations - OAI responses are valid XML documents
- Consistency and data quality is ensured by
using XML Schema Definitions (XSD) for all
responses - XML Namespaces used to identify which parts of
response are metadata and which parts support the
Protocol
8Illustration of an OAI Transaction
- Request http//an.oai.org/OAI-script?verbGetReco
rdidentifieroaian.oai.org123metadataPrefixo
ai_dc - lt?xml version"1.0" encoding"UTF-8" ?gt
- ltOAI-PMH xmlns xmlnsxsi xsischemaLocation
gt - ltresponseDategt2004-05-01T192030Zlt/responseDate
gt - ltrequest verb"GetRecord" identifier"oaian.o
ai.org123 metadataPrefix"oai_dc"gt http//an.oa
i.org/OAI-scriptlt/requestgt - ltGetRecordgt
- ltrecordgt
- ...
- lt/recordgt
- lt/GetRecordgt
- lt/OAI-PMHgt
9Illustration of an OAI Record
- ltrecordgt
- ltheadergt
- ltidentifiergtoaian.oai.org123lt/identifiergt
- ltdatestampgt2002-02-28lt/datestampgt
- ltsetSpecgtcslt/setSpecgt
- lt/headergt
- ltmetadatagt
- ltoai_dcdc xmlns xmlnsxsi
xsischemaLocationgt - ltdctitlegtUsing Structural Metadatalt/dctitle
gt - lt/oai_dcdcgt
- lt/metadatagt
- ltaboutgt
- ltprovenance xmlns xmlnsxsi
xsischemaLocationgt - .
- lt/provenancegt
- lt/aboutgt
- lt/recordgt
10What it takes to implement OAI
- Dynamic Web server functionality (e.g., CGI)
- Capacity to respond with XML
- Descriptive metadata in a standard format
- OAI persistent identifiers date stamps may
require changes to metadata creation workflow - Open source implementations available (starting
points) - OAI-PMH included in turnkey publishing solutions
- Public Knowledge Project (UBC)
- Open Repository (BioMed Central), ...
- Eprints.org, DSpace, Fedora, ARNO, CDSware, ...
11Provider Performance Issues
- Database design biggest impact on performance
- e.g., load to dynamically map to DC, other
formats - Webserver performance load can be kept quite low
- Use resumptionTokens, other flow control
mechanisms to improve performance - Fetch only records needed to satisfy current
request - resumptionTokens should retain state information
for best performance and for idempotency - Scale example OCLC repository with 4 million
records
12OAI Implementation Guidelines for Repositories
- Tools Required
- Basic program strategies (incl. object-oriented
approaches) - Guidance for use of
- optional container elements
- Metadata generation / mapping, data cleaning
- Use of OAI Sets
- resumptionToken, flow control, load-balancing
- Denial-of-service prevention
- Error handling
- Strategies for deleted metadata records
- http//www.openarchives.org/OAI/2.0/guidelines-rep
ository.htm
13Why OAI?
- OAI is not synonymous with open access -- content
provider maintains access control over full
content - Implement once, provide metadata to multiple
services - Less performance impact than robotic Web
harvesting - Simpler than z39.50
- Puts your metadata in additional portals
- But, less control over
- How your metadata is presented to end-user
- What your metadata is put next to by service
providers - How valuable a commodity is your metadata?
14Whos Using OAI to Expose Metadata
- OAI Data Provider Registry (http//oai.grainger.ui
uc.edu/registry) - As of 1 March 2005 607 active OAI metadata
provider repositories - Range in size from millions of items, to less
than 100 items - More than half are institutional repositories or
eprint archives - Handful of publisher / publisher-aggregators,
e.g. - PubMed Central BioOne BioMed Central (partial)
Project Euclid Africa Journals Online Institute
of Physics (user id password) American
Physical Society (restricted access) ... - Individual journals, e.g.
- J. of STEM Education Electronic J. of
Probability J. of Cognitive Affective Learning
Canadian J. of Communication ...
15Whos Harvesting Metadata Using OAI-PMH
- Portals encouraging Open Access, e.g.
- OAIster Public Knowledge Project Citebase
Cyclades ... - NSDL (STEM Education) NCSTRL (computer
science)SAIL (physical science e-prints) ... - Local harvesting projects
- As way to share data internally
- As a collation service to their users e.g.,
Grainger Search ServiceOAI harvesting supported
by some Library meta-search utilities - Web search engines that use OAI as one input
stream - Yahoo! ingests from OAIster Google looking to
harvest DSpace sites Scirus includes OAI
metadata ... - mod_OAI (Apache Web servers) as an alternative to
Web robotic harvesting?
16Indirect Benefits from OAI-PMH
- From Bibusages study (French National Library)
- Digital Libraries are used in conjunction with
Web search engines, generalist portals,
commercial sites - Mix of intensive casual users
- DL users seeking answer for specific information
need most time spent discovering, viewing,
downloading documents - Digital Libraries are now attracting a new
type of public, bringing about new, unique and
original ways for reading and understanding
texts.Houssem Assadi, et al. Users Uses of
Online Digital Libraries in France, ECDL 2003
17Evolution of Scholarly Communication
- Ubiquitous nature of electronic pre-prints
post-prints - Extensive linking to supporting content on the
Web - Mixing of author-paid publication with
traditional subscription based business models
(e.g., AIP, Springer trials) - Citation frequency up for articles also available
in arXiv - Demographic and citation trends in Astrophysical
Journal papers and preprints / Greg J. Schwarz
and Robert C. Kennicutt, Jr. BAAS 361654-1663,
2004 also http//arxiv.org/abs/astro-ph/0411275
- Some publishers encouraging self-archiving of
pre-prints - IMS APS AIP ... see http//www.sherpa.ac.uk/ro
meo.php - OAI-PMH underpins these kinds of self-archiving
services
18A Librarians Perspective
- The information landscape can be seen as a
contour map in which there are mountains,
hillocks, valleys, plains and plateaus. A
specialized collection of particular importance
is like a sharp peak. Upon a plateau there might
be undulations representing strengths and
weaknesses. The landscape is, however,
multidimensional. Where one scholar may see a
peak another may see a trough. The task is to
devise mapping conventions which enable scholars
to read the map of the landscape fruitfully, at
the appropriate level of generality or
specificity. - Michael Heaney (2000), An Analytical Model of
Collections and their Catalogues.