MARC21 and Z39'50

1
MARC21 and Z39.50

• Jo Rademakers
• K.U.Leuven / LIBIS

2
MARC21

• Machine-Readable Cataloging record
• Standards for the representation and
  communication of bibliographic and related
  information in machine-readable form
• 3 elements
• Structure (based on Z39.2 - ISO 2709)
• Content designation (codes and conventions)
• Content eg. Based on AACR, MESH, LCSH

3
MARC21

• For Bibliographic data
• For Holdings data
• For Authority data
• For Classification data
• For Community information

4
(No Transcript)
5
(No Transcript)
6
Definitions

• Fields are marked with tags
• Further refinement with indicators
• Subfields are marked by subfield codes and
  delimiters
• Leader 24 characters (process info)
• Directory what tags and position of them
• 008 field Fixed-length data-element

7
What is Z39.50

• International standard for information search and
  retrieval
• Established by NISO (National Information
  Standards Organisation)
• Accepted by ISO (International Standards
  Organisation) as ISO 23950
• Developed by the Z39.50 Implementors Group (ZIG)
• Maintained at Library of Congress (Ray Denenberg)

8
What is Z39.50

• ...
• Allows searching in databases over a network
• Independent of the search software that is used
• Independent of database system where data are
  stored
• Primarily (but not uniquely) used in
  bibliographic contexts

9
History

• Linked Systems Project
• LC, OCLC, RLG, WLN
• Z39.50 - 1988
• Used in the WAIS protocol
• Z39.50 - 1992 (version 2)
• Z39.50 - 1995 (version 3)
• Superseded ISO 10162/10163
• ISO 23950 - 1998
• ZING
• Z39.50-2002 Revision

10
Purpose

• Interoperability for search and retrieval of
  information with client/server systems
• Interoperability between vendors
• Different databases and user interfaces
• Interoperability between different organisations
• Eg. using different library formats
• Interoperability between groups of users
• Eg. Public libraries/Academic libraries
• Eg. libraries in different countries
• Interoperability between communities
• Eg. libraries, publishers, archives, museums

11
Some concepts

• Origin / Target
• Client / Server
• Session
• Connection is established and maintained between
  Origin and Target (statefullness)
• Exchange of messages (Protocol Data Units)
• Services and facilities
• sets of PDU exchanges
• e.g. Init, Search, Retrieve
• Some facilities consist of a logical set of
  services

12
Some concepts

• Abstract database
• Standardised access points
• Attribute sets
• Standardised queries
• Standardised views
• Schemas
• Possibilities to select record syntax
• Possibilities to select part of record
• Searches not tied to record content
• Contains information to map
• the Z39.50 searches to database specific queries
• the retrieved results to Z39.50 exchange formats

13
Some concepts

• The abstract database is implemented in the
  target as a front-end to the real database

Database
14
Target

• Special development
• Toolkits
• Requires advanced configuration
• How shall the real database be represented as an
  abstract one?

15
Origin

• Origin is normally part of a graphical client
• Hides complexity from the user
• Often needs extensive configuring
• Can sometimes access several targets
  simultaneously
• There are clients with a raw Origin interface

16
Gateway

• A program that has 2 interfaces
• One where it acts as Origin to a Z39.50 Target
• One where it handles communication with a client
  application
• Client protocol may be HTML, Telnet, Z39.50, etc.

17
Web gateway
Web browser
Z39.50 server
18
Multi-target gateway
Z39.50 client
Z39.50 server
Z39.50 server
Z39.50 server
19
Advanced Gateway

• A more advanced Gateway can connect to several
  Z39.50 Targets
• Parallel search
• Serial search
• Merging of results
• Even more advanced Gateways handle several
  different protocols on both interfaces
• SQL, LDAP, HTML, DNS...

20
Advanced gateway
Z39.50 client
Z39.50 server
Web reader
SQL database
SQL client
Proprietary system
LDAP server
21
Facilities and services

• Some optional Services and Facilities besides
  Init, Search, Retrieve
• Access Control, Accounting/Resource Control,
  Browse/Scan, Explain, Result-Set-Delete, Sort,
  Termination
• Extended Services
• Access to services outside the protocol
• e.g. Persistent result sets, persistent query,
  periodic query schedule, Item order, Database
  update, Export specification, Export invocation

22
Diagram
23
Difficulties

• Different databases have different capabilities
• Truncation, search indices, implementation of
  features
• Different databases have different sets of
  information
• US MARC, UNIMARC, MAB
• Embedded holdings or separate holdings
• Different servers may provide different services

24
Profiles

• Profile agreement on how to use the standard
• Which access points?
• Which attributes?
• Which record formats?
• Which services and supplementary services?
• What options?
• Which data for certain fields

25
Application profiles

• WAIS (1994)
• Wide Area Information Servers
• GILS
• Government Information Locator Service (US)
• Profie for document S/R in public administration
• CIP
• Catalogue Interoperability Protocol
• The Committee on Earth Observation Satellites
  (CEOS)
• Search profile for geo-spatial data

26
Application profiles

• GEO
• US government profile for geo-data
• CIMI
• Consortium for the Computer Interchange of Museum
  Information
• Not only text. Also specifies how to retrieve
  images
• International Registered Profile (IRP)
• STAS
• Scientific and Technical Attribute Set
• Not a real profile

27
Application profiles

• ZSQL
• extend Z39.50 with SQL features query, records
  and diagnostics
• Zthes
• browsing hierarchical structures
• using existing protocol facilities
• Access to digital collections
• semantics for navigating digital collections, to
  locate and retrieve objects of interest.

28
Library profiles

• ATS-1 (obsolete)
• Author, Title, Subject
• ONE
• OPAC Network Europe (1996)
• Minimum requirements for access points and
  element sets
• CENL
• Conference of European National Librarians (1997)
• Expands on the ONE profile

29
Library profiles

• National profiles
• adding functionality to international profiles
• Finnish Z39.50 profile
• Finnish National Library Network
• including Union Catalogue Profile
• Danish Z39.50 profile
• National classifications
• including also ILL-protocol
• Texas profile
• interoperability between Texas libraries
• different vendors

30
Bath profile

• Problems
• specific profiles discipline, national and
  regional sharing basic functions
• vendor/system interpretation
• complicated, expensive and old-fashioned
• heavily focussed to libraries
• difficult interoperability
• regional, national
• vendors/system
• cross-domain
• Based on other profiles
• http//www.nlc-bnc.ca/bath/bath-e.htm

31
Bath profile

• Minimisation of defaults
• Where possible, every attribute is defined in the
  Profile
• 3 Functional Areas
• Basic Bibliographic Search Retrieval
• Bibliographic Holdings Search Retrieval
• CrossDomain Search Retrieval
• 3 Levels of Conformance in each Area.

32
Bath profile

• Basic Bibliographic Search Retrieval
• Level 0
• Author, Title, Subject, Any
• Level 1
• Author, Title, Subject, Standard Identifier, Date
  of Publication, Any
• more exact ATS searches
• Scan

33
Bath profile

• Bibliographic Holdings SearchRetrieval
• Level 0
• Holdings info embedded in the record, etc.
• Level 1
• Access to Locations, Summary Information and
  Holdings Count.

34
Bath profile

• CrossDomain Search Retrieval
• Level 0
• Creator, Title, Subject, Any
• Level 1
• Creator, Title, Subject, Standard Identifier,
  Date of Publication, Any
• more exact ATS searches

35
Bath profile

• Bibliographic Search results
• SUTRS
• UNIMARC or MARC21
• All three at Level 1 (for Targets)
• Bibliographic holdings
• GRS-1 (according to holdings schema)
• Cross-domain search results
• SUTRS
• Dublin Core (in XML)
• Other record syntaxes also allowed

36
Bath profile

• Impact
• re-indexing database
• upgrading target and origin
• additional character sets (international
  retrieval)
• Benefits
• more accurate search results
• easier search and retrieval from multiple
  (international) databases
• no different interfaces to each database anymore
• productivity improvements based on more reliable
  multiple database searches

37
Bath profile

• Implemented by
• National libraries and union catalogue agencies
  Finland, Sweden, Norway, Denmark, United Kingdom,
  Italy, Hungary, Canada
• Texas state-wide library network
• Canada Council of Atlantic University Librarians
• Endorsed by
• IT19 Committee of Standards Australia/New Zealand
• FinZIG (Finnish Z39.50 Implementors Group)
• International Registered Profile (IRP)

38
Library profiles

• Union Catalogue Profile
• Defines requirements for cataloguing activity to
  union catalogue as well as local system through
  Z39.50
• Developed in Australia (NLA)
• Accepted spring 1998
• International Registered Profile (IRP)

39
How does it work?

• Based on facilities
• corresponding on
• can be seen as operations
• 12 facilities
• A request always triggers a response

40
Initialisation facility
Origin
Target
Init request Version, (id/password), option
flags, message sizes, implementation information
Init response Result, version, option flags,
message sizes, implementation information

• Init-service
• establishing a Z-association
• negotiating which services and options to use
• origin proposes and target filters with own
  capabilities and sends back result
• user authentication

41
Search facility
Origin
Target
Search request Search type, query, databases,
result set, limits for small, medium, large
Search response Number of records found, number
of records attached, status information, (records)

• Search service
• Search request database(s) to search, query,
  name of new result set
• Search response number of records identified by
  specified query

42
Example search

• Search request
• Search the music database for records with
  Mozart Symphonies in the title
• Search response
• 27 records found

43
Queries

• Query types
• Type-0 proprietary between 2 parties
• Type-1 RPN (standard)
• Type-2 ISO 8777
• Type-100 Z39.58
• Type-101 Extended RPN (including proximity)
• Type 102 Ranked List query

44
Queries Type 1

• Also known as RPN (Reverse Polish Notation)
• Linking one or more operands with boolean
  operators AND, OR, AND_NOT
• Operand
• result set name
• term and attributes

45
Queries Type 1

• Examples
• A
• records in database for which A is true
• A B C AND OR
• records for which both B and C are true, or A is
  true
• A B AND C OR
• records for which both A and B are true, or C is
  true
• (result set R) AND A
• records in result R for which A is true

46
Queries Type 1

• Term
• what you want to find
• Attributes
• representation of abstract database
• based on attribute sets
• Examples
• BIB-1
• STAS-1
• CIMI-1
• Now new - more generic - attribute architecture

47
Queries BIB-1

• Library based
• 6 Attribute types
• 1 use abstract access point to use
• Subject subdivision 47 No. nat'l biblio.
  48
• No. legal deposit 49 No. govt pub. 50
• No. music publisher 51 Number db 52
• Number local call 53 Code--language 54
• Code--geographic area 55 Code--institution
  56
• Name and title 57 Name geographic
  58
• Place publication 59 CODEN 60
• Microform generation 61 Abstract 62
• Note 63 Author-title 1000
• Record type 1001 Name 1002
• Author 1003 Author-name
  personal 1004
• Author-name corporate 1005 Author-name
  conference 1006
• Identifier--standard 1007 Subject--LC
  children's 1008
• Subject name -- personal 1009 Body of text 1010
• Date/time added to db 1011 Date/time last
  modified 1012
• Authority/format id 1013 Concept-text 1014

48
Queries BIB-1

• 6 Attribute types
• 2 relation between term and data in the access
  point
• equals, less than, greater or equal, relevance...
• 3 Position where in access point is the term
• first in field, any position...
• 4 Structure how is the term to be treated
• phrase, word, year, key, structure...
• 5 Truncation apply truncation on the match
• no, left, right, regular expression...
• 6 Completeness term should match
• incomplete subfield, complete field...

49
Queries BIB-1 example

• (Mark Twain, 11003, 23, 31, 41, 5100,
  61) (Clemence, Samuel, 11003, 23,
  33, 4101, 5100, 62) AND-NOT

50
Retrieval facility
Origin
Target
Present request Number of records, starting
point, result set
Present response Number of returned records,
status, (records)

• Present service
• Present request result set from which to
  retrieve records, range of records, composition
  and syntax
• Present response records

51
Example present

• Present request
• Present the first two records in MARC format,
  brief records
• Present response
• Shapiro, Deane H. Institute for the Advancement
  of Human Behavior (Portola, Calif.)
  Self-control East and West sound recording
  toward a precision nirvana / by Deane
  Shapiro Biomonitoring
  Applications, New York p1978.
• Nichols, John Treadwell, 1940- Nirvana blues.
  Selections. Readings sound recording /
  John Nichols. American Audio Prose Library,
  Columbia, Mo. p1982

52
Example present

• Present request
• Present the second record in MARC format, full
  records
• Present response
• 001 83740050 /R
• 003 DLC
• 005 19881006000000.0
• 007 ssulmnjlcnnnu
• 008 840605s1982 mounnnn fb1 eng
• 050 00 a RYA 244
• 100 10 a Nichols, John Treadwell, d 1940-
• 240 10 a Magic journey. k Selections
• 245 10 a Readings h sound recording / c John
  Nichols.
• 260 0 a Columbia, Mo. b American Audio
  Prose Library, c p1982
• 300 a 1 sound cassette (61 min.) b analog,
  mono.
• 306 a 010049
• 511 0 a Presented by Kay Bonetti.
• 518 a Recorded in Albuquerque, N.M., in Apr.
  1982.
• 520 a Following introductory comments about
  the author's life and works, John Nichols reads
  excerpts from two of his novels,
  The magic journey and The nirvana blues.
• 700 10 a Bonetti, Kay.

53
Retrieval model
Database record

• Schema
• common understanding of the information in the
  database
• Element Spec
• set of elements to be retrieved
• Record Syntax
• specification for the packaging of a Retrieval
  Record

Schema
Abstract Database Record 1
Element Spec.
Abstract Database Record 2
Record Syntax
Retrieval Record
54
Retrieval Record Syntax

• MARC record formats
• UniMarc
• USMarc
• UKMarc
• CanMarc
• DanMarc
• OPAC record
• MARC record and 1 or more holdings
• and/or holdings circulation records

55
Retrieval Record Syntax

• Generic formats
• SUTRS (Simple Unstructured Text Record Syntax)
• GRS-1 (Generic Record Syntax)
• Other formats
• HTML
• XML
• ...

56
Retrieval Element Spec

• Element
• retrieval unit
• Element Set Name (ESN)
• name for a collection of one or more elements
• standard Z39.50
• F (Full) - all available elements
• B (Brief) - target defined subset of elements
• primitive
• AU - author only
• AUTI - author and title
• list-style
• AU,TI,AB,SU - author, title, abstract and subjects

57
Retrieval Element Spec

• Composition specification (ComSpec)
• allows more complicated records
• selecting elements based on
• Elements Set Name
• External Structure Espec-1
• complex data structures
• element meta-data
• forms of elements
• schema based on tagsets
• TagSet-M (Meta-data elements), TagSet-G (Generic
  elements), ...
• record syntax

58
Retrieval facility

• Segment service
• split large present responses in segments
• two levels
• level 1 only whole records in segment
• level 2 records can be split

59
Browse facility
Origin
Target
Scan request database, term list, starting
point, number of terms, (step size)
Scan response status, number of elements,
(elements)

• Scan service
• Scan request index to obtain records from, where
  to start (term), number of terms
• Scan response terms

60
Example scan

• Scan request
• Scan the title index of the Music database, begin
  scan with mozart chamber music, as the fifth
  term and return nine records
• Scan response
• mozart beethoven
• mozart prague
• mozart salieri
• mozart the archbooby
• mozart chamber music
• mozart in new york
• mozart speaks
• mozart symphonies
• mozart und haydn in london

61
Sort facility
Origin
Target
Sort request result set to sort, sorted result
set, sort directives
Sort response status

• Sort service
• specified result set in sorted according to
  specified criteria
• new result set created
• or result set overwritten

62
Result-set-delete facility
Origin
Target
Delete request list of result sets to delete
Delete response status

• Delete service
• delete previously created result sets

63
Access control facility
Origin
Target
ANY request
Access control request Security-challenge-respons
e
Access control response Security-challenge
ANY response

• Access control service
• re-verifying the user by asking for a user-id /
  password
• giving higher authorisation by allowing new
  identification

64
Accounting/Resource control facility

• Resource control service
• Trigger resource control service
• Resource report service
• control and report on resource usage
• complex
• interesting for fee based operations

65
Duplicate detection facility

• Duplicate detection service
• NOT IN STANDARD (amendment)
• analysing one or more result sets for potential
  duplicates
• new result set based on origin specified criteria
  for detecting, retaining, grouping, and ordering
  the records including duplicates

66
Extended Service facility

• Extended services service
• Persistent Result Set Extended Service
• Persistent Query Extended Service
• Periodic Query Schedule Extended Service
• Item Order Extended Service
• Database Update Extended Service
• Export Specification Extended Service
• Based on
• Task packages in Extended Services database
• Search and Present based on Extended Services
  Record Syntax and attribute set

67
Explain facility

• Explain service
• Information about the Z39.50 target
• databases
• access points
• query languages
• element sets
• Based on
• Explain database
• Search and Present based on Explain Record Syntax
  and attribute set

68
Termination facility

• Close service
• terminating the Z-association

69
Some other definitions

• TCP/IP using port 210
• BER Basic encoding rules
• way to code data for transmission
• not human readable
• ASN.1 Abstract Syntax Notation 1
• implementation independent way of describing data
• APDU Application Protocol Data Unit
• package containing the request or response

70
Example
InitializeRequest SEQUENCE referenceId
ReferenceId OPTIONAL,
protocolVersion ProtocolVersion,
options Options,
preferredMessageSize 5 IMPLICIT
INTEGER, exceptionalRecordSize 6
IMPLICIT INTEGER, idAuthentication
7 ANY OPTIONAL, -- see note below
implementationId 110 IMPLICIT
InternationalString OPTIONAL,
implementationName 111 IMPLICIT
InternationalString OPTIONAL,
implementationVersion 112 IMPLICIT
InternationalString OPTIONAL,
userInformationField 11 EXTERNAL
OPTIONAL, otherInfo
OtherInformation OPTIONAL
--Note -- For idAuthentication, the type ANY is
retained -- for compatibility with earlier
versions. -- For interoperability, the following
is recommended -- IdAuthentication 7
CHOICE -- open VisibleString, --
idPass SEQUENCE -- groupId
0 IMPLICIT InternationalString OPTIONAL, --
userId 1 IMPLICIT
InternationalString OPTIONAL, --
password 2 IMPLICIT InternationalString
OPTIONAL , -- anonymous
NULL, -- other EXTERNAL
-- May use access control formats for
'other'. See Appendix 7 ACC.
71
Out there...

• Library systems
• All large systems support Z39.50
• Including dedicated client or a web gateway
• Some smaller systems use (or rely fully on)
  Z39.50
• Many systems are still version 2, though
  sometimes with features from version 3
• Especially American systems
• Aleph, VTLS, SIRSI, EpixTech, Geac, ADLiB,
  DOBIS/LIBIS, OLIB, Voyager, Horizon, AMICUS

72
Out there...

• Organisations
• National libraries
• Library of Congress
• British Library
• National Library of Canada
• National Library of Australia
• Bibliothèque Nationale de France
• SBN
• LIBRIS
• Universities
• just too much

73
Out there...

• Commercial databases
• Silver Platter
• EBSCO
• Dialog
• OVID
• Swets (work in progress)
• Clients
• Endnote
• Reference Manager

74
Out there...

• Gateways
• OCLC
• LibriVision
• Decomate
• Toolkits
• IndexData (YAZ)
• Finsiel (ZetaPerl)
• OCLC
• Crossnet
• ...

75
Conclusion

• Z39.50 is a complex standard
• Allowing interoperability at several levels
• However, interoperability is not for free. It
  takes knowledge and a lot of hard work to make
  systems truly interoperable

76
ZING Z39.50-International Next Generation

• Goal make Z39.50 more attractive to information
  providers, developers, vendors, and users by
  lowering the barriers to implementation while
  preserving the existing intellectual
  contributions
• Different initiatives
• SRW/SRU
• ZOOM
• ez3950
• Explain - -
• ...

77
ZING SRW

• SRW Search/Retrieve Web Service
• Building on Z39.50 along with web technologies
  XML, SOAP/RPC, HTTP
• focussing on getting information to the user
• Concepts retained
• result sets
• abstract database
• explain
• diagnostics
• Different
• Search and Present are combined
• stateless
• No distinction between server and database
• Record syntax XML
• String Query CQL
• XML encoded

78
More information

• One address
• Z39.50 maintenance agency
• http//lcweb.loc.gov/z3950/agency
• standards text
• profiles
• implementations
• amendments, defects, clarifications, ZIG
  commentaries
• upcoming meetings and minutes from previuous
• ZING
• Aiming at quality and coverage combined blending
  physical and virtual union catalogues.
• Gatenby, Janifer - Online Information Review
  Volume 26 No. 5 2002

79
http//www.loc.gov/z3950/agency/