Building Collections Using Greenstone

1
Building Collections Using Greenstone

• Tod A. Olson lttod_at_uchicago.edugt
• Sr. Programmer/Analyst
• Digital Library Development Center
• University of Chicago Library
• http//www.lib.uchicago.edu/dldc/talks/2003/dlf-g
  reenstone/

2
Greenstone

• New Zealand Digital Library Projectat the
  University of Waikato
• In cooperation with UNESCO, Human Info NGO
• International, every continent
• Examples
• Academic
• Digitization projects
• Classes on digital libraries
• Non-academic
• UNESCO humanitarian documentation

3
Greenstone features

• Works with existing documents
• Imports several formats
• Searching full text and metadata
• Dublin Core, custom metadata
• Browse
• Structured documents
• Indexing, access
• Extensible customizable
• OpenSource software (GPL)

4
User Interface overview

• Finding documents
• Search full text and metadata indexes
• Classifiers browse lists for navigating
  collections
• Navigating documents
• Navigate hierarchical documents by logical
  structure
• Simple page turning (not shown)
• Single page for simple documents (not shown)

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Greenstone Architecture
Receptionist
Receptionist
Protocol
Collection Server
Collection Server
Collection
Collection
Collection
DB Indexes
DB Indexes
DB Indexes
Import
Import
Import
Redrawn from Witten Bainbridge, How to Build a
Digital Library, p. 356
17
Greenstone Architecture

• Receptionist
• Provides user interface
• Accept user input
• Send to appropriate collection server
• Accept results
• Dynamic page generation

• Collection Server
• Handle collection content
• Search and filter information
• Return results
• multiple collections

18
Building Collections
HTML
DB Indexes
Import
Build
GSAF
PDF
???
19
Building collections

• Create a collection framework
• or work with an old collection
• Select documents
• Import documents
• Converts to internal XML format (GSAF)
• Build collection
• creates search indexes and browse listings

20
GSAF internal XML format
ltSectiongt ltDescriptiongt ltMetadata nameTitle
valuegt ltContentgt Text, images, links,
etc. ltSectiongt ltDescriptiongt ltMetadata
nameTitle gt ltContentgt ltSectiongt ltSectio
ngt ltSectiongt
21
GSAF internal XML format

• Section
• Description
• Metadata fields
• Content
• Text,internal markup, images
• Section
• No limit in number or depth
• Hierarchical documents
• Sections nest, tree structure

22
Config file collect.cfg

• Collection-specific configuration file,
  collect.cfg, specifies  
• file types to import
• Indexes and browse lists
• Document or section level
• paragraph (text index only)
• display of results and browse listings
• document displays

23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Chopin Early Editions

• Over 400 early edition Chopin scores
• 1830s to 1880s
• Target audience music scholars musicians.
• On web, page-turnable JPEG images.
• Online in March 2003
• Currently 372 scores in online collection
• Usage
• Nearly100 hits per day, gt 30 of use is
  international.

29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Build overview
GreenstoneArchiveFormat
GreenstoneDig. Library Software
XSLT
METS MODS
Human processing
XML-based automated processing
37
Catalog records

• Detailed MARC/AACR2 record for each score
• Luxury few print music collections have this
  much metadata

38
Scanned score images

• Scanned by Preservation staff
• Archival TIFF images
• 400dpi, 24-bit color, uncompressed
• JPEG derivatives for web-based delivery

39
Structural and other metadata
"chopin","108","001","","1","""chopin","108","002
","","1","" "chopin","108","003","1","1","Nocturne
, no.15" "chopin","108","004","2","1","" "chopin",
"108","005","3","1",""
40
Structural metadata

• Identify each image
• document (score) no. sequencial image no.
• Image content
• page no. as printed, milestones
• Staff use familiar RDB product
• Export data in CSV format
• Technical metadata recorded, not yet used

41
Build overview
GreenstoneArchiveFormat
GreenstoneDig. Library Software
XSLT
METS MODS
Human processing
XML-based automated processing
42
METS MODS
dmdSec MODS fileSec URL page1.jpg URL
page2.jpg structMap div DMDID1 div
FILEID1 div FILEID2
Catalog record (MARC)
Scanned images (JPEG)
Structural metadata
43
METS MODS

• Program uses structural metadata to
• Generate structMap
• Generate image URLs for fileSec
• Images stored by naming convention
• Structural md carries catalog record no.
• Extract MARC from catalog
• crosswalk to MODS
• Embed in dmdSec

44
GSAF

• XML format for internal storage
• Hierarchical document structure
• Nested sections e.g. part 1, chapt. 2
• METS to GSAF via XSLT
• Natural mapping from METS to GSAF
• Map structural hierarchy
• Follow links
• Descriptive metadata
• File content

45
METS to GSAF
Section Description Metadata Title,
Content Title, Section Content Page
1 page1.jpg Section Content Page 2
page2.jpg
dmdSec MODS Title, fileSec page1.jpg page2.j
pg structMap div Score div Page 1 div
Page 2
46
METS to GSAF
Section Description Metadata Title,
Content Title, Section Content Page
1 page1.jpg Section Content Page 2
page2.jpg
dmdSec MODS Title, fileSec page1.jpg page2.j
pg structMap div Score div Page 1 div
Page 2
47
METS to GSAF
Section Description Metadata Title,
Content Title, Section Content Page
1 page1.jpg Section Content Page 2
page2.jpg
dmdSec MODS Title, fileSec page1.jpg page2.j
pg structMap div Score div Page 1 div
Page 2
48
METS to GSAF

• Walk structural metadata to create the tree of
  ltSectiongt elements
• Descriptive metadata
• ltDescriptiongt
• Crosswalk to desired metadata names
• ltContentgt
• Format metadata desired for display
• File data
• ltContentgt
• Inline text, link to images, etc.

49
Customizing Chopin collection

• Focus on navigation
• Metadata for custom access
• E.g. genre, dedicatee not in MARC/AACR2
• Can support with METS, MODS, Greenstone
• Custom document navigation
• Separate description from scores
• Custom page navigation
• Improves usability
• Branding in next phase

50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
Comments on Chopin Early Editions

• Data created by staff using familiar tools
• Structural md created in desktop application
• Catalog records a luxury
• Catalog is DB of record
• Project IDs in 909
• POIs point into Greenstone
• METS/MODS assembled by program
• Expect to repurpose METS for other applications
• Customization navigation, not branding
• Faster to bring up collection, get user reaction

59
Greenstone benefits for Chopin

• Robust, mature system
• Recovered time in project
• Fast to bring up
• UI out of the box
• Dynamic page generation
• Incremental customization
• XML compliant
• Natural mapping from METS to GSAF

60
Future work Chopin

• Add DjVu image format
• Repurpose METS for other applications
• OAI
• Standardize new digitization production flow
• Project was first for METS, MODS, GS, 6 depts.
• Standardize collection of structural metadata
• Plug in descriptive metadata as appropriate
• Store archival descriptive metadata in METS
  object
• Repurpose via XSLT for delivery

61
Other custom UI examples

• Lehigh Digital Bridges
• Extensive changes to look
• Washington Research Libraries Consortium (WRLC)
• Custom page banner
• Popup page turner in Perl
• GS as component of DL suite

62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
Ongoing work Greenstone

• Greenstone Librarian Interface (GLI)
• Greenstone 3

69
Greenstone Librarian Interface (GLI)

• Collection management
• Informed by work at GS sites
• Assist collection designer
• Support all phases of collection build process
• Do not specify workflow

• Java-based GUI tool
• Formerly called the Gatherer
• 2 yrs in development
• In beta outside of lab
• Bangalore, other sites
• in current distribution

70
GLI functions

• Establish new collection (or work on old)
• Select files to include in collection
• Enrich files with metadata
• Select indexes, classifiers
• Build collection
• Customize appearance
• Preview collection

71
Greenstone 3

• GS2 mature, 5 yrs., wide deployment
• Constraints support legacy systems
• Other technologies have matured Java, XML
• GS3 rewrite in Java, XML, XSLT
• Distributed architecture, SOAP
• METS as internal format
• Group assembled for Greenstone METS profile(s)
• OAI support planned
• 1 year in dev alpha testing in lab

72
Conclusion

• Positive experiences
• Good direction for development
• Strong user community
• Proven in real digital library projects

73
Links Further Information

• Chopin Early Editions http//chopin.lib.uchicago.
  edu/
• Greenstone http//www.greenstone.org/
• Downloads, documentation, examples
• New Zealand Digital Library Project
  http//www.nzdl.org/
• UNESCO related collections, many demos
• Witten Bainbridge. How to Build a Digital
  Library. Morgan Kaufman, 2003.