InterLibRelated Activities at SDSCDICE

1
InterLib(-Related) Activities at SDSC/DICE

• Bertram Ludaescher
• ludaesch_at_sdsc.edu

• IBM HPSS (Storage/Archival, e.g. ADL)
• SDSC SRB/(E)MCAT (Data Handling/Information
  Discovery)
• AMICO Image Collection (CDL Testbed)
  
  
• Excelon as XML Data Server
• MIX Mediation of Information using XML (with
  DB-Lab UCSD)

2
HPSS, SRB, MCAT

• HPSS Storage/Archival of large datasets
• (UCB, UCSB, Stanford)
• SRB/(E)MCAT Data Handling/Information Discovery
• transparent access to remote storage
• replication
• containers for large number of small items
• caching
• authorization
• proxy operation support (filtering, data
  subsetting)
• usage of security infrastructure (GSI)

3
SRB Interface
Application
Application
MCAT Core
SRB Master
SRB Agent
SRB Server
MCAT
Dublin Core
Eco Core
SRB Server
SRB Server
4
Managing Metadata EMCAT

• Extensible Meta Data Catalog - EMCAT
• Exploits dependencies relationships (mn, tc,
  ltgt, )
• T-Language - Markup, Filter Presentation
• Meta Data Repository (Object-, System-,
  Collection-level)
• Based on Kernel Meta Meta Data
• Extensible
• Uniform Access and Federation interface
• Metadata exchange Interface Protocol
• MAPS- Meta data Attribute Presentation Structure
• query, update and result structures
• Close to Z39.50

5
SRB/MCAT Future

• Performance Improvements and Consolidation
• Delayed Action Manager - mirror, cronjobs
• Support for Methods
• Handling Very Large Data sets - partitions
• More Drivers - Sybase, NTFS, LDAP
• Extensible MCAT
• Language Support - Perl, Fortran
• http//www.npaci.edu/DICE/SRB

6
The AMICO Digital Library Projecthttp//www.amic
o.orghttp//www.npaci.edu/DICE/AMICOArt Museum
Image Consortium

• Richard Marciano et. al.
• 55,146 objects 750 MB
• 53,763 thumbnail images 319 MB
• 57,609 full tiff images 180 GB

7
AMICO Consortium of 26 (now 31) museums
AGO_ Art Gallery of Ontario AIC_ Art
Institute of Chicago AKAG Albright-Knox Art
Gallery, Buffalo, NY ASIA Asia Society
BMFA Boston Museum of Fine Arts CCP_
Center for Creative Photography, U. Arizona
CMA_ The Cleveland Museum of Art DMCC
Davis Museum and Cultural Center, Wellesley
College, MA FASF Fine Arts Museums of San
Francisco GEH_ George Eastman House,
Rochester, NY JPGM J. Paul Getty Museum, Los
Angeles, CA LACM Los Angeles County Museum
of Art LOC_ Library of Congress MACM
Musée d'art contemporain de Montréal MBAM
Musée des beaux-arts de Montréal MCAS
Museum of Contemporary Art, San Diego MIA_
The Minneapolis Institute of Arts MMA_ The
Metropolitan Museum of Art NGC_ National
Gallery of Canada, Ottawa/Ontario NMAA
National Museum of American Art, Smithsonian
Institution PMA_ Philadelphia Museum of
Art SFMO San Francisco Museum of Modern Art
SJMA San Jose Museum of Art TFC_ The
Frick Collection, NY WAC_ Walker Art Center,
Minneapolis, MN WMAA Whitney Museum of
American Art, NY
8
Raw Metadata Structure
- catdata 8 files 16,604
year1.d990429 14,430
year1.d990512 22,938
year1.d990520 54,303
year1.d990627 15
year1.d990708 54,298
year1.d990731 93
year1.d990806 657
year1.d990813
- tiffmetadata 23 files 2963
AGO_.tiffmetadata.txt 1016 AIC_.tiffmetadata.t
xt 894 AKAG.tiffmetadata.txt 187
ASIA.tiffmetadata.txt 7591 BMFA.tiffmetadata.t
xt 401 CCP_.tiffmetadata.txt 1455
CMA_.tiffmetadata.txt 56 DCMC.tiffmetadata.t
xt 470 DMCC.tiffmetadata.txt 10141
FASF.tiffmetadata.txt 2137 GEH_.tiffmetadata.t
xt 1459 JPGM.tiffmetadata.txt 1013
LACM.tiffmetadata.txt 20654 LOC_.tiffmetadata.t
xt 86 MACM.tiffmetadata.txt 50
MBAM.tiffmetadata.txt 31 MCAS.tiffmetadata.t
xt 1440 MIA_.tiffmetadata.txt 550
MMA_.tiffmetadata.txt 1507 NGC_.tiffmetadata.t
xt 1416 NMAA.tiffmetadata.txt 154
PMA_.tiffmetadata.txt 158 SFMO.tiffmetadata.t
xt 86 SJMA.tiffmetadata.txt 68
Such.tiffmetadata.txt 396 WAC_.tiffmetadata.t
xt 37069 replacements.txt 57499
replacements2.txt
- thumbmeta 52,689 files
AGO_.1016.25_thum.met AGO_.1016.32_thum.met
AGO_.1016.39_thum.met ...
WAC_.994C_thum.met WAC_.996C_thum.met
WAC_.998C_thum.met WAC_.99C_thum.met
WMAA.1557_56_thum.met WMAA.31_426_thum.met
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
AMICO Metadata Conversion Steps
Raw Metadata files - catdata (8 files), -
tiffmetada (23 files), - thumbmeta (52,689
files)
Consolidated Metadata files - 1 catdata - 1
tiffmetadata - 1 thumbmeta
Tape Read
Merge
Convert to XML
Multiple XML files per museum
Split-by- museums
1 XML file per museum
Split-by- file size
1 XML file per museum
3 XML files - 1 catdata - 1 tiffmetadata -
1 thumbmeta
eXcelon Data Server
Multiple museum XML files per machine
eXcelon Data Server
eXcelon Data Server
eXcelon DumpLoad Utility
Split-by- machines
13
Alternative System Architectures
AMICO metadata server
eXcelon
Oracle 8i DB2
14
Current catalog metadata count (per museum)
15
Average tiff size in MB (per museum)
16
Excelon Metadata Layout
XMLStore
Museum directories
Museum1
Museum2
Museum-n
File1.xml
File2.xml
17
MIX Mediation of Information Using XML ... for
the AMICO CDL Prototype
BBQ Interface (slide carousel interface)
XMAS query
XML doc
MIXm engine
Wrapper
AMICO XML Database
AMICO XML Database
MARC Database
XMAS XML Matching and Structuring query language
18
SDSC/DICE Discussion Topics

• ADL caching of HPSS data
• ADEPT access to ADL for CDL testbed SRB?
• Union Catalog
• AMICO DTD ltXMASgt MARC
• SDLIP access to SRB/MCAT and MIX
• Use of GINF (Stanford)
• ...