Internet Archives and the umich.edu Domain

1
Internet Archives and the umich.edu Domain

• Part 1 Technical Challenges

2
Project Goals

• Demonstrate the feasibility of comprehensively
  archiving an entire website.
• Document the process so that others may learn
  from our experience.

3
Three Step Program

• Gathering Information
• Processing Appraisal
• Providing Access

4
Gathering Information

• 2004 crawls Performed in San Francisco by the
  Internet ArchiveNever underestimate the
  bandwidth of a station wagon full of tape.
• 2005 crawls Performed locally using Heritrix,
  the Internet Archives open source web crawler

5
Process Appraisal

• Information stored in ARC Files was indexed and
  parsed.
• Large files and duplicates were eliminated from
  the online version.

6
Providing Access

• We got SI Computingsattention by filling
  upSELMA with ARC files.
• PHP MySQL were used to serve up the content.
• http//www.si.umich.edu/mirror/

7
Challenges Conclusions

• Server space is an issue, but the cost is coming
  down. In time, we will be able to provide access
  to all the content weve gathered.

8
Challenges Conclusions

• Accessibility Archivability
• The less accessible a website is, because of
  JavaScript, Flash, etc., the more likely it is
  to break the archive.

9
Challenges Conclusions

• Asynchronous JavaScript XML,Ajax is the
  future of web design.
• Ajax breaks our model entirely!
• Ajax poses an exciting new challenge!
• http//www.adaptivepath.com/publications/essays/ar
  chives/000385.php

10
Questions!