Thoughts before "The Future of the Past of the Web"

Tomorrow I'm going to be in London for a joint JISC/DPC event on web archiving, "The Future of the Past of the Web" (hashtag #fpw11 if you're so inclined.) It's the third in an occasional series; I gave the closing presentation at the second event and I have been asked to be on a closing panel this time round. One of the things we've been asked to reflect on is what changes have taken place since the last event and how far our expectations have been realised. I thought it would be useful to set my thoughts on this down in advance, partly to help me articulate my own thinking. It will be interesting to see how various views develop during the panel session tomorrow.


Looking back at my concerns in mid-2009 I'm greatly reassured. There were a number of worrying trends apparent in web archives at that time and an apparent lack of bold vision in how we might use web archives in the future - or even in the present. My fear was that the collecting policies, preservation policies and interfaces offered were all taking a very human and document-centric view of what a web archive should do. In OAIS terms, the Designated Community was people who wanted to view individual old web pages having done a search for a particular site, or possibly for a keyword of some sort. The National Archives had taken one incremental but powerful step beyond that, automatically linking archived web pages to 404 pages on government web sites via simple plugins for Apache & IIS, but in the end this still involved serving individual pages for people to read.

That's a valid use case, but by no means the only ones. I set out a few other things we might want to be able to do but could not with the interfaces that web archives gave us.

• What search results would we have got on the web of 1998 using the search engines of 1998?


• What results would we have got using current search engines on the web of 1998?
  


• How can we visualise the set of links to or from a particular site changing over time?


• Treating the web as a corpora of text over time how can we track the emergence of words or concepts and their emergence from specialist vocabulary to general use?


• As historians of technology, how we can use a web archive to track things like the emergence of PNG as an image format and the decline of XPM (the original icon format for graphical browsers such as Mosaic)?

I also wanted to show how open APIs or RESTful interfaces can allow others to develop innovative ways to view content. Since there weren't any web archives with such interfaces I fell back on demonstrating the point with Flickr, more particularly with simple visual beauty that is TagGalaxy. TagGalaxy shows how the ability to search and retrieve images and tags lets someone else build a completely different interface to the Flickr repository, one which minimises textual interaction and which encourages serendipitous discovery. It would have been wonderful to be able to do that with a web archive. Similarly, if Brian Kelly had been able to say to the Internet Archive 'give me all the versions of the home page of the University of Bath between these dates' in a single interaction, it would have been much easier for him to build the informative animation he used in his own presentations for JISC PoWR. I could go on, and at the time I did.

Much of what I hoped for then has happened. The architecture of Memento makes it straightforward to view collections of web archives as a single entity from some viewpoints. Projects funded by "Digging Into Data" have shown the power of large web collections in viewing the web as data at many levels. And although most (all?) web archives are not yet offering the APIs or interfaces that would permit us to do some of the things above, I think they at least accept that these are valid aspirations.

Moreover, web archiving has moved from being a specialist concern to something that appears in the letters pages of national newspapers. That, and the type of talks we're going to hear tomorrow, show how far we've moved in 2 1/2 years. I'm quietly confident that things are getting better.

Persistence of domain names

I had a chat before Christmas with Henry Thompson, who works both in Edinburgh Informatics and also on the W3C TAG. Insofar as the Internet is important in sustaining long term access to information in digital form, there is a sustainability problem that we rather seem to have ignored. Everything on the Internet (literally) depends on domain names, and these are only ever rented. There is no mechanism for permanently reserving a domain name. Domain names can be lost by mistake (overlooking a bill, perhaps having moved in the interim and not informed the relevant domain name registrar), but they can also be lost on business failure. Although domain names can be a business asset, I understand that the registrars have some discretion on transfers, and in particular one cannot make a "domain name will" seeking transfer of the domain name to some benevolent organisation. Note, the mechanism for renting domain names has sustainability advantages, providing sustainability to important services that underpin the DNS.

There are two kinds of problem, one on a massive scale and one more fine-grained. The massive problem is that the entire infrastructure of the Internet depends on URIs, most of which are http URIs that in turn depend on the domain name system. So there are a number of organisations whose domain names are embedded in that infrastructure in a way and to an extent that is very difficult to change. W3C is clearly such an organisation. Many of these organisations seem rather fragile (not a comment on W3C, by the way, although its sustainability model is opaque to me). Should they fail and the domain names disappear, the relevant URIs will cease to work and various pieces of Internet machinery will fall apart.

(By the way, this does seem to be one case where a persistent ID that is independent of the original domain, such as a DOI, has advantages over a HTTP URI plus a redirect table. If the domain name no longer exists, you can't get to a redirect, whereas someone can still relink the DOI to a new location.)

On the more fine-grained scale, many documents (particularly in HTML) are not easily separable from their location, depending on other local files and documents. In addition of course, documents in some sense exist through their citations or bookmarks, that begin to exist separately from the document. Moving a document to a new domain can make it "fail" or disappear. So sustainability is linked to the domain as well as the other preservation factors.

This seems to me to be not at all a technical problem, but it seems to have legal/regulatory, governance, social, business and economic aspects.

Among the solutions might be creating a new top level domain designed for persistence, with different rules of succession, etc. Another (either instead of or in conjunction with the first) might be creating an organisation designed for persistence, to hold endowed domain names. Somehow the ongoing revenue stream for those underpinning services must be retained indefinitely into the future.

We don't think we have the answers, but we do think there is a problem here; I'm not yet sure if we have articulated it accurately at all. I would appreciate any comments. Thanks,

iPres 2008 Web archiving

Thorstheinn Hailgrimsson from Iceland, talking about the IIPC. Established to foster development and use of common tools, techniques and standards, and to encourage national libraries to collect web materials. Now have 38 members, heavily concentrated in Europe and US. Currently in Phase 2 and planning Phase 3; Phase 2 has 4 working groups, on access, harvesting, preservation and standards. Achievements include enhancements to Heretrix crawler, tools like Web Curator from BL/NZNL, and Netarchive Curator Tool Suite from Denmark, plus access tools including NutchWAX for indexing, and an open source version of the Wayback machine. Also a WARC standard near the end of the ISO process. Three main approaches: bulk, selective based on criteria, eg publicly funded stuff, and event-based such as around an election, disaster, etc. Iceland with 300,000 population already has 400 million documents, 8 TB data, so larger domains soon get into billions of documents, and many, many terabytes!

Helen Hockx Yu from the BL talking about web archiving in the UK. Started with UKWAC in 2004, which does permission-based selective archiving. Covers 3,700 unique web sites and >11,000 instances with about 2 TB data. Building an Ongoing Web Archiving Programme for the UK domain. Biggest problem really is that legal deposit legislation not yet fully implemented, and not expected until 2010! Failing he legislative mandate, permission-based archiving is slow and typical success rate is 25%. So suspect valuable web sites are already being lost.

Birgit Henriksen from Denmark: they have been archiving some parts of the web since 1998, and following new legislation, have been doing full Danish domains since 2005. In 2008 2.2 billion objects, 71 TB data, 5 FTE staff. Access is only for research and statistical purposes, which is hard to implement. They do bulk harvests quarterly, selective sites more frequently (sometimes daily), and event-based. Events are things that create debate or are expected to be of importance to Danish history. Preservation involved from the start: redundancy in geography, hardware architecture and vendor, storage media and software, and do active bit preservation based on checksum comparisons.

Gildas Illian from BNF in France; they have a legal deposit law since 2006. Now have 12 Billion URIs, 120TB, 7 FTE, 100 curators and partners involved. Partnered with Internet Archive. Provided in-house access since April 2008. Challenge of change: digital curators not a new job, need to change librarianship. Legal Deposit team as “honest broker” between collections/curators and the IT development and operations.

Colin Webb from NLA in Australia: been doing selective web archiving since 1 April, 1996, with negotiated permissions, quality control and access (however, get about 98% permission success). Also full domain harvests each year since 2005 (1 billion files this year). No legal deposit law (yet, despite lobbying), so have not yet provided access to the full domain crawls. Challenges are interconnected: what we want to collect, what we’re allowed to collect, what we’re able to collect (?) and what we can afford to collect.

Colin then talking about the preservation working group of IIPC, set up in 2007 after the initial concerns to get capture organised had abated. Brief: to identify preservation standards and practices applicable to web archives. Raises questions about whether web preservation differs from “ordinary” digital preservation. What about the scale effects? Work plan includes annual survey of web technical environment (browsers needed, etc). Comment suggested how important this was, hoped they would publish results and continue for many years to do so.

Question: can people whose web site is at risk get it preserved? This has started to happen. Also starting to see requests from people who have had disasters, wanting (and getting) help in recovering their web sites! Nice French story about political parties who have re-done their web sites, and now don’t remember what they promised a few years ago!

iPres 2008 session 2

Oya Rieger has been speaking about their large scale book digitisation processes. They first entered an agreement with Microsoft, and later with Google; they were naturally very disappointed when Microsoft pulled out, although this did give them unrestricted access to the books digitised under that programme. On the down-side, they suddenly found they need 40 TB of storage to manage these resources, and it took a year or so before they could achieve this. Oya related their work to the OAIS preservation reference model, and it was interesting to see not only that infamous diagram, but also a mapping of actual tools to the elements of the process model. It’s worth looking at her paper to see this; I noted that they were using ADORe for the archival storage layer, but there were several other tools that I did not manage to note down.

The next speaker, Kam Woods from Indiana has been talking about problems in making access to, and preserving, large CD collections published by the Government Printing Office in the US. With time, the old method of issuing CDs to individual readers and having them mount and view these on library workstations, has become increasingly impractical, so they needed to make these available online and preserve them. Notable here was the extent of errors in an apparently well-standardised medium. Because of the wide variety of environments, their solution was a virtualisation approach; they have now made over 4000 CDs available online, see www.cs.indiana.edu/svp/.

Brian Kelly spoke very rapidly (40 slides in 15 minutes, but the slides are on Slideshare and a video of the talk will be posted as well) on the JISC PoWR project (Preservation of Web Resources, http://jiscpowr.jiscinvolve.org/). This is a very entertaining talk, and well worth looking up. Brian is not a preservationist, but is a full-blown technogeek discussing the roles of the latest Web 2.0 technologies on his blog, in his role as UK Web Focus. His initial finding at Bath, slightly tongue-in-cheek, was that neither the archivist nor the web-master were initially interested in the preservation of their web presence, although by the end of the project both were much more convinced! This project achieved a strong level of interaction through its several workshops, and will shortly release a guide resource.

Adrian Brown of TNA spoke for his colleague Amanda Spencer, unable to attend, about trying to improve web resource continuity through web archiving. He quoted, I think, an initial finding that 60% of web links in Hansard (the record of parliamentary debate) were broken!

Steve Abrams closed the session with an interesting talk on JHOVE2, currently in development. Most notable was a change from a 1-to-1 match between digital object (file) and format, recognising that sometimes one object has multiple components with different formats. There is an interesting change of terminology here: source units and reportable units, with often a tree relationship between them. Their wiki is at http://confluence.ucop.edu/display/JHOVE2Info/Home .

Influence the PoWR

I've been very interested in the JISC Preservation of Web Resources project, which is being undertaken jointly by UKOLN at Bath University and the ULCC Digital Archives department. Some good discussions have taken place on their blog. Now I hear from Marieke Guy that the 3rd PoWR workshop is to take place in Manchester on 12 September, 2008:

"The JISC-sponsored Preservation of Web Resources project (JISC-PoWR) will be running its third and final workshop in Manchester. The series of workshops is aimed at the UK HE/FE records management and Web management communities.

The workshop, entitled 'Embedding Web Preservation Strategies Within Your Institution', will be held from 10.30 am - 4pm on Friday 12th September 2008 at the Flexible Learning Space, University of Manchester. It is free to attend and open to all members of HE/FE Institutions and related HE and FE agencies although we may need to restrict the numbers per institution if we are over-subscribed.

The aim of the workshop is to gain and share feedback from institutional Web, information and records managers on the proposed JISC-PoWR handbook, to address ways of embedding the proposed recommendations into institutional working practices and to solicit ideas for further work which may be needed.

Further details and the booking form are available from the JISC-PoWR blog:
http://jiscpowr.jiscinvolve.org/workshops/workshop-3/

The booking deadline is Friday 5th September 2008 and places are limited."

That should be a must-attend for anyone interested in preservation and the web!