DCC Curation Lifecycle Model

I have not written much in this blog about Digital Curation Centre products, but I think it’s time to remedy that, and mention some of them. In particular, I wanted to mention the DCC Curation Lifecycle model, which is attracting widespread interest. Primarily put together by Sarah Higgins with input from colleagues across the DCC and external experts, like all such models it is of course a compromise between succinctness and completeness. Sarah has run a couple of workshops on it, including one in the US at JCDL 08 in Pittsburgh, and the response was extremely positive.

I hope to mention later how we will be using it to structure information on standards, and we are expecting to use it as an entry point to the DCC web site and the DCC DIFFUSE Standards Frameworks Project. In addition, I learned only recently that the more detailed proposals for the UK Research Data Service (which went to their Steering Committee a week or so back) lean heavily on the model (not apparent from their interim report). It is also used to explain the roles of data managers and data scientists in the JISC report “Skills, Role & Career Structure of Data Scientists & Curators: Assessment of Current Practice & Future Needs” by Alma Swan and Sheridan Brown.

Here is the graphic summary of the model:



I won’t attempt here to explain it in detail, as there’s sufficient additional information on the DCC web site and the longer IJDC article about the model.

iPres 2008 Foundations session and closing remarks

David Rosenthal from Stanford on bit preservation; is it a solved problem? Would like to keep a very large amount of data (say 1 PB) for a very long time (say a century) with a high probability of survival (say 50%). Mean time to data loss predictions are optimistic because they make assumptions (often not published) such as uncorrelated failure (whereas failure is often correlated… how many pictures did we see of arrays of disks in the same room, candidates for correlated failures from several possible causes). Practical tests with large disk arrays show much more pessimistic results. David estimates that to meet his target of a PB for a century with 50% probability of loss would require “9 9s” reliability, which is unachievable. I find his logic unassailable and his conclusions unacceptable, but I guess that doesn’t make them wrong! I suppose I wonder what the conclusions might have been about keeping a Terabyte for 10 years, if we had made the calculation 20 years ago based on best available data, versus doing those calculations now.

(Later: perhaps I’m thinking of this wrong? If David is saying that keeping a PB for a century with a 50% or greater chance of ZERO loss is extraordinarily difficult, even requiring 9 9s reliability, then that’s much, much more plausible, in fact downright sensible! So maybe the practical certainty of loss, forces us to look for ways of mitigating that loss, such as Richard Wright’s plea for formats that fail more gracefully..)

Yan Han from Arizona looked at modelling reliable systems, once you have established data for the components. Some Markov chain models that rapidly got very complex, but I think he was suggesting that a 4-component model might approach the reliability requirements put forward by RLG (0.001%? where did that number come from?). How this work would be affected by all the other error factors that David was discussing.

Christian Keitel from Ludwigsburg quoted one Chris Rusbridge as saying that digital preservation was “easy and cheap”, and then comprehensively showed that it was at least complicated and difficult. He did suggest that sensible bringing together of digital preservation and non-digital archival approaches (in both directions) could suggest some improvements in practice that do simplify things somewhat. So after the gloom of the first two papers, this was maybe a limited bit of good news.

Finally Michael Hartle of Darmstad spoke on a logic rule-based approach to format specification, for binary formats. This was quite a complex paper, which I found quite hard to assimilate. A question on whether the number of rules required was a measure of format complexity got a positive response. For a while I though Michael might be on the road to a formal specification of Representation Information from OAIS, but subsequently I was less confident of this; Michael comes from a different community, and was not familiar with OAIS, so he couldn’t answer himself.

In discussion, we felt that the point of David’s remarks was that we should understand that perfection was not achievable in the digital world, as it never was in the analogue world. We have never managed to keep all we wanted to keep (or to have kept) for as long as we wanted to keep it, without any damage.

At the end of the session, I felt perhaps I should either ask the web archivists to withdraw my quoted remarks, or perhaps we should amend them to “digital preservation is easy, cheap, and not very good”! (In self defence I should point out that the missing context for the first quote was: in comparison with analogue preservation, and we only needed to look out of our window at the massive edifice of the British Library, think of its companions across the globe, and compare their staff and budgets for analogue preservation with those applied to digital preservation, to realise there is still some truth in the remark!)

In the final plenary, we had some prizes (for the DPE Digital Preservation Challenge), some thanks for the organisers, and then some closing remarks from Steve Knight of NZNL. He picked up the “digital preservation deprecated” meme that appeared a few times, and suggested “permanent access” as an alternative. He also demonstrated how hard it is to kick old habits (I’ve used the deprecated phrase myself so many times over the past few days). Steve, I think it’s fine to talk about the process amongst ourselves, but we have to talk about the outcomes (like permanent access) when we’re addressing the community that provides our sustainability.

Where I did part company with Steve (or perhaps, massively misunderstand his intent), was when he apparently suggested that “good enough”… wasn’t good enough! I tried to argue him out of this position afterwards (briefly), but he was sure of his ground. He wanted better than good enough; he felt that posterity demanded it (harks back to some of Lynne Brindley’s opening comments about today’s great interest being yesterday’s ephemera, for instance). He thought if we aimed high we would fail high. Fresh from my previous session, I am worried that if we aim too high we will spend too high on unachievable perfection, and as a result keep too little, and ultimately fail low. But Steve knows his business way better than I do, so I’ll leave the last word with him. Digital Preservation: Joined Up and Working (the conference sub-title) was perhaps over-ambitious as a description of the current state, he suggested, but it was a great call to arms!

iPres 2008 Web archiving

Thorstheinn Hailgrimsson from Iceland, talking about the IIPC. Established to foster development and use of common tools, techniques and standards, and to encourage national libraries to collect web materials. Now have 38 members, heavily concentrated in Europe and US. Currently in Phase 2 and planning Phase 3; Phase 2 has 4 working groups, on access, harvesting, preservation and standards. Achievements include enhancements to Heretrix crawler, tools like Web Curator from BL/NZNL, and Netarchive Curator Tool Suite from Denmark, plus access tools including NutchWAX for indexing, and an open source version of the Wayback machine. Also a WARC standard near the end of the ISO process. Three main approaches: bulk, selective based on criteria, eg publicly funded stuff, and event-based such as around an election, disaster, etc. Iceland with 300,000 population already has 400 million documents, 8 TB data, so larger domains soon get into billions of documents, and many, many terabytes!

Helen Hockx Yu from the BL talking about web archiving in the UK. Started with UKWAC in 2004, which does permission-based selective archiving. Covers 3,700 unique web sites and >11,000 instances with about 2 TB data. Building an Ongoing Web Archiving Programme for the UK domain. Biggest problem really is that legal deposit legislation not yet fully implemented, and not expected until 2010! Failing he legislative mandate, permission-based archiving is slow and typical success rate is 25%. So suspect valuable web sites are already being lost.

Birgit Henriksen from Denmark: they have been archiving some parts of the web since 1998, and following new legislation, have been doing full Danish domains since 2005. In 2008 2.2 billion objects, 71 TB data, 5 FTE staff. Access is only for research and statistical purposes, which is hard to implement. They do bulk harvests quarterly, selective sites more frequently (sometimes daily), and event-based. Events are things that create debate or are expected to be of importance to Danish history. Preservation involved from the start: redundancy in geography, hardware architecture and vendor, storage media and software, and do active bit preservation based on checksum comparisons.

Gildas Illian from BNF in France; they have a legal deposit law since 2006. Now have 12 Billion URIs, 120TB, 7 FTE, 100 curators and partners involved. Partnered with Internet Archive. Provided in-house access since April 2008. Challenge of change: digital curators not a new job, need to change librarianship. Legal Deposit team as “honest broker” between collections/curators and the IT development and operations.

Colin Webb from NLA in Australia: been doing selective web archiving since 1 April, 1996, with negotiated permissions, quality control and access (however, get about 98% permission success). Also full domain harvests each year since 2005 (1 billion files this year). No legal deposit law (yet, despite lobbying), so have not yet provided access to the full domain crawls. Challenges are interconnected: what we want to collect, what we’re allowed to collect, what we’re able to collect (?) and what we can afford to collect.

Colin then talking about the preservation working group of IIPC, set up in 2007 after the initial concerns to get capture organised had abated. Brief: to identify preservation standards and practices applicable to web archives. Raises questions about whether web preservation differs from “ordinary” digital preservation. What about the scale effects? Work plan includes annual survey of web technical environment (browsers needed, etc). Comment suggested how important this was, hoped they would publish results and continue for many years to do so.

Question: can people whose web site is at risk get it preserved? This has started to happen. Also starting to see requests from people who have had disasters, wanting (and getting) help in recovering their web sites! Nice French story about political parties who have re-done their web sites, and now don’t remember what they promised a few years ago!

iPres 2008 Building Trust

Tyler Walters of GIT comparing trust in the banking system with trust in repositories. The upheavals in banking currently show how easily trust can be upset, and how hard it is to regain once lost. Further desk study on trust; 3 major sources (see paper for references). Useful idea of trust antecedents: what comes before an event that helps you trust, subjectively. Trust that endures is more likely to continue if there’s a breach of trust. Trust is a relationship between people, more than institutions. Trust development: engage, listen, frame issues, envision options, commit. Looks like a paper that’s well worth reading; it’s always seemed to me that trust is about much more than certifying against checklists.

Susanne Dobratz from Humboldt University on quality management and trust, work done in the nestor programme. Should we have separate standards and certification mechanisms (as in the CCSDS Repository Audit and Certification work), or build on existing security and quality management standards? Working though a project to assess applicability of these latter standards (ISO 9000, ISO 27000 etc). Did an analysis and a survey with a questionnaire to 53 persons, got responses from 18 (most responded but were not really able to answer the questions). DINI guidelines the most popular, formal standards little used. Standards mostly used as guidelines rather than formally (makes sense to me, having used ISO security standards that way). 94% knew their designated community, but only 53% have analysed the needs of their designated community. 61% were unsure whether a separate standard for trusted repositories was important; similar proportion unsure whether they would go through any certification process. As a result, I think they are interested in creating a smaller, more informal quality standard along the lines of DINI etc.

Sarah Jones on developing a toolkit for a Data Audit Framework: to help institutions to determine what data are held, where they are located and how they are being managed. 4 pilot sites including Edinburgh and Imperial already under way, UCL and KCL in planning. Detailed workflow has been developed as a self-audit tool. Four phases in the audit process, the second being identifying and classifying the assets; looks like major work. Turns out the pilots are related to department level rather than institutions, which makes sense knowing academic attitudes to “the Centre”! I did hear from one institution that it was difficult getting responses in some cases. Simple online tool provided. DAF to be launched tomorrow (1 October) at the British Academy, together with DRAMBORA toolkit.

Henk Harmsen from DANS on the Data Seal of Approval.. Since 2005, DANS has been looking after data in the social sciences and humanities, in the Netherlands. Aim was make archives DANS-proof, ie to follow acceptable standards; initially not sure how to implement. Looked at existing documents, aim to be complementary to all these requirements. Go for guidelines and overview rather than certifying and many details, trust rather than control. So Seal of Approval almost more a Seal of Awareness. Funding agencies beginning to require the Seal. Data producer responsible for quality of research data, repository responsible for storage, management and availability of data, and the data consumer responsible for quality of use. 17 guidelines, 3 for producers, 11 for repositories, 3 for consumers! Requirement 13 says technical infrastructure must explicitly support tasks and functions from OAIS… not necessarily implemented the same way, but using that vocabulary. How to control? Certification takes a long time, major standards, elaborate agencies, high cost.. Trust does need some support, though. First step: Data Seal of Approval Assessment, then other instruments like DRAMBORA etc. Looks like a nice, low impact, light touch first step towards building trust.

iPres 2008 National approaches

A bit late this morning (ahem!), so I missed some of the start of this. A very interesting and thoughtful exposition from Steve Knight of NZNL. They have a recently revised act, and have really taken digital preservation to heart: quote from their CEO “no job un-changed”! It’s not the platform or software, it’s the business approach, and management of provenance. They gave up on ”pointless” discussions on build vs buy, commercial vs open source: what counts is the requirements. In the end they opted for a commercial solution from Ex Libris that is being implemented now (phase 1 to go live 30 October). Need joined up and working together tools; most are being implemented in isolation.

Plenty of discussion about collaboration. Vicky Reich suggested that perhaps collaboration might be more focused with specific issues to solve. For example, Steve Knight had suggested there was a need to decide on registry approaches (PRONOM, GDFR, RRORI). Maybe integrating tools might be another issue. Martha Anderson pointed out that successful partnerships centred around content on the one hand and mutual local benefit on the other.

Comment that the Digital Preservation Coalition in the UK and nestor in Germany and neighbouring German-speaking countries are both very valuable; is there a role for similar long-lived (non-project) pan-national organisations? Similar organisations suggested in Netherlands and Denmark, and also the Alliance for Permanent Access to the Records of Science.

Question from Steve Knight about how we move to a position where there is a market for digital preservation solution?

iPres 2008- past, present and future

Heike Neuroth talking about the start of iPres, conceived at IFLA in 2003. Starting in Beijing, then Germany, next the US at Cornell, then back to Beijing, before coming to the UK this year. She thinks this child is growing up unstoppably now. Adam Farquhar commenting on the increasingly practical nature of reports: “doing it”. Tools being improved through experience in use, at scale. More tools in the toolbox, and we can put them together more flexibly. This year at iPres we have full papers, which should allow more people to get involved in the discussion. Patricia Cruse on iPres 2009 to be hosted by CDL in San Francisco, October 5-6, brand new conference facility at UCSF, downtown (immediately after a large, free blue-grass music festival in Golden Gate Park). Theme is “moving into the mainstream”. Some continuity of programme committee. Question on how to better integrate research with iPres, for example the NSF Datanet initiative, which explicitly bridges the science-library divide. Seek to coordinate with the IDCC and Sun PASIG conferences, so we don’t get to hear the same thing each time.

iPres 2008: Risks and Costs

Paul Wheatley of the BL talking about LIFE2, a refinement of the earlier LIFE project. The LIFE model is a map of the digital life cycle from the point of view of the preserving organisation. The LIFE V2.0 model is slightly reorganised and refined, for example including sub-elements. It also has a methodology for application. In addition, there is the beginning of a Generic Preservation Model (GPM), derived from some desk study. Many issues still remain with the revised GPM, but further work will be carried out with the associated experts group, in anticipation of a possible LIFE3 project. Paul commented on how expensive it was to obtain these costs. It’s clear that much still needs to be done before these costs can be used as predictors

Rory McLeod of the BL talking about a risk-based approach to preservation. The 8 Business Change Managers act through the risk assessment to identify the assets, the risks to those assets, and possible reactions to those risks (ie “save” those assets). 23 separate risks were identified, and were aggregated into 6 direct and 2 indirect groups of risks. Physical media deterioration was the major risk, together with technical obsolescence for hand-held media. These two groups were complemented by format obsolescence and 3 further software-related risk groups. The two indirect risks are related to policy. In risk assessment, they are using the AS/NZ 4360:2004 risk standard, plus the DCC/DPE DRAMBORA toolkit.

Richard Wright talking about storage and the “cost of risk”. In early days dropping a storage device meant losing a few kilobytes, now it could be GBytes and years of work. Storage costs declining and capacity increasing exponentially roughly related to Moore’s law (doubling every 18 months). Usage is going up, too, and risk is proportionate to usage, so risk is going up too. Risk proportional to no of devices and to size and to use… plus the more commonly discussed format obsolescence, IT infrastructure obsolescence etc. So if storage gets really cheap, it gets really risky! Control of loss gets most attention: reduce MTBF, make copies, use storage management layers, introduce virtual storage, using digital library technology, etc. Mitigation of loss gets much less attention. Simple files can be read despite errors; more complex compressed files can be extremely fragile. Files with independent units have good properties. Reports work from Manfred Thaller of Koln: one bad byte affects only that byte of a TIFF (does this depend on selected compression?), 2% of a JPEG, and 17% of a JPEG2000. Demonstrated 5 errors on a PNG and a BMP: former illegible, latter has a few dots scattered about. Text files the best: one byte corruption affects only that byte! Risk can always be reduced by adding money: more copies, more devices, more reliable devices, less data per data manager. However, you actually have a finite budget, so the trade-offs are important. Can’t emphasise how important this is: one of the most worrying preconceptions in digital preservation is that the bit preservation element is a solved problem. It isn’t!

Bill Lefurgy of Library of Congress setting the scene for a report on the impact of international copyright laws on digital preservation. Adrienne Muir summarising the 4 country reports. All had preservation-related exceptions to copyright and related laws, however none of them are fully adequate to allow preservation managers to do what they need to do. All inadequate in different ways; UK maybe the strictest (as ever!). Differences in scope (which libraries), purpose and timing of copying, and material types within scope. Different legal and voluntary deposit arrangements; none comprehensive, they are out of date and don’t reflect the digital, online world. Orphan works (those whose copyright owners cannot be identified) a problem. Technical protection measures are a real preservation problem: current laws often prohibit reverse engineering or other circumvention. In the UK, contracts can trump copyright! Access to preserved content is an issue. Rights holders worry about the effects of the market of any legal changes, and aim to prevent such change. Recommendations include: there should be laws and policies to encourage digital preservation.

Questions: should we just ignore the copyright problems like Internet Archive and Google? We don’t have their resources in case there are serious problems, and there are serious potential impacts if we stamp all over these rights. Maybe for high value stuff, rights would be more vigorously pursued by rights holders, but they may also have motivation to preserve themselves. So maybe we should be looking for the stuff that’s most at risk of loss, some of which is low risk from the rights holders.