iPres 2008: Risks and Costs

Paul Wheatley of the BL talking about LIFE2, a refinement of the earlier LIFE project. The LIFE model is a map of the digital life cycle from the point of view of the preserving organisation. The LIFE V2.0 model is slightly reorganised and refined, for example including sub-elements. It also has a methodology for application. In addition, there is the beginning of a Generic Preservation Model (GPM), derived from some desk study. Many issues still remain with the revised GPM, but further work will be carried out with the associated experts group, in anticipation of a possible LIFE3 project. Paul commented on how expensive it was to obtain these costs. It’s clear that much still needs to be done before these costs can be used as predictors

Rory McLeod of the BL talking about a risk-based approach to preservation. The 8 Business Change Managers act through the risk assessment to identify the assets, the risks to those assets, and possible reactions to those risks (ie “save” those assets). 23 separate risks were identified, and were aggregated into 6 direct and 2 indirect groups of risks. Physical media deterioration was the major risk, together with technical obsolescence for hand-held media. These two groups were complemented by format obsolescence and 3 further software-related risk groups. The two indirect risks are related to policy. In risk assessment, they are using the AS/NZ 4360:2004 risk standard, plus the DCC/DPE DRAMBORA toolkit.

Richard Wright talking about storage and the “cost of risk”. In early days dropping a storage device meant losing a few kilobytes, now it could be GBytes and years of work. Storage costs declining and capacity increasing exponentially roughly related to Moore’s law (doubling every 18 months). Usage is going up, too, and risk is proportionate to usage, so risk is going up too. Risk proportional to no of devices and to size and to use… plus the more commonly discussed format obsolescence, IT infrastructure obsolescence etc. So if storage gets really cheap, it gets really risky! Control of loss gets most attention: reduce MTBF, make copies, use storage management layers, introduce virtual storage, using digital library technology, etc. Mitigation of loss gets much less attention. Simple files can be read despite errors; more complex compressed files can be extremely fragile. Files with independent units have good properties. Reports work from Manfred Thaller of Koln: one bad byte affects only that byte of a TIFF (does this depend on selected compression?), 2% of a JPEG, and 17% of a JPEG2000. Demonstrated 5 errors on a PNG and a BMP: former illegible, latter has a few dots scattered about. Text files the best: one byte corruption affects only that byte! Risk can always be reduced by adding money: more copies, more devices, more reliable devices, less data per data manager. However, you actually have a finite budget, so the trade-offs are important. Can’t emphasise how important this is: one of the most worrying preconceptions in digital preservation is that the bit preservation element is a solved problem. It isn’t!

Bill Lefurgy of Library of Congress setting the scene for a report on the impact of international copyright laws on digital preservation. Adrienne Muir summarising the 4 country reports. All had preservation-related exceptions to copyright and related laws, however none of them are fully adequate to allow preservation managers to do what they need to do. All inadequate in different ways; UK maybe the strictest (as ever!). Differences in scope (which libraries), purpose and timing of copying, and material types within scope. Different legal and voluntary deposit arrangements; none comprehensive, they are out of date and don’t reflect the digital, online world. Orphan works (those whose copyright owners cannot be identified) a problem. Technical protection measures are a real preservation problem: current laws often prohibit reverse engineering or other circumvention. In the UK, contracts can trump copyright! Access to preserved content is an issue. Rights holders worry about the effects of the market of any legal changes, and aim to prevent such change. Recommendations include: there should be laws and policies to encourage digital preservation.

Questions: should we just ignore the copyright problems like Internet Archive and Google? We don’t have their resources in case there are serious problems, and there are serious potential impacts if we stamp all over these rights. Maybe for high value stuff, rights would be more vigorously pursued by rights holders, but they may also have motivation to preserve themselves. So maybe we should be looking for the stuff that’s most at risk of loss, some of which is low risk from the rights holders.

Responses to RAW versus TIFF: compression, error and cost-related

This is the second post summarising responses to the “RAW versus TIFF” post made originally by Dave Thompson of the Wellcome Library. The key elements of Dave’s post were whether we should be archiving using RAW or TIFF (image-related responses to this question are summarized in a separate post). A subsidiary question on whether we should archive both is greatly affected by cost, which is dependent on issues of compression, errors etc. Responses on these topics are covered in this post. As I mentioned before, these responses came on the semi-closed DCC-Associates email list.

Part way through the email list discussion, Marc Fresko of Serco reminded us that there are two questions:

“One is the question addressed by respondents so far, namely what to do in principle. The answer is clear: best practice is to keep both original and preservation formats.

The second question is "what to do in a specific instance". It seems to me that this has not yet been addressed fully. This second question arises only if there is a problem (cost) associated with the best practice approach. If this is the case, then you have to look at the specifics of the situation, not just the general principles.”

Marc’s comment brings us back to costs, and the responses to this are summarised here.

Sean Martin of the British Library picked up Marc’s point on costs:

“The cost of storage has and will get cheaper (typically 30% a year) but storage is not free. While most other examples here have cited raw vs TIFF, I will use an example based on TIFF vs a compressed format, such as JP2K.

(1) While there will be considerable variations, loss less compression with JP2K requires ~1/3 the storage of a TIFF. (I have a specific example of a 135 Mbyte TIFF and a 42 Mbyte loss less JP2K.)

(2) The cost of ownership of bulk commodity storage is currently of the order of £1000 per Tera Byte (However, let's treat this as a guide rather than a precise figure.)

(3) If you are dealing with only a few Tbytes then it will probably not matter much.

(4) However, if you envisage, say 100 Tbytes compressed or 300 Tbytes raw then it probably does matter, with indicative costs of £100K and £300K respectively.

(5) One way to approach this is to assess (in some sense) what value will be created using the cheaper method and what ADDITIONAL VALUE will be created using the more expensive method. This is a form of options appraisal.

(6) In my example, probably only a small amount of additional value is created for the additional expense approaching £200K. This leads to the question "if we had £200K on what would we spend it?" and probably the answer is "not in this way".

(7) However, this line of reasoning can only be applied in the context of a specific situation. Hence, for example, some might rate that considerable additional value is created and is affordable by storing RAW and TIFF, while others might chose to store only JP2K.

(8) This means that different people facing different challenges are likely to come to different conclusions. There is no one answer that applies to everyone.”

Chris Puttick, CIO of Oxford Archaeology thought that Sean’s cost figures were high:

“But the more nearly precise figure is what affects the rest of the calculation. For protected online storage (NAS/iSCSI, RAID 5 SATA drives with multiple hotspares) you can source 30TB useable space for £10k i.e. nearer to £50/TB [compared to Sean’s £1000 per TB; excludes power and cooling costs].

So using the boxes described above in a mesh array we can do 300TB for ~£120k. Anyone who needs 300TB of storage should find £120k of capital expenditure neither here nor there. After all, if we say the average HQ RAW is around 60MB, go with the 1/3 ratio for the additionally stored derivative image, we need 80MB per image i.e. 300TB=3.75m images... So a cost per stored image of 1p/year for the worst-case lifespan of the kit.”

To Sean’s cost versus value question, Chris responds:

“And in the wider world we have to fight to get digital preservation accepted as an issue, let alone tell people we have to pay for it. Put the question in the way that leads to the favourable response i.e. are you willing to spend a penny per image?”

Responding to the question about whether storing RAW as well as a compressed form is worthwhile, Chris points out:

“[…] many, asked the question in the wrong way, would opt for the cheapest without actually understanding the issues: JP2K is lossless, but was information lost in the translation from the RAW?

Richard Wright of BBC Future Media & Technology had an interesting point on the potential fragility risk introduced by compression:

“Regarding compression -- there was a very interesting paper yesterday at the Imaging Science & Technology conference in Bern, by Volker Heydegger of Cologne University. He looked at the consequences of bit-level errors. In a perfect world these don't occur, or are recovered by error-correction technology. But then there's the real world.

His finding was that bit and byte losses stay localised in uncompressed files, but have consequences far beyond their absolute size when dealing with compressed files. A one-byte error has a tiny effect on an uncompressed TIFF (as measured by 'bytes affected', because only one byte is affected). But in a lossless JP2 file, 17% of the data in the file is affected! This is because the data in the compressed file are used in calculations that 'restore' the uncompressed data -- and one bad number in a calculation is much worse than one bad number in a sequence of pixel data.

It goes on and on. A 0.1% error rate affects about 0.5% of a TIFF, and 75% of a lossless JP2.

The BBC is very interested in how to cope with non-zero error rates, because we're already making about 2 petabytes of digital video per year in an archive preservation project, and the BBC as a whole will produce significantly larger amounts of born-digital video. We're going for uncompressed storage -- for simplicity and robustness, and because "error-compensation" schemes are possible for uncompressed data but appear to be impossible for compressed data. By error-compensation I mean simple things like identifying where (in an image) an error has occurred, and then using standard technology like "repeating the previous line" to compensate. Analogue technology (video recorders) relied on error-compensation, because analogue video-tape was a messy world. We now need robust digital technology, and compression appears to be the opposite: brittle.”

I do think this is an extremely interesting point, and one that perhaps deserved a blog post all of its own, to emphasise it. In later private correspondence, Richard mentioned there was some controversy about this paper, with one person apparently suggesting that the factor-three compression might allow two other copies to be made, thus reducing the fragility. Hmmmm...

David Rosenthal of Stanford and LOCKSS picked up on this, noting that the real world really does see significant levels of disk errors. He also picked up on Sean’s comments, but unlike Chris, thinks that Sean’s costs are too low (and that Chris was way too optimistic):

“The cost for a single copy is not realistic for preservation. The San Diego Supercomputer Center reported last year that the cost of keeping one disk copy online and three tape copies in a robot was about $3K/Tb/yr.

Amazon's S3 service in Europe costs $2160/TB/yr but it is not clear how reliable the service is. Last time I checked the terms & conditions they had no liability whatsoever for loss and damage to your data. Note also that moving data in or out of S3 costs too - $100/TB in and sliding scale per month for transfers out. Dynamic economic effects too complex to discuss here mean that it is very unlikely that anyone can significantly undercut Amazon's pricing unless they're operating at Amazon/Google/... scale.

On these figures [costs] would be something like $300K per year compressed or $900K per year [un-compressed].

And discussions of preservation with a one-year cost horizon are unrealistic too. At 30% per year decrease, deciding to store these images for 10 years is a commitment to spend $10M compressed or $30M uncompressed over those ten years. Now we're talking serious money.

Sean is right that there is no one answer, but there is one question everyone needs to ask in discussions like this - how much can you afford to pay?”

Reagan Moore amplified:

“Actually the cost at SDSC for storing data is now:

~$420/Tbyte/year for a tape copy
~$1050/Tbyte/year for a disk copy. This cost is expected to come down with the availability of Sun Thumper technology to around $800/Tbyte/year.

As equipment becomes cheaper, the amortized cost for a service decreases. The above costs include labor, hardware and software maintenance, media replacement every 5 years, capital equipment replacement every 5 years, software licenses, electricity, and overhead.”

David responded:

“[…] To oversimplify, what this means is that if you can afford to keep stuff for a decade, you can afford to keep it forever. But this all depends on disk and tape technology continuing to get better at about the rate it has been for at least the next decade. The omens look good enough to use as planning assumptions, but anything could happen. And we need to keep in mind that in digital preservation terms a decade is a blink of an eye.”

David Barret-Hague of Plasmon believes that the cost of long term storage “is also a function of the frequency of data migrations - these involve manual interactions which are never going to get cheaper even if the underlying hardware does.” So he was not sure how affording it for 10 years means you can store it forever. (His company also has a white paper on cost of ownership of archival storage, available at http://www.plasmon.com/resources/whitepapers.html.)

Reagan responded agreeing that a long term archive cannot afford to support manual interactions with each record. He added:

“Our current technology enables automation of preservation tasks, including execution of administrative functions and validation of assessment criteria. We automate storage on tape through use of tape silos.

With rule-based data management systems, data migrations can also be automated (copy data from old media, write to new media, validate checksums, register metadata). Current tape capacities are 1 Terabyte, with 6,000 cartridges per silo. With six silos, we can manage 36 Petabytes of data.

We do have to manually load the new tape cartridges into the silo. Fortunately, tape cartridge capacities continue to increase, implying the number of files written per tape increases over time. As long as storage capacities increase by a factor of two each tape generation, and the effective cost of a tape cartridge remains the same, the media cost remains finite (factor of two times the original media cost).

The labor cost, the replacement of the capital equipment, the electricity, software licenses, floor space, etc. are ongoing costs.”

Well this discussion could (and maybe will) go on. But with my head reeling at the thought of one location with 36 Petabytes, I think I’m going to call this a day!