SNIA "Terminology Bridge" report

Quite a nice report has just been published by the Storage Networking Industry Association's Data Management Forum, called "Building a Terminology Bridge: Guidelines for Digital Information Retention and Preservation Practices in the Datacenter". I don't think I'd agree with everything I saw on a quick skim, but overall it looks like a good set of terminology definitions.

The report identifies "two huge and urgent gaps that need to be solved. First, it is clear that digital information is at risk of being lost as current practices cannot preserve it reliably for the long-term, especially in the datacenter. Second, the explosion of the amount of information and data being kept long-term make the cost and complexity of keeping digital information and periodically migrating it prohibitive." (I'm not sure that I agree with their apocalyptic cost analysis, but it certainly deserves some serious thought!)

However, while still addressing these large problems, they found that what "began as a paper focused at developing a terminology set to improve communication around the long-term preservation of digital information in the datacenter based on ILM[*]-practices, has now evolved more broadly into explaining terminology and supporting practices aimed at stimulating all information owning and managing departments in the enterprise to communicate with each other about these terms as they begin the process of implementing any governance or service management practices or projects related to retention and preservation."

It's worth a read!

(* ILM = Information Lifecycle Management, generally not related to the Curation Lifecycle, but oriented towards management of data on appropriate storage media, eg moving less-used data onto offline tapes, etc.)

Load testing repositories

One of the issues that has worried me about moving from repositories of e-prints to repositories of data is the increased challenges of scale. Scale could be vastly different for data repositories in several dimensions, including

• rate of deposit
• numbers of objects
• size of objects
• rate of access
• rate of change to existing objects...
  

Now Stuart Lewis is reporting on the first stage of the JISC-funded ROAD project, where they have load-tested a DSpace implementation (on a fairly chunky configuration), loading 300,000 digital objects of 9 MB each.

Stuart reports

• "As expected, the more items that were in the repository, the longer an average deposit took to complete.
• On average deposits into an empty repository took about one and a half seconds
• On average deposits into a repository with three hundred thousand items took about seven seconds
• If this linear looking relationship between number of deposits and speed of deposit were to continue at the same rate, an average deposit into a repository containing one million items would take about 19 to 20 seconds.
• Extrapolate this to work out throughput per day, and that is about 10MB deposited every 20 seconds, 30MB per minute, or 43GB of data per day.
• The ROAD project proposal suggested we wanted to deposit about 2Gb of data per day, which is therefore easily possible.
• If we extrapolate this further, then DSpace could theoretically hold 4 to 5 million items, and still accept 2B of data per day deposited via SWORD."

They plan to repeat these tests with ePrints and FEDORA platforms.

Like all such, it's an artificial test, but it does give encouragement that DSpace could scale to handle a data repository for some tasks. I don't know if other issues would be show-stoppers or not, for something like a lab repository, but most of the scale issues seem OK.