Digimap is 10

I had a very enjoyable day yesterday helping EDINA celebrate 10 years of the Digimap service. What began as an eLib project and experiment with 6 Universities in 1996 has grown to a mature service with over 100,000 users, 45,000 of them active, in pretty much every UK University, and soon in UK schools as well.

In 1996 I was Programme Director of the eLib Programme, and my earliest email about Digimap was from the JISC money man, Dave Cook, on 30 January 1996 to Peter Burnhill of the Edinburgh Data Library (as it then was). Dave told Peter we were interested in his idea (for an Images project!) but had a few concerns (that the Ordnance Survey might not agree to let us use their mapping data; it’s hard to remember now how difficult some of those 1990s persuasions were!). Three days later, Dave was offering real money, although it had to be spent by 20 March that year. Done!

By late 1997 the Digimap project (*) had a trial service; I remember experimenting with it and having some problems (this was with Netscape 3 on a PowerMac Duo or something like that; woefully under-powered in retrospect). By the end of 1999, they were moving to a new GIS system, and we were beginning to discuss turning Digimap into a service, and that went live in January 2000. They had to get 37 subscribing Universities by a particular deadline, and I think managed 39 by somewhat earlier.

Since then the service has grown in scope, quality, usage and value. In my personal opinion (full disclosure, I’m not neutral here, having been associated with it through advisory groups of various kinds throughout its life), Digimap is the best service funded by JISC. Best in quality, best in professionalism, best in innovation, best in support. A lot of people deserve credit for that, and EDINA should all be extremely proud of what they have created. By the way, the OS have managed some major shifts in attitude over the years, from suspicious tolerance through to strong support, and the success is partly down to them, and to the efforts of the negotiators in what is now JISC Collections.

As well as various forms of OS mapping for GB (whose trademark names always escape me... and it is GB rather than UK, for weird historical reasons), Digimap now offers 4 “epochs” of historic maps from Landmark, plus Geology maps from BGS and Marine maps from SeaZone. Due to licence restrictions it is only available to registered staff and students at subscribing UK institutions, but I hope that those of you unlucky enough not to fall in that category can soon read more about it on the pages to be put up related to the celebration.

Digimap has been a bit clunky at times compared with the innovations introduced by some others, but with the new underlying GIS, the interfaces are being upgraded; they now have “slippy maps” (called Digimap Roam) on the base service, and it looks really smart and much more functional. It's tough for a small group to keep up with the likes of Google, Yahoo and MS! Soon this slippy map interface will be extended to the Historic service (“Ancient Roam”?), Geology (“Rock’n Roam”?) and Marine (a rather dull “H2Roam”!)… I think those might be internal names, but if you can complete the set with an even punnier marine name, who knows they might keep them!

The day was good fun, and we heard quite a bit about what Digimap is and how it is being used (far more widely than geography departments). The most exciting was a student project using Digimap and a GPS for a light aircraft CFIT-avoidance system (CFIT is Controlled Flight Into Terrain, referred to as “having a bad day”!). We heard from the data suppliers, with a bit more about what’s coming. It was interesting to hear the OS man talking about moves towards Linked Data; I wasn’t sure how that would square with the closed access, but I think I muddled my question (confused Linked Data with OGC web services, I suspect). The service providers didn’t appear to be talking to each other about Linked Data, which might be a good start.

A highlight was the closing keynote from Vanessa Lawrence, CEO of OS, clearly extremely supportive of OS. Choosing her words very carefully (she is not allowed to influence anyone) she outlined the government’s open data initiative and the consultation on its implications for the OS; this consultation closes late March 2010, but she urged us to make any responses, whether collectively or as private citizens well before then. The consultation isn’t simply “should we open up access to OS data?”, it’s much more “how can we open up access to OS data and still sustain the quality of the data into the future”.

The celebration ended with a reception and dinner, with an amusing after-dinner talk by Michael Parker, author of Map Addict. All in all, a very enjoyable and worthwhile day to celebrate a significant anniversary.

PS the twitter tag is #digimap10; I’m not going to tag the post with it, as I’ve got far too many one-time tags that are a pain to manage…

PPS (*) Unfortunately the original Digimap project pages seem to have vanished, and the earliest Wayback Machine gathers appear to be faulty; the first successful gather I can find is

http://web.archive.org/web/20011021051021/edina.ac.uk/digimap/

... which seems to refer to the service, not the project.

Notes from NERC Data Management workshop 1

David Bloomer, NERC CIO (and Finance Director) opened the workshop, and talked about data acquisition, data curation, data access, and data exploitation, in the context of developing NERC Information Strategy. Apparently NERC does not currently have an Information Strategy, as the last effort was thrown out in Council. Clearly from his point of view, the issue was about working out whether data management is being done well enough, and how it can be done better within the resources available.

There were some interesting comments about licensing and charging: principles that I summarise as:

• encouraging re-use and re-purposing,
• all free for teaching and research,
• that the licence and its cost depends on the USE and not the USER, and
• that NERC should support all kinds of data users equally.

Not sure yet the full implications of this; it clearly doesn’t mean that all data is freely accessible to everyone! However, it sounds like a major improvement over recent practices, with some NERC bodies charging high prices for some of their data.

In the second session, after my talk which I have already blogged (although not the 20 minutes of panic when first the PowerPoint transferred from my Mac would not load up on Windows, and second my Mac would not recognise the external video for the projectors!), there were two short talks by Dominic Lowe from BADC on Climate Science Modelling Language, and Tim Duffy from BGS on GeoSciML.

My notes on the former are skeletal in the extreme (ie nonsense!), other than that it is an application of GML, and is based on a data model expressed with UML. However, I picked up a bit more about the latter.

The scope of GeoSciML is the scientific information linked to the geography, except the cartography. It is mainly interpretive or descriptive, and so far includes 25 internationally-agreed vocabularies. Taken 5 years or so to develop to this point. Based around GML, using UML for modelling, using OGC web services to render maps, format, query etc. Provides interoperability and perhaps a “unified” view, but does not require change to local (end) systems, nor to local schemas. Part of the claimed value of the process was exposing that they did not understand their data as well as they thought!

Data Curation for Integrated Science: the 4th Rumsfeld Class!

I had to talk today to a workshop of NERC (Natural Environment Research Council) Data Managers, on data curation for integrated science. Integrated science does seem to be a term that comes up a lot in environmental sciences, for good reasons when contemplating global change. However, there doesn’t seem to be a good definition of integrated science on the web, so I had to come up with my own for the purposes of the talk: “The application of multiple scientific disciplines to one or more core scientific challenges”. The point of this was that scientists operating in integrated science MUST be USING unfamiliar data. Then the implications for data managers, particularly environmental data managers, were that they must make their data available for unfamiliar users. What does this imply?

By some strange mental processes, and a fortuitous Google search, this led me to the Poetry & Philosophy of Donald H Rumsfeld, as exposed to the world by Hart Seely, initially in Slate in April 2003, and now published in a book, “Pieces of Intelligence”, by Seely. These poems are well worth a look for their own particular delight, but the one I was looking for you will probably have heard of in various guises, The Unknown:

‘As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.’
—Feb. 12, 2002, Department of Defense news briefing

Now this insightful (!) poem (set to music by Bryant Kong, available at http://www.stuffedpenguin.com/rumsfeld/lyrics.htm) perhaps defines 3 epistemological classes:

Known knowns,
Known unknowns, and
Unknown unknowns

Logically there should be a 4th Rumsfeld class: the unknown knowns. And I think this class is especially important for data management for unfamiliar users.

The problem is that in many research projects, there are too many people who “know too much”; with so much shared knowledge, much is un-documented. In OAIS terms, we are looking here at a small, tight Designated Community with a shared Knowledge Base, and consequently little need for Representation Information. In integrated science, and particularly environmental sciences, as the Community broadens and time extends, effectively the need for Representation Information increases. I’m using the terms very broadly here, and RepInfo can be interpreted here in many different ways. But the requirement is to make more explicit the tacit knowledge, the unknown knowns implicit in the data creation and acquisition process.

Interestingly, subsequent speakers seemed to pick up on the idea of making explicit the unknown knowns, so maybe the 4th Rumsfeld Class is here to stay in NERC!

A National Research Data Infrastructure?

Two weeks without a post? My apologies! How about this piece of speculation that has been brewing at the back of my mind for some time...

It is clear that a national research data infrastructure is needed, but there are problems with all of the approaches taken so far to address this. Subject data centres provide subject domain curation expertise, but there are scalability issues across the domain spectrum: it appears unlikely that research funders will extend their funding to a much larger set of these data centres (indeed the AHDS experience might suggest a concern to cut back). Institutional data repositories are being explored, but while disclosing institutional data outputs might provide sustainability incentives, and such data repositories might be managed at a storage level by developments from existing institutional library/archive and IT support services, it is difficult to see how domain expertise can be brought to bear from so many domains across so many disciplines. Meanwhile, various of the studies done by UKOLN/DCC with Southampton University suggest the value of laboratory or project repositories in assisting with curation in a more localised context.

To square this circle, perhaps we have to realise that the storage infrastructure and the curation expertise are orthogonal issues. It is reasonable to suggest that institutions, faculties, departments, laboratories or projects should manage data repositories, databases etc with varying degrees of persistence. But in terms of the curation of the data objects (ie aspects of appraisal, selection, retention, transformation, combination, description, annotation, quality etc), somehow expertise from each domain as a whole has to be brought to bear. It is tempting to think of this as a parallel with the "editorial board" function of a "virtual data journal". This would clearly only be scalable if it were managed across the sector, rather than individually for each data repository.

So we might suggest a federation of repositories on the one hand, and a collective organisation (or set of differing collective organisations) of curation expertise in different disciplines or domains on the other hand; the latter is referred to below as national curation mechanisms.

In such a system, we might see roles for some of the main stakeholders as follows:

1. Research funders define their policies and mandates, and their compliance mechanisms; Research Information Network to participate and assist?
2. Publishers likewise define their own policies and mandates with regard to data supporting publication; JISC and RIN could assist coordination here.
3. Research Institutions define their own policies, establish local research data infrastructure, and encourage appropriate researchers to participate in national curation mechanisms (could participation in these become an element of researcher prestige as membership of editorial boards currently is?).
4. Researchers are responsible for their own good practice in managing and curating their data (possibly in laboratory or project data repositories or databases), and where appropriate for participating in national curation mechanisms.
5. Subject/discipline domain data centres are responsible for managing and curating data in their domain, as required by their funders. They also assist in defining good practice in their domains and more widely, undertake community proxy roles (NSB, 2005), and participate in national curation mechanisms.
6. A number of bodies such as the Digital Curation Centre and the proposed UK Research Data Service could undertake some coordination roles in this scenario, and could also undertake good practice dissemination and skills development through knowledge exchange activities.

(NSB. (2005). Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century. Retrieved from http://www.nsf.gov/pubs/2005/nsb0540/)

What do you think? Is that at all plausible?

Load testing repositories

One of the issues that has worried me about moving from repositories of e-prints to repositories of data is the increased challenges of scale. Scale could be vastly different for data repositories in several dimensions, including

• rate of deposit
• numbers of objects
• size of objects
• rate of access
• rate of change to existing objects...
  

Now Stuart Lewis is reporting on the first stage of the JISC-funded ROAD project, where they have load-tested a DSpace implementation (on a fairly chunky configuration), loading 300,000 digital objects of 9 MB each.

Stuart reports

• "As expected, the more items that were in the repository, the longer an average deposit took to complete.
• On average deposits into an empty repository took about one and a half seconds
• On average deposits into a repository with three hundred thousand items took about seven seconds
• If this linear looking relationship between number of deposits and speed of deposit were to continue at the same rate, an average deposit into a repository containing one million items would take about 19 to 20 seconds.
• Extrapolate this to work out throughput per day, and that is about 10MB deposited every 20 seconds, 30MB per minute, or 43GB of data per day.
• The ROAD project proposal suggested we wanted to deposit about 2Gb of data per day, which is therefore easily possible.
• If we extrapolate this further, then DSpace could theoretically hold 4 to 5 million items, and still accept 2B of data per day deposited via SWORD."

They plan to repeat these tests with ePrints and FEDORA platforms.

Like all such, it's an artificial test, but it does give encouragement that DSpace could scale to handle a data repository for some tasks. I don't know if other issues would be show-stoppers or not, for something like a lab repository, but most of the scale issues seem OK.

Bryan Lawrence on metadata as limit on sustainability

Opening the Sustainability session at the Digital Curation Conference, Bryan Lawrence of the Centre for Environmental Data Archival and the British Atmospheric Data Centre (BADC), spoke trenchantly (as always) on sustainability with specific reference to the metadata needed for preservation and curation, and for facilitation for now and the future. Preservation is not enough; active curation is needed. BADC has ~150 real datasets but thousands of virtual datasets, tens of millions of files.

Metadata, in his environment, represents the limiting factor. A critical part of Bryan’s argument on costs relates to the limits on human ability to do tasks, particularly certain types of repetitive tasks. We will never look at all our data, so we must automate, in particular we must process automatically on ingest. Metadata really matters to support this.

Bryan dashed past an interesting classification of metadata, which from his slides is as follows:

• A – Archival (and I don’t think he means PREMIS here: “normally generated from internal metadata”)
• B – Browse: context, generic, semantic (NERC developing a schema here called MOLES: Metadata Objects for Linking Environmental Sciences)
• C – Character and Citation: post-fact annotation and citations, both internal and external
• D – Discovery metadata, suitable for harvesting into catalogues: DC, NASA-DIF, ISO19115/19139 etc
• E – Extra: Discipline-specific metadata
• O – Ontology (RDF)
• Q – Query: defined and supported text, semantic and spatio-temporal queries
• S – Security metadata

The critical path relates to metadata, not content; it is important to minimise the need for human intervention, and this means minimising the number of ingestion systems (specific processes for different data types and data streams), and to minimise the types of data transformations required (problem is validating the transformations). So this means that advice from data scientists TO the scientists is critical before creation; hence the data scientist needs domain knowledge to support curation.

Can choose NOT TO TAKE THE DATA (but the act of not taking the data is resource intensive). Bryan showed and developed a cost model based on 6 components; it’s worth looking at his slides for this. But the really interesting stuff was on his conclusions on limits, with 25 FTE:

"• We can support o(10) new TYPES of data stream per year.
• We can support doing something manually if it requires at most:
– o(100) activities of a few hours, or
– o(1000) activities of a few minutes, but even then only when supported by a modicum of automation.
• If we have to [do] o(10K) things, it has to be completely automated and require no human intervention (although it will need quality control).
• Automation takes time and money.
• If we haven’t documented it on ingestion, we might as well not have it …
• If we have documented it on ingestion, it is effectively free to keep it …
• … in most cases it costs more to evaluate for disposal than keeping the data.
• (but … it might be worth re-ingesting some of our data)
• (but … when we have to migrate our information systems, appraisal for disposal becomes more relevant)”

Interestingly, they charge for current storage costs for 3 years (only) at ingest time; by then, storage will be “small change” provided new data, with new storage requirements, keep arriving. Often the money arrives first and the data very much later, so they may have to be a “banker” for a long time. They have a core budget that covers administration, infrastructure, user support, and access service development and deployment. Everything changes next year however, with their new role supporting the Inter-Governmental Panel on Climate Change, needing Petabytes plus.

Curation services based on the DCC Curation Lifecycle Model

I’ve had a go at exploring some curation services that might be appropriate at different stages of a research project. I thought it might also be worth trying to explore curation services suggested by the DCC Curation Lifecycle Model, which I also mentioned in a blog post a few weeks ago.



The model claims that it “can be used to plan activities within an organisation or consortium to ensure that all necessary stages are undertaken, each in the correct sequence… enables granular functionality to be mapped against it; to define roles and responsibilities, and build a framework of standards and technologies to implement”. So it does look like an appropriate place to start.

At the core of the model are the data of interest. These should absolutely be determined by the research need, but there are encodings and content standards that improve the re-usability and longevity of your data (an example might be PDF/A, which strips out some of the riskier parts of PDF and which should make the data more usable for longer).

Curation service: collected information on standards for content, encoding etc in support of better data management and curation.
Curation service: advice, consultancy and discussion forums on appropriate formats, encodings, standards etc.

The next few rings of the model include actions appropriate across the full lifecycle. “Description and Representation Information” is maybe a bit of a mouthful, but it does cover an important step, and one that can easily be missed. I do worry that participants in a project in some sense “know too much”. Everyone in my project knows that the sprogometer raw output data is always kept in a directory named after the date it was captured, on the computer named Coral-sea. It’s so obvious we hardly need to record it anywhere. And we all know what format sprogometer data are in, although we may have forgotten that the manufacturer upgraded the firmware last July and all the data from before was encoded slightly differently. But then 2 RAs leave and the PI is laid up for a while, and someone does something different. You get the picture! The action here is to ensure you have good documentation, and to explicitly collect and manage all the necessary contextual and other information (“metadata” in library-speak, “representation information” when thinking of the information needed to understand your data long term).

Curation service: sharable information on formats and encodings (including registries and repositories), plus the standards information above.
Curation service: Guidance on good practice in managing contextual information, metadata, representation information, etc.

Preservation (and curation) planning are necessary throughout the lifecycle. Increasingly you will be required by your research funder to provide a data management plan.

Curation service: Data Management Plan templates
Curation service: Data Management Plan online wizards?
Curation service: consultancy to help you write your Data Management Plan?

Community Watch and Participation is potentially a key function for an organisation standing slightly outside the research activity itself. The NSB Long-lived data report suggested a “community proxy” role, which would certainly be appropriate for domain curation services. There are related activities that can apply in a generic service.

Curation service: Community proxy role. Participate in and support standards and tools development.
Curation service: Technology watch role. Keep an eye on new developments and critically obsolescence that might affect preservation and curation.

Then the Lifecycle model moves outwards to “sequential actions”, in the lifecycle of specific data objects or datasets, I guess. As noted before, curation starts before creation, at the stage called here “conceptualise: Conceive and plan the creation of data, including capture method and storage options”. Issues here perhaps covered by earlier services?

We then move on to “doing it”, called here “Create and Receive”. Slightly different issues depending on whether you are in a project creating the data, or a repository or data service (or project) getting data from elsewhere. In the former case, we’re back with good practice on contextual information and metadata (see service suggested above); in the latter case, there’s all of these, plus conformance to appropriate collecting policies.

Curation service: Guidance on collection policies?

Appraisal is one of those areas that is less well-understood outside the archival world (not least by me!). You’ve maybe heard the stories that archivists (in the traditional, physical world) throw out up to 97% of what they receive (and there’s no evidence they were wrong, ho ho). But we tend to ignore the fact that this is based on objective, documented and well-established criteria. Now we wouldn’t necessarily expect the same rejection fraction to apply to digital data; it might for the emails and ordinary document files for a project, but might not for the datasets. Nevertheless, on reasonable, objective criteria your lovingly-created dataset may not be judged appropriate for archiving, or selected for re-use. My guess is that this would often be for reasons that might have been avoided if different actions had been taken earlier on. Normally in the science world, such decisions are taken on the basis of some sort of peer review.

Curation service: Guidance on approaches to appraisal and selection.

At this point, the Lifecycle model begins to take on a view as from a Repository/Data Centre. So whereas in the Project Life Course view, I suggested a service as “Guidance on data deposit”, here we have Ingest: the transfer of data in.

Curation service: documented guidance, policies or legal requirements.

Any repository or data centre, or any long-lived project, has to undertake actions from time to time to ensure their data remain understandable and usable as technology moves forward. Cherished tools become neglected, obsolescent, obsolete and finally stop working or become unavailable. These actions are called preservation actions. We’ve all done them from time to time (think of the “Save As” menu item as a simple example). We may test afterwards that there has been no obvious corruption (and will often miss something that turns up later). But we’ve usually not done much to ensure that the data remain verifiably authentic. This can be as simple as keeping records of the changes that were made (part of the data’s provenance).

Curation service: Guidance on preservation actions
Curation service: Guidance on maintaining authenticity
Curation service: Guidance on provenance

I noted previously that storing data securely is not trivial. It certainly deserves thoughtful attention. At worst, failure here will destroy your project, and perhaps your career!

Curation service: Briefing paper on keeping your data safe?

Making your data available for re-use by others isn’t the only point of curation (re-use by yourself at a later date is perhaps more significant). You may need to take account of requirements from your funder or publisher or community on providing access. You will of course have to remain aware of legal and ethical restrictions on making data accessible. The Lifecycle Model calls these issues “Access, Use and Reuse”.

Curation service: Guidance on funder, publisher and community expectations and norms for making data available
Curation service: Guidance on legal, ethical and other limitations on making data accessible
Curation service: Suggestions on possible approaches to making data accessible, including data publishing, databases, repositories, data centres, and robust access controls or secure data services for controlled access
Curation service: provision of data publishing, data centre, repository or other services to accept, curate and make available research data.

Combining and transforming data can be key to extracting new knowledge from them. This is perhaps situation-specific, rather than curation-generic?

The Lifecycle model concludes with 3 “occasional actions”. The first of these is the downside of selection: disposal. The key point is “in accordance with documented policies, guidance or legal requirements”. In some cases disposal may be to a repository, or another project. In some cases, the data are, or perhaps must be destroyed. This can be extremely hard to do, especially if you have been managing your backup procedures without this in mind. Just imagine tracking down every last backup tape, CD-ROM, shared drive, laptop copy, home computer version etc! Sometimes, especially secure destruction techniques are required; there are standards and approaches for this. I’m not sure that using my 4lb hammer on the hard drives followed by dumping in the furnace counts! (But it sure could be satisfying.)

Curation Service: Guidance on data disposal and destruction

We also have re-appraisal: “Return data which fails validation procedures for further appraisal and reselection”. While this mostly just points back to appraisal and selection again, it does remind me that there’s nothing in this post so far about validation. I’m not sure whether it’s too specific, but…

Curation service: Guidance on validation

And finally in the Lifecycle model, we have migration. To be honest, this looks to me as if it should simply be seen as one of the preservation actions already referred to.

So that’s a second look at possible services in support of curation. I’ve also got a “service-provider’s view” that I’ll share later (although it was thought of first).

[PS what's going on with those colours? Our web editor going to kill me... again! They are fine in the JPEG on my Mac, but horrible when uploaded. Grump.]

Project data life course

This blog post is an attempt to explore the “life course” of an arbitrary small to medium research project with respect to data resources involved in the project. (I want to avoid the term life cycle, since we use this in relation to the actual data.)

This seems like a useful exercise to attempt, even if it has to be an over-generalisation, as understanding these interactions may help to define services that support projects in managing and curating their data better, and a number of such services are suggested here. This is about “Small Science”; James M. Caruthers, a professor of chemical engineering at Purdue University has claimed ‘Small Science will produce 2-3 times more data than Big Science, but is much more at risk’ (Carlson, 2006). Large and very large projects tend to have very particular approaches, and probably represent significant expertise; I’ll not presume to cover them.

Is this generalisation way off the mark? I’d really like to know how to improve it, accepting that it is a generalisation!

The overall life course of a project (I suggest) is roughly:

• pre-proposal stage (when ideas are discussed amongst colleagues, before a general outline of a proposal is settled on), followed by
• the proposal stage (constrained by funder’s requirements, and by the need to create a workable project balancing expected science outcomes, resources available, and papers and other reputational advantages to be secured). If the project is funded, there follows
• an establishment phase, when the necessary resources are acquired, configured and tested, (which overlaps and merges with)
• an execution phase when the research is actually done, and
• a termination phase as the project draws to a close.
• Somewhere during this process, one or more papers will be produced based on the research understandings gained, linked to the data collected and analysed.

My message here is that curation is most impacted by decisions in the first 3 phases, even though it requires attention throughout!

Pre-proposal stage
At the pre-proposal stage, you and your potential partners will mostly be concerned with working out if your research idea is likely to be feasible, how and with roughly what resources. During these conversations, you will need to identify any external or already existing data resources that will be required to enable the project to succeed. You should also start to identify the data that the project would create, or perhaps update, and how these data will support the research conclusions. In doing so, attention must be paid to resource constraints, not necessarily financial, but also legal and ethical issues affecting use of the data. However, discussions are more likely to be generic rather than focused! Thus begins the impact on curation…

Curation service: 5-minute curation introduction, with pointers to more information…
Curation service: Help Desk service. Ask a Curator?

Proposal stage
Apart from the obvious focus on the detailed plan for the research itself, this stage is particularly constrained by funder requirements. Increasingly, these requirements include a data management plan. This is unlikely to be a problem for large projects, which will already have spent some time (often lots of time) considering data management issues, but may well be a new requirement for smaller projects. However, forcing attention to be paid to data management is definitely a Good Thing! I’ll have a closer look at data management plans in a later blog post.

Curation service: Data Management Plan templates
Curation service: Data Management Plan online wizards?
Curation service: consultancy to help you write your Data Management Plan?

Key issues that the project should address, and which should be reflected in the data management plan, include the standards and encodings for analysing, managing and storing data. While these must be determined by a combination of local requirements, such as the availability of licences, skills and experience, and the local IT environment, they must take account of community norms and standards. While in some cases maverick approaches can lead to breakthrough science, they can also lead to silo data, inaccessible and not re-usable (sometimes not re-usable even by their creators, after a short while). This stage begins to have a significant effect on curation, as the decisions that are fore-shadowed here will have major effects later on. Repeat after me: curation begins before creation!

Curation service: Guidance on managing data for re-use in the medium to long term
Curation service: registries and repositories to support re-use, sharing and preservation
Curation service: pointers to relevant standards, vocabularies etc.

[Note, we have concerns about the scalability of the latter, which is why the DCC DIFFUSE service is being re-modelled to be more participative.]

Establishment stage
It’s during the establishment phase that the real work is in sight. In the case of a well-established laboratory or research centre, this may merely be “yet another project” to be fitted into well-known procedures, perhaps with some adaptations. But from my observation and anecdotally, there is often a high degree of individual idiosyncratic decision-making taking place here; the focus is often on “getting down to the research”. The consequences of this for curation may be very significant, but are unlikely to be apparent until much later.

The key curation aim is to set the data management plan into operation. This requires ensuring access to external data required by the project (an activity that should have been started, at least in principle, ahead of project approval). While you’re thinking of access, rights and ownership, negotiating and documenting data ownership and access agreements for the data created in the project would be a smart idea (too often ignored, or left until problems begin to arise). Of course, it’s not just rights that are at issue, but also responsibilities, particularly where data have privacy and ethical implications.

Curation service: Briefing papers on legal, ethical and licence issues.

Meanwhile a critical step is establishing the IT infrastructure. While the focus is likely to be on acquiring and testing the data acquisition and analysis equipment, software, capabilities and workflows, in practice the apparently background activities of ensuring adequate data storage, sensible disaster recovery procedures (or at the very least, proper data backup and recovery), and thoughtful data curation processes will be critical for later re-use.

Curation service: Briefing paper on Keeping your data safe?
Curation service: tools for various tasks, eg data audit, risk assessment

It’s worth remembering that there are different kinds of data storage. While you will often be most concerned with temporary or project-lifetime storage, you may need to think about storage with greater persistence. There may be a laboratory or institutional or subject repository, external database or data service that will be critical to your project; negotiating with the curation service providers associated with these services should be an important early step. In effect, right at the start you are planning for your exit! But more on this another time…

Curation service: Guidance on data repositories for projects?
Curation service: thought-stimulating discussion papers, blog posts and case studies, journal, conference, discussion fora…

Project execution
Through the lifetime of your project, your focus will rightly be on the research itself. But you should also take time out to test your disaster recovery procedures, and check on your quality assurance. You care about the quality of your materials and syntheses (or your algorithms, or your documentary sources), you need to care as much about the quality of your data, both as-observed and as-processed. You’ll be collecting a lot of data; if some support your hypothesis and some contradict it, you’ll need to ensure you are collecting (perhaps in laboratory notebooks or equivalent) sufficient provenance and context to justify the selection of the former rather than the latter.

Not sure if there is a curation service here, or if these are good practice research guides for your particular domain… but I guess the same could be said for standards and vocabularies. Question is whether domains HAVE good practice guides for data quality?

As the project goes forward, issues may arise in your original data management plan, and you should take the time to refine it (and document the changes), and your associated curation processes. If you did take idiosyncratic decisions in haste at project establishment, at some point you may need to review them. It’s a warning sign when a staffing change is followed by proposals for major architectural changes!

Finally, at some stage you should test any external deposit procedures built into your plan.

Curation service: Guidance on data deposit…

I don’t think I’m claiming that this is the limit of curation issues during project execution; rather that here the course of projects is so varied that generalisations are even less justifiable than elsewhere. Nevertheless, the analysis here should be strengthened, so input would be helpful!

Project termination stage
As the project comes towards an end, you enter another risk phase. Firstly, most management focus is likely to be on the next project (or the one after that!). Secondly, if employment is fixed term, tied to the particular project, then research staff will have been looking for other posts as the end of the project draws near, and the project may start to look denuded (and the data management and informatics folk may be the most marketable, leaving you exposed). And of course, the final reports etc cannot be written until the project is finished, by which time the project’s staffing resources will have disappeared. It’s the PI’s job to ensure these final project tasks are properly completed, but it’s not surprising that there are problems sometimes!

Curation service: Guidance on (or assistance with) evaluation

During this final phase, you must identify any data resources of continuing value (this should have occurred earlier, but does need re-visiting), and carry out the plans to deposit them in an appropriate location, together with the necessary contextual and provenance information. Easy, hey? Perhaps not: but failure here could have a negative impact on future grants, if research funders do their compliance checking properly.

Curation service: Guidance on appraisal, Guidance on data deposit.

Finally, given that data management plans are currently in their infancy, it’s worth commenting on the plan outcomes in your project final report!

Writing papers
This is not really a project phase as such, as often several papers will be produced at different stages of the project. The key issues here are:

• Include supplementary data with your paper where possible
• Ensure data embedded in the text are machine readable
• Cite your data sources
• Ensure supportive data are well-curated and kept available (preferably accessible; this is where a repository service may come in useful).

I’ll write more on this in a separate post.

Curation service: Proposals for data citation
Curation service: Suggestions for the Well-supported article

So: the message is that curation should be a constant theme, albeit as background during most of the project execution. But the decisions taken during pre-proposal, proposal and establishment phases will have a big effect on your ability to curate your data, which may affect your research results, and will certainly affect the quality and re-usability of the data you deposit.

Carlson, S. (2006, June 23). Lost in a Sea of Science Data: Librarians are called in to archive huge amounts of information, but cultural and financial barriers stand in the way. The Chronicle of Higher Education, 52, 35.

ARROW Repositories day: 3

Dr Alex Cook from the Australian Research Council (a money man! Important!) talking on the Excellence in Research Framework (ERA), the Access Framework and ASHER. ERA appears to be like the UK’s erstwhile RAE, and will use existing HE Research Data Collection rules for publication and research income information where possible. 8 clusters of disciplines have been identified. Currently looking at the bibliometric and other indicators which will be discipline-specific (principles, methodologies and a matrix showing which are used where). Developing the System to Evaluate Excellence of Research (SEER) that will involve institutions uploading their data; sounds like something that MUST (and they say will) work with repositories (RAE in UK too often went for separate bibliographic databases, not interoperable, which meant doing the same thing twice). Copyright still an issue (I wonder if they are brave enough to take a NIH mandate approach? See below… not so explicit, but some pressure that way). Where a research output is required for review purposes, institutions will be required to store and reference their Research Output Digital Asset. Repositories a natural home for this.

Accessibility framework requires research outputs to be made sufficiently accessible to allow maximum use of the research. ASHER a short term (2 year?) funding stream to help institutions make their systems and repositories more suitable for working in this new framework.

Andrew Treloar talking about ANDS, the Australian National Data Service, and its implications for repositories. Mentions again the Australian Code for the Responsible Conduct of Research. Institutions need to think about their obligations under this code, obligations that are quite significant! Good story about Hubble telescope: data must be made available at worst 6 months after capture; most of published data from Hubble is not “first use”! (Would this frighten researchers? I put all the work in, but someone else just does the analysis and gets the credit?)

Structure of ANDS: developing frameworks (eg encouraging moves towards discipline-acceptable default sharing practices), providing utilities (building and delivering national technical services to support the data commons, eg discovery, persistent identifiers, and collections registry), seeding the commons, and building capabilities, plus service development activities. ISO 2146 information model important here (I think I’ve already talked about this stuff from the iPres posts [update: no, it was from the e-Science All Hands Meeting, but the links still points to the right place]).

Australian Strategic Roadmap Review talks about national data fabric, based on institutional nodes, and a coordination component to integrate eResearch activities, and see expertise as part of infrastructure. There’s also a review of the National Innovation System: ensure data get into repositories, try to get data more freely available.

Implications for repositories: ANDS dependent on repositories, but doesn’t fund repositories! May need range of repository types for storing data. Big R and little r repositories: real scale issues in data repositories. Lots of opportunities for groups to take part in consultation etc. Different but related discovery service. Links to Research Information Systems. Persistent Identifier Service (in collaboration with Digital Education Revolution). (I worry about this: surely persistence requires local commitment, rather than remote services?)

Panel discussion… question about CAIRS, the Consortium of Australian Institutional Repositories, funding left over from ARROW, being run by CAUL, the Australian University Librarians, which now has an Invitation to Offer to find someone to run the proposed service.

Some questions about data, whether they should be in repositories or in filestore. Scale issues (numbers, size, rapidity of deposit and re-use etc) suggest filestore, but there are then maybe issues about integrity and authenticity. There will be ways of fixing these, but they imply new solutions that we don’t yet have. The data world will soon go beyond the current ePrint/DSpace/Fedora realms.

A few more questions, too hard to summarise, on issues such as the importance of early career scientists, on the nature of raw data etc. But overall, an extremely interesting day!

ARCHER: a component of Australian e-Research infrastructure?

At the e-Science All Hands meeting, David Groenewegen from Monash spoke (PPT, also from Nick Nicholas and Anthony Beitz) about the outputs of the ARCHER project, almost finished, intended to provide tools for e-Research infrastructure. They see these e-Research challenges:

• "Acquiring data from instruments
• Storing and managing large quantities of data
• Processing large quantities of data
• Sharing research resources and work spaces between institutions
• Publishing large datasets and related research artifacts
• Searching and discovering"

The tools they have nearly completed (the project finishes this month) include:

• "ARCHER Research Repository - for storing large datasets, based on SRB
• Distributed Integrated Multi-Sensor & Instrument Middleware – concurrent data capture and analysis
• Scientific Dataset Manager (Web) - for managing datasets
• Metadata Editing Tool
• Scientific Dataset Manager (Desktop) – for managing datasets
• Analysis Workflow Automation Tool - streamlining analysis
• Collaborative Workspace Development Tool - bringing researchers together"

The research repository is based around a simplified version of what they call "CCLRC Scientific metadata", which they said was the only appropriate set of metadata they could find at the time. Here's a simpified diagram:


This does look to me like the metadata for a Current Research Information System (CRIS); I've spoken of these before. Maybe the CERIF metadata might do a similar job? Later we heard that the ANDS programme was to use yet another metadata set or standard (ISO2146); OK guys, I got a bit confused at this point!

Clearly this is a generic system; they also have some customisations coming for crystallographers, including the wonderfully-named TARDIS (The Australian Repositories for Diffraction Images). It's not yet clear to me how domain science curation and domain-specific metadat fit into this model, but I hope to find out more next month.

So where next for ARCHER? David wrote:

• "Currently testing the tools for release by late September
• Some tools already out in the wild and in use
• Expecting that the partners will continue to develop the tools they created
• New enhanced versions already being worked on
• Looking at how these tools might be used within ANDS (Australian National Data Service) & ARCS (Australian Research Collaborative Service) and beyond!"

This does look like good stuff!

(I think ARCHER is a collaboration of ANU, James Cook, Monash and Queensland Universities.)

Semantic web on the Today programme

Turning on the radio this morning, I was surprised to hear someone discussing data on the Today programme on BBC Radio 4. It turned out to be Sir Tim Berners Lee talking about the semantic web; he even managed to mention RDF and HTML without confusing the interviewers too much. The interesting 8 and a half minute discussion is available via the BBC iPlayer.

I'm quite keen on understanding better how the semantic web might relate to science data. When people talk about data in relation to the semantic web, they often seem to be thinking the sort of relatively unitary facts or simple relationships that RDF triples are quite good at expressing, such as the population of China is 1,330,044,605. It's certainly not clear to me how this generalises to express the changing population, let alone how it could express the data from a spectrometer or
a crystal structure determination or remote-sensing data. If anyone can point me to a good resource discussing this, I would be grateful!

Research Repository System data management

This is the sixth of a series of posts aiming to expand on the idea of the negative click, positive value repository, which I'm now calling a Research Repository System. I've suggested it should contain these elements:

• web orientation
• researcher identity management
• authoring support
• object disclosure control
• data management support
• persistent storage
• full preservation archive, and
• spinoffs
  

Data management support is where this starts to link more strongly back to digital curation. Bear in mind here, this is a Research Repository System; not all of these functions, or the next group, need to be supported by anything that looks like one of the current repository implementations! I’m not quite clear on all of the features you might need here, but we beginning to talk about a Data Repository.

It is essential that the Data Management elements support current, dynamic data, not just static data. You may need to capture data from instruments, process it through workflow pipelines, or simply sit and edit objects, eg correcting database entries. Data Management also needs to support the opposite: persistent data that you want to keep un-changed (or perhaps append other data to while keeping the first elements un-changed).

One important element could be the ability to check-point dynamic, changing or appending objects at various points in time (eg corresponding to an article). In support of an article, you might have a particular subset available as supplementary data, and other smaller subsets to link to graphs and tables. These checkpoints might be permanent (maybe not always), and would require careful disclosure control (for example, unknown reviewers might need access to check your results, prior to publication).

Some parts of Data Management might support laboratory notebook capabilities, keeping records with time-stamps on what you are doing, and automatically providing contextual metadata for some of the captured datasets. Some of these elements might also provide some Health and Safety support (who was doing what, where, when, with whom and for how long).

4th International Digital Curation Conference - Radical Sharing: Transforming Science?

This is a reminder that closing date for submission of full papers, posters and demos for the conference (details below) is 25 July 2008 . We invite submissons from individuals, organisations and institutions across all disciplines and domains engaged in the creation, use and management of digital data, especially those involved in the challenge of curating data in e-Science and e-Research. Templates for submissions and other details are available at http://www.dcc.ac.uk/events/dcc-2008/ .

The 4th International Digital Curation Conference will be held on 1-3 December 2008 at the Hilton Grosvenor Hotel, Edinburgh, Scotland, UK. The conference will be held in partnership with the National e-Science Centre and supported by the Coalition for Networked Information (CNI).

We are sure there are plenty of you out there with great ideas to share with colleagues; still time to get writing!

Tranche

Tranche sounds interesting. This possibly over-ambitious sound-bite from its web page:

"This project's goal is to solve the problems commonly associated with sharing scientific data, letting you and your collaborators focus on using the data."

In effect it is an encrypted, highly distributed file sharing system, independent of any central authority, and suitable for "any size of file", but maybe there are still some sharing problems left for the scientists (;-)?

The Science Commons blog reports:

"The National Cancer Institute will soon be using Tranche to store and share mouse proteomic data from its Mouse Proteomic Technologies Initiative (MPTI). Tranche, a free and open source file sharing tool for scientific data, was one of the earliest testers of CC0. Many thanks to Tranche for providing us with such valuable early feedback on CC0."

Tranche uploads are known as projects, and there are apparently 5399 of them so far. The largest number of replicas appears to be 3, and 44 projects are reported as having missing chunks.

National Statistics no joke

Today the UK's new independent National Statistics Authority began work, replacing the Office for National Statistics. This is clearly an important agency; in the Today Programme this morning they were claiming that they would make a clear distinction between the statistics and political interpretation. I thought it was worth a look at their web site.

Two things initially discouraging: first, a Google search for National Statistics Agency [sic] produces a web page for the old ONS at http://www.statistics.gov.uk/default.asp; this is ALMOST but not quite the same URL as the new authority at http://www.statistics.gov.uk/... but what about the apparently similar but possibly different http://www.statisticsauthority.gov.uk/? Well, teething troubles no doubt. Second there's a prominent link on the (first of the above) home page that "ONS independence comes into effect on 1 April", and the link is broken. More teething...

A quick explore led me to the UK snapshot, with lots of interesting web pages summarising data. As an example, there is a page Headed "Acid Rain" under the "Environment" section; you get a graph and a few paragraphs of text, eg "Emissions of chemicals that can cause acid rain fell by 53.8 per cent between 1990 and 2005, from 6.9 million tonnes to 3.2 million tonnes." In fact, the page doesn't tell us whether rain became less acidic during this period, but again that's a quibble.

But I was looking for data, not summaries. I found some under "Time Series data", but had to go through a complicated sequence of selections to find an actual dataset. I selected Share Ownership, then "Total market value by sector of beneficial owner: end-2006", then "DEYQ, SRS: Ungrossed: Total Market Value:Individuals" before I got a download button. The 3 download options were:

• View on Screen
• Download CSV
• Download Navidata

Where Navidata is a tool they make available. View on screen gives me a nicely formatted web page displaying a two-column table. Downloading the CSV gave me a cryptically named CSV file, which looks as follows (Crown Copyright, from the national Statistics Authority, Reproduced under the terms of the Click-Use Licence):

,"DEYQ",
" 1998",154.8,
" 1999",163.3,
" 2000",181.0,
" 2001",148.4,
" 2002",104.0,
" 2003",136.0,
" 2004",122.3,
" 2005",..,
" 2006",155.8,

"DEYQ","SRS: Ungrossed: Total Market Value:Individuals"
,"Not seasonally adjusted"
,"Updated on 8/ 6/2007"

All fine, I think... except I could not find any way to do this automatically. Maybe they have an API I haven't found, maybe they have plans not yet come to fruition. Anyway, some good stuff here, but perhaps room for improvement?

UK Repositories claiming to hold data

The OpenDOAR and ROAR services both present self-reported claims by repositories across the world about their contents, backed up by some harvested facts. I’m interested in those UK repositories that claim to hold data.

My first problem is that neither repository allows me simply to choose data. OpenDOAR allows me to search on “Datasets” (63 world-wide, 8 in the UK), while ROAR allows me to search for “Database/A&I Index” (24 world-wide, 6 in the UK). I thought the latter was a surprisingly “library science” classification, given the origins of ROAR. Not surprisingly, most repositories are in only one of the lists. Also not surprisingly given the origins of these services in the Open Access and OAI-PMH movements, there are many first class data repositories NOT listed here (UKDA and BADC, for example).

The UK repositories listed are:

OpenDOAR “Datasets”

• Bristol Repository of Scholarly Eprints (ROSE)
• DSpace @ Cambridge
• eCrystals - Southampton
• Edinburgh DataShare
• Edinburgh Research Archive (ERA)
• Leicester Research Archive (LRA)
• NDAD (National Digital Archive of Datasets)
• Nature Precedings
  

Looking at the OpenDOAR listing, and linking through to the repositories themselves, I find it very difficult to actually FIND the datasets in most cases. Looking at ERA, for example, there is no effective search for these datasets. Browsing soon leads to the realisation that the contents are papers, articles, theses, etc. Some of these may have datsets associated or within them, but they are a bit shy! The Edinburgh Datashare repository is a pilot, but does have a couple of real datasets. In a different way, Nature Precedings also is shy of disclosing its datasets.

The 3 that do have serious amounts of data are DSpace @ Cambridge, eCrystals and NDAD. DSpace @ Cambridge is dominated by the 100,000 ++ collection of chemical structures encoded in CML, but there are plenty of other datasets there, including some from Archaeology. Sadly, there are plenty of empty collections, and many collections where the last deposit was 2006 (I guess around when the funded project died). eCrystals is completely crystal structures, and has some very nice features; find a compound, and as you look perhaps rather bemused at the page, a Java object loads and there you have a rotatable image of the molecular structure before your eyes on the data page! NDAD also has many ex-Government datasets, some of them very large.

ROAR “Database/A&I Index”

• Higher Education Empirical Research Database (1642 records)
• NDAD - UK National Digital Archive of Datasets (66 records)
• ReOrient Knowledge Base
• Research Findings Register (1496 records)
• Southampton Crystal Structure Report Archive (165 records)
• The Linnean Collections (14275 records)

NDAD and eCrystals (under a slightly different name) appear again in this ROAR set. Of the others, HEER seems to be closed (it wanted a password for every page I found), and ReFer seems to have been withdrawn. ReOrient seems to be frozen and to present its data in the form of maps, while the Linnaean Collection seems to be images (which can, of course, be good data as well).

It’s a rather sad study! I do hope that the Open Repositories 2008 conference in Southampton over the next couple of days leads to an improvement. I can't get there, unfortunately, but I hope someone will report from it here. I particularly liked the idea of the developers challenges. Can we have some oriented to data, please?

European e-Science Digital Repository Consultation

Philip Lord wrote to tell me that he and and Alison Macdonald are conducting a study for the European Commission “Towards a European e-Infrastructure for e-Science Digital Repositories”, (e-SciDR) – see www.e-SciDR.eu. This is a short study to summarize the situation regarding repositories in Europe and to propose policies for the Commission for repository development in Europe. As part of the study process the Commission is hosting a public consultation through a questionnaire... The letter inviting participation follows:

"Dear Sir, Dear Madam,

May I invite you, as key stakeholders, to contribute to the development of a knowledge society and digital infrastructure in Europe, by taking part in the Commission’s online public consultation on e-Science Digital Repositories which is available at http://ec.europa.eu/yourvoice/ipm/forms/dispatch?form=eSciDR."

[NB Safari on the Mac appears not to work with this questionnaire, but Firefox does.]

"This consultation forms a key part of the e-SciDR study funded by the Commission into repositories holding digital data and publications for use in the sciences (in the widest sense encompassing disciplines from the humanities and social sciences to the life sciences).

Your answers will help identify needs, priorities and opportunities which the European Union, through the Commission, can help address and drive forward in the FP7 Capacity Programme and will provide an important input to developing future policy initiatives.

I would be grateful if you could respond to the consultation by no later than 30 July 2007.

All answers will be strictly confidential and anonymised.

If you would like to receive a summary of the consultation results, please tick the corresponding box on the questionnaire.

Best regards,

Mário Campolargo

Head of Unit GÉANT & e-Infrastructure"

Open Access everyone? Or not?

I occasionally look at the OpenDOAR service, which list information about repositories, and check out those which claim to include data (the term they use is datasets, although it is possible that “other” might also be applicable!). Last time I looked there were 7 listed in the UK which claimed to collect datasets; 4 of these were institutional repositories, one is the Southampton eCrystals archive, the 6th is the National Digital Archive of Datasets (NDAD), which collects government datasets on behalf of The National Archive, and the 7th is Nature Precedings.

I used to think that was a creative piece of listing by NDAD, since OpenDOAR was created for institutional repositories, wasn’t it? But as time has gone on, I’ve begun to think NDAD are right; the old definitions of repositories were much too tight, and NDAD and many other data archives ought to be considered as repositories, and listed in these kind of resources. On that basis perhaps AHDS should have been listed (although they might not bother now) and, I thought, the UK Data Archive, part of the Economic and Social Data Service, should also be listed. After all, they are repositories, and open access is a good thing, isn’t it?

At the UKDA’s 40th birthday celebrations yesterday, it was clear that Kevin Schurer (Director of UKDA) doesn’t quite share that view. The view expressed certainly was not a total rejection of Open Access in favour of a commercial approach. But he was certainly arguing for some barriers between some data and the users. In particular, while access to UKDA data is free for certain categories of users (certainly UK academics and probably some others), ALL users are required to register. Kevin made a strong case for this being an advantage; registration means that he can report to his funders who his users are (including how many of them might be independent or Government researchers as well as academic ones), and can also monitor which datasets they are using. He knows, for example, that the user base has grown from 25 or so after the first 5 years to 45,000 (I don't remember the exact figure, but of that order) after the first 40 years.

The same registration mechanism (and associated authentication and authorisation mechanisms) also allow them to apply greater access controls to more sensitive datasets, including the possibility that the user may have to sign and observe special licences. It’s worth remembering that much of the data they hold is about people, and some is extremely sensitive information, which has been provided (under “informed consent”) for certain specific purposes.

In a very different environment the day before, I met some scientists with very long-term experiments collecting data on insects. For perhaps different reasons, they were happy to make their data available to collaborators, but not openly available on the web. Too many risks of mis-interpretation, which then requires extra effort to refute, was one reason. No doubt an extra paper as co-author from a collaboration was another motive (not unreasonable).

Are these approaches Open? Are they consistent with the OECD Principles and Guidelines for Access to Research Data that I wrote about earlier? Let’s remember that the Openness Principle was carefully worded:

“Openness means access on equal terms for the international research community at the lowest possible cost, preferably at no more than the marginal cost of dissemination. Open access to research data from public funding should be easy, timely, user-friendly and preferably Internet-based.”

So it looks as though the approach is reasonably consistent, provided access is on “equal terms for the international research community”, by that definition. I suspect there are other definitions (who said “the nice thing about standards is there are so many to choose from”?) by which these approaches would fail (eg the Berlin Declaration).

What the UKDA approach might do is make certain kinds of comparative work, including automated data mining, more difficult. There was a plea with respect to the former, from an Australian speaker, for more sophisticated international cross-archive access management (code: Shibboleth-enable). But I suspect with respect to the latter (preventing data mining) Kevin might argue “very right and proper too”!

UKDA 40th Birthday: back to basics?

There are not many digital data management organisations that can claim 40 years of continuous service. This week the UK Data Archive (UKDA, which holds mainly social science datasets) celebrates 40 years since its founding in 1967, with a party yesterday in the House of Commons followed by a small workshop in the UKDA’s fancy new quarters at the University of Essex. The DCC would like to congratulate the UKDA, Director Kevin Schurer and his 6 predecessor Directors, and the UKDA staff, on achieving this milestone.

[Pause to suppress pangs of envy at such longevity. Sigh!]

There were many very interesting aspects to this workshop, but here I will focus on one particular contribution, during the closing sessions, from Myron Gutmann. Myron is Director of ICPSR, a roughly equivalent organisation in the US, based at Michigan. Myron said he wanted to argue for retaining the basics. A data archive, he said, should do (at least) 5 things:

• Appraise
• Curate
• Preserve
• Train
• Protect

This is not quite the language of OAIS, but perhaps closer to the language of archives. Appraise because everything an archive does is expensive, so it has a responsibility to select high quality resources appropriate to its mission (and as OAIS might suggest, its Designated Community). Curate because (apparently) even the best resources tend to need extensive work to make the usable (this work includes making the contextual and other metadata about the resource usable by non-insiders; we might perhaps describe this perhaps as preservation metadata, maybe as part of its representation information). Preserve for obvious reasons, to make resources available in the long term (although some people at the workshop seemed to be using the word in a special sense: institutional repositories are not about preservation, said someone, although I may have mis-quoted). Train, because these resources are specialist, and often need significant work and extra knowledge on the part of users. And protect, both as protection of the intellectual property in the resource, but more importantly protecting the subjects of the data, whose sensitive data must be made available only under appropriate conditions.

It’s a pretty good summary of the job of an archive, give or take a verb or two. Myron added something like “serve new user bases” and “innovate” in various ways, and it’s hard to argue with.

There were some sober reflections on sustainability at the workshop, partly relating to the difficult funding position for long-term longitudinal surveys, but partly reflecting the recent AHRC decisions. Paraphrasing Kevin Schurer, we never could or should take things for granted, but now we must be doubly persuasive, even if the UKDA's main funder ESRC regards ESDS (of which UKDA is a key part) as “a jewel in its crown”.