Managing and sharing data

I really like the UK Data Archive publication "Managing and Sharing Data: a best practice guide for researchers". It's been available in print form for a while, although I did hear they had run out and were revising it. Meanwhile, you can get the PDF version from their web site. It is complemented by sections of their web site which reflect, and sometimes expand on, the sections in the Guide. The sections are:

Sharing data - why and how?
Consent, confidentiality and ethics
Copyright
Data documentation and metadata
Data formats and software
Data storage, back-up, and security

This is an excellent resource!

Arts and Humanities Data Service... next steps?

In an earlier post, I mentioned the decision by the AHRC (and later JISC) to cease funding the
AHDS from March 2008. Since then the AHRC have re-affirmed their decision. On 28 June, the Future Histories of the Moving Image Research Network made public an open letter to the AHRC, to no avail it would appear.

In her response to the announcements, Sheila Anderson (Director of AHDS) wrote:

"In the meantime, and at least until 31st March 2008, the AHDS will continue to give advice and guidance on all matters relating to the creation of digital content arising from or supporting research, teaching and learning across the arts and humanities, including technical and metadata standards and project management. If you have a data creation project, please do not hesitate to contact us for advice.

The AHDS will continue to work with those creating important digital resources to advise on the best methods for keeping these valuable resources available and accessible for the long-term in a form that encourages their further use for answering new research questions, and their use in teaching and learning. This advice will include exploring with content creators and owners suitable repositories in which they might deposit their materials for long term curation and preservation, and how to ensure that their materials can continue to be discovered and used by the wider community. If you are currently in negotiation with the AHDS to deposit your digital collection, please continue to work with us to ensure the future sustainability and accessibility of your resource.

The AHDS will continue to make available its rich collection of digital content for use in research, teaching and learning, and to preserve those collections in its care. The AHDS intends to discuss with the JISC and the AHRC the long term future of these collections beyond April 2008 with the intention of securing their continued preservation and availability."

This is a great expression of commitment, and deserves our support. However, the lack of long-term funding must raise questions of sustainability.

Explicit in the AHRC's decision was the view that the community is mature enough to manage its own resources. There is doubt in many people's minds about this, but we are effectively stuck with it. So what are the implications? There are implications both for existing collections and for future arts and humanities resources. I would like to spend a few paragraphs thinking about the existing collections.

AHDS is not monolithic; it is comprised of several separate services (I suspect in what follows I may be using historical rather than current names). We already know that AHRC is privileging the Archaeology Data Service (ADS), which will continue to receive some funding (and which has a diverse funding base), so their resources are presumably safe. The History Data Service (HDS) resources are embedded within the UK Data Archive (which has recently received an additional 5 years funding from ESRC); it would presumably cost more to de-accession those resources than to keep preserving them and making them available, so even if HDS can take no more resources, the existing ones should be safe. Literature, Language and Linguistics is closely related to the Oxford Text Archive; I imagine the same kinds of arguments would apply there.

I have heard suggestions that Kings College London might continue to support the AHDS Executive for a period, and it appears there are some discussions with JISC about some kind of support "to ensure the expertise and achievements of the AHDS are not lost to the community".

That leaves Performing Arts and Visual Arts; I can't even surmise what their future might be, since I don't know enough about their funding and local environment.

I appreciate that it's still early days, and no doubt crucial discussions are going on behind the scenes. But if any part of AHDS resources are in danger of loss, the resource owners need to consider plans to deal with those resources in the future. This will take some time, particularly since for more complex resources, it is clear that existing repositories are generally NOT yet adequate for purpose. I guess the picture itself will be complex; I can think of at least these categories:

• Some resources still exist outside of AHDS, and no action may be needed.
• Some resources will not be felt worth re-homing.
• Some resources can be re-homed in the time and funding institutions have available.
• Some resources should be re-homed, provided that time and funding are provided by some external source (this might be for developments on an institutional repository; it might be for work on the resource to fit a new non-AHDS environment).

I tried to find out what resources were deposited by the University of Edinburgh. It's not an easy search, but ignoring a bunch of stuff where the University was only peripherally mentioned, I came up with:

1. HOTBED (Handing on Tradition By Electronic Dissemination) pa-1028-1
2. Lemba Archaeological Project arch-279-1
3. Gateway to the Archives of Scottish Higher Education (GASHE) exec-1003-1
4. Avant-Garde/Neo-Avant-Garde Bibliographic Research Database lll-2503-1
5. Survey of Scottish Witchcraft, 1563-1736 hist-4667-1
6. National Sample from the 1851 Census of Great Britain hist-1316-1

HOTBED appears on further research to be a RSAMD project that ended in 2004; however, a Google search produced a web site (http://www.hotbed.ac.uk/) that did not appear to be functional this afternoon. GASHE still exists as a working resource, so far as I can tell, and anyway is run from Glasgow. Lemba is archaeology, and so maybe safe. The Avant-Garde bibliography still exists separately. The two history resources seem safe.

Are we OK? Is there more? Who knows! I think we need much better tools to tell what is "at risk", so that plans can start being made. Of course, this could be happening, maybe I'm just not "in the loop".

Will AHRC consider bids for funding transitional work? I certainly hope so, although I don't know how this might be done. JISC is (I believe) planning one last round of its Capital Programme. Will they include provisions to enhance repositories so as to take these more complex resources? I certainly hope so!

Open Access everyone? Or not?

I occasionally look at the OpenDOAR service, which list information about repositories, and check out those which claim to include data (the term they use is datasets, although it is possible that “other” might also be applicable!). Last time I looked there were 7 listed in the UK which claimed to collect datasets; 4 of these were institutional repositories, one is the Southampton eCrystals archive, the 6th is the National Digital Archive of Datasets (NDAD), which collects government datasets on behalf of The National Archive, and the 7th is Nature Precedings.

I used to think that was a creative piece of listing by NDAD, since OpenDOAR was created for institutional repositories, wasn’t it? But as time has gone on, I’ve begun to think NDAD are right; the old definitions of repositories were much too tight, and NDAD and many other data archives ought to be considered as repositories, and listed in these kind of resources. On that basis perhaps AHDS should have been listed (although they might not bother now) and, I thought, the UK Data Archive, part of the Economic and Social Data Service, should also be listed. After all, they are repositories, and open access is a good thing, isn’t it?

At the UKDA’s 40th birthday celebrations yesterday, it was clear that Kevin Schurer (Director of UKDA) doesn’t quite share that view. The view expressed certainly was not a total rejection of Open Access in favour of a commercial approach. But he was certainly arguing for some barriers between some data and the users. In particular, while access to UKDA data is free for certain categories of users (certainly UK academics and probably some others), ALL users are required to register. Kevin made a strong case for this being an advantage; registration means that he can report to his funders who his users are (including how many of them might be independent or Government researchers as well as academic ones), and can also monitor which datasets they are using. He knows, for example, that the user base has grown from 25 or so after the first 5 years to 45,000 (I don't remember the exact figure, but of that order) after the first 40 years.

The same registration mechanism (and associated authentication and authorisation mechanisms) also allow them to apply greater access controls to more sensitive datasets, including the possibility that the user may have to sign and observe special licences. It’s worth remembering that much of the data they hold is about people, and some is extremely sensitive information, which has been provided (under “informed consent”) for certain specific purposes.

In a very different environment the day before, I met some scientists with very long-term experiments collecting data on insects. For perhaps different reasons, they were happy to make their data available to collaborators, but not openly available on the web. Too many risks of mis-interpretation, which then requires extra effort to refute, was one reason. No doubt an extra paper as co-author from a collaboration was another motive (not unreasonable).

Are these approaches Open? Are they consistent with the OECD Principles and Guidelines for Access to Research Data that I wrote about earlier? Let’s remember that the Openness Principle was carefully worded:

“Openness means access on equal terms for the international research community at the lowest possible cost, preferably at no more than the marginal cost of dissemination. Open access to research data from public funding should be easy, timely, user-friendly and preferably Internet-based.”

So it looks as though the approach is reasonably consistent, provided access is on “equal terms for the international research community”, by that definition. I suspect there are other definitions (who said “the nice thing about standards is there are so many to choose from”?) by which these approaches would fail (eg the Berlin Declaration).

What the UKDA approach might do is make certain kinds of comparative work, including automated data mining, more difficult. There was a plea with respect to the former, from an Australian speaker, for more sophisticated international cross-archive access management (code: Shibboleth-enable). But I suspect with respect to the latter (preventing data mining) Kevin might argue “very right and proper too”!

UKDA 40th Birthday: back to basics?

There are not many digital data management organisations that can claim 40 years of continuous service. This week the UK Data Archive (UKDA, which holds mainly social science datasets) celebrates 40 years since its founding in 1967, with a party yesterday in the House of Commons followed by a small workshop in the UKDA’s fancy new quarters at the University of Essex. The DCC would like to congratulate the UKDA, Director Kevin Schurer and his 6 predecessor Directors, and the UKDA staff, on achieving this milestone.

[Pause to suppress pangs of envy at such longevity. Sigh!]

There were many very interesting aspects to this workshop, but here I will focus on one particular contribution, during the closing sessions, from Myron Gutmann. Myron is Director of ICPSR, a roughly equivalent organisation in the US, based at Michigan. Myron said he wanted to argue for retaining the basics. A data archive, he said, should do (at least) 5 things:

• Appraise
• Curate
• Preserve
• Train
• Protect

This is not quite the language of OAIS, but perhaps closer to the language of archives. Appraise because everything an archive does is expensive, so it has a responsibility to select high quality resources appropriate to its mission (and as OAIS might suggest, its Designated Community). Curate because (apparently) even the best resources tend to need extensive work to make the usable (this work includes making the contextual and other metadata about the resource usable by non-insiders; we might perhaps describe this perhaps as preservation metadata, maybe as part of its representation information). Preserve for obvious reasons, to make resources available in the long term (although some people at the workshop seemed to be using the word in a special sense: institutional repositories are not about preservation, said someone, although I may have mis-quoted). Train, because these resources are specialist, and often need significant work and extra knowledge on the part of users. And protect, both as protection of the intellectual property in the resource, but more importantly protecting the subjects of the data, whose sensitive data must be made available only under appropriate conditions.

It’s a pretty good summary of the job of an archive, give or take a verb or two. Myron added something like “serve new user bases” and “innovate” in various ways, and it’s hard to argue with.

There were some sober reflections on sustainability at the workshop, partly relating to the difficult funding position for long-term longitudinal surveys, but partly reflecting the recent AHRC decisions. Paraphrasing Kevin Schurer, we never could or should take things for granted, but now we must be doubly persuasive, even if the UKDA's main funder ESRC regards ESDS (of which UKDA is a key part) as “a jewel in its crown”.