Showing posts with label Open formats. Show all posts
Showing posts with label Open formats. Show all posts

Friday, 13 February 2009

Open Office as a document migration on demand tool- again

We’ve seen suggestions in comments on this blog, and on other blogs, that code is better than specifications as representation information, and that well-used, running open source code is better than proprietary code. We’ve also had assertions that documents should be preserved in their original format, rather than migrated on ingest (I’ve some reservations on this in some cases for data, but as long as the original form is ALSO preserved, it’s fine).

The appropriate strategy for documents in obsolete formats would therefore seem to be to preserve in the original format and migrate on demand, from the original format to a current format, when an actual customer wants to use it. This process should always be left as late as possible, based on the possibility that the migration tool will improve and render the document better with later versions (and allowing the cost to be placed onto the user, not the archive, if appropriate). By the way, this exactly parallels the case in real archives; they don't translate their documents from old Norse to modern English each time the latter changes. If you want to read them, go learn old Norse, or hope that someone has earned a brownie point by translating it for publication somewhere...

I have suggested a couple of times that a plausible hypothesis for “office documents” (ie text documents, spreadsheets, simple drawings, presentations, simple databases), is that the OpenOffice.org should be the migration tool of choice. After all, it supposedly reads files in 180+ different file formats, it is open source, it is widely used, it is actively developed, and it can produce output in at least one internationally standardised format. I’ve noted already that it isn’t perfect; the Mac version, for instance, fails to open my problem PowerPoint version 4 files (to be fair, it doesn’t claim to). But perhaps it’s worth taking a look again at the range of formats it claims to deal with. All these figures relate to vanilla OpenOffice.org 3.0.0.0 (build 9358) for the Mac (the first fully native, up to date Mac version I've been able to lay my hands on).

So, a health warning: this rather long post goes into more detail on something I've covered before!

The single OpenOffice.org application opens 6 main classes of document: text document, spreadsheet, presentation, drawing, database and formula (the lists of file formats has moved to an appendix of the Getting Started guide). In each case, a majority (or close to it) of the supported formats are OpenOffice native, or its predecessors, plus a good selection of Microsoft Office formats (Office 6.0/95/97/XP/2000 and the XML versions from Office 2003 and Office 2007), which means that the large majority of documents for the past 10 years or so should be readable, when these formats have been the dominant office suites

The most well-known (and presumably widely-used) remaining supported word processing format is WordPerfect; not clear which versions (.wpd). Then there are some interesting ones: for example DocBook, and the Chinese-developed Uniform Office Format, the Korean Hangul WP 97, AportisDoc for the Palm, Pocket Word, and the Czech T602. Interesting that: significant investment to ensure these minority but presumably significant formats can be handled.

Similarly for the spreadsheets, as well as the various native and Microsoft formats, there is also support for two earlier significant players: Lotus 1-2-3 and Quattro Pro 6.0, also dBase (.dbf) and Data Interchange Format (DIF), not to mention CSV. It’s interesting that dBase is treated as a spreadsheet rather than a database; I wonder what the limitations are.

Presentations are more limited; apart from the basic OpenOffice and MS Office variants, they include Computer Graphics Metafile (CGM) as a presentation format but not a drawing format, which is a bit odd. PDF is also included; well, it does do presentations, and they seem to have some advantages over PowerPoint.

Graphics formats have always been popular in the open source community, so it’s not surprising that a wide range of formats is supported for graphics. Aside from several OpenOffice formats, these include a few surprises such as AutoCAD’s DXF Interchange Format, and Kodak PhotoCD, as well as a large range of usual suspects (GIF, PNG, TIFF, JPEG, many bit-mapped formats).

Finally the only database supported is the OpenOffice native database (remembering that dBase is apparently supported, as a spreadsheet, presumably with limitations). I tried to open a Microsoft Access database from a previous computer (Win95?), without success. Old databases do tend to be a bit of a problem; I have heard there are significant compatibility problems even between successive versions of Access. And Formula supports a couple of OpenOffice formats, plus MathML, which should be good for scientific use today.

So, for nothing, you get a migration tool that deals with a substantial proportion of current or recent documents. I don’t have enough experience yet to judge how effective it is. I did try a trivial round-trip test: opening a Microsoft Word 2004 for Mac document in OpenOffice, saving as native OpenOffice, then re-opening and saving as Word again, followed by a document compare in Word; it revealed very small layout differences in nested bullets (which resulted in pagination changes), and a few minor changes in styles. Not quite a fully reversible migration, but the result was a perfectly acceptable rendition of the original.

Now a migrate on demand tool is only useful in this role as long as (or if) the original file format is supported. If you are interested in older documents, from what one might call the baroque period of early personal and office documents (say from the invention of microcomputers for home use, through early “personal computers”, up to the big shakeouts of the mid to late 1990s), you will find OpenOffice rather less helpful as a migrate on demand tool. On one argument, this doesn’t matter much, as comparatively speaking such formats represent a tiny minority of surviving documents (unproven but pretty safe assertion!). However, this class of baroque period documents is starting to become important to archives (real archives, not collections of backups, or even digital preservation repositories), as they begin to collect them as part of the “papers” of eminent individuals. See for example, the Digital Lives project mentioned here before.

So, here are two proposals (for both of which, specifications as well as known working code would be useful!):
  • Funders, Foundations etc - please fund efforts to add input filters to OpenOffice for such older document formats, and
  • Computing Science departments - please set group assignments that would result in components of such filters being contributed to the OpenOffice effort.
Collectively, we might suggest the underlying effort here as an OpenOffice Legacy Files Project. Does anyone know how to set up such a project?

BTW after my last posting on this topic, a linkback led me to a post where Leslie Johnston mentioned Conversions Plus as having been a life-saver on several occasions. It’s a commercial tool, so maybe there are licence and survivability issues, but the list of formats it claims is impressive. In the Word Processing area alone, you get:
  • 3 versions of Ami Pro
  • 2 versions of AppleWorks
  • ClarisWorks 1.0 - 5.0
  • 3 versions of MacWrite
  • DCA-RFT
  • 3 versions of Multimate
  • Many versions pf Word, back to MS Word DOS 5.5
  • Several versions of MS Works
  • PerfectWorks
  • WordPerfect for DOS and Windows
  • WordPerfect Works
  • WordStar for DOS
  • Several versions of Lotus Word Pro
Is a tool like this a better bet than OpenOffice for migration on demand? In the longer term, I don’t think so, even if it might be more helpful in the short term. You’d have to be convinced that the company will still exist to supply it, and that it will still run on your then current hardware. It might, but the odds seem somewhat better for a very popular open source application like OpenOffice.

But in the end, one way or another, you pays your money and you place your bets!

Tuesday, 6 January 2009

Email discussion on the usefulness of file format specifications

This is a summary of an email exchange on the DCC Associates email list over a few days in late November, early December. I thought it was revealing of attitudes to preservation formats and to representation information (in the form of both specifications and running code), so I’ve summarised it here. Emails lists are great for promoting discussion, but threads tend to fracture off in various directions, so a summary can be useful. Quotes are reproduced with permission; my thanks to all those involved.

Steve Rankin from the DCC down in Rutherford Labs noticed and drew the list’s attention to the Microsoft pages relating to their binary formats, made available under a so-called "Microsoft Open Specification Promise”.

http://www.microsoft.com/interop/docs/OfficeBinaryFormats.mspx
and http://www.microsoft.com/interop/osp/default.mspx

Chris Puttick of Oxford Archaeology pointed out that the pages had been up for a while (since February 2008 at least). He made a couple of interesting points:
“I have it on excellent authority that the specifications are useful but incomplete […]; secondly that as is this not the first time MS have published such information only to take it down again later [so] anyone interested in them should download them as soon as possible. I have on slightly less excellent authority that a ‘promise’ as encased in the [Open Specification Promise] is specifically something in US law and may not have any validity outside of the US.”
Kevin Ashley from ULCC/NDAD agreed:
“It's my understanding - from those who have tried - that earlier specs that MS published failed exactly that test. It wasn't possible to use them to write software that dealt with all syntactic and semantic variations.

“It's a fairly fundamental test for network protocols that one can […] get two separate implementations to communicate with each other. The same is true of file formats, to my mind, and one can see the creating application and the reading application as equivalent to the two ends of a network connection, albeit not necessarily in real time.”
David Rosenthal from Stanford and LOCKSS injected some engineering reality from direct experience into the discussion. He has already released a longer blog post based on the discussion and his contribution; effectively he seemed to be aiming to demolish the argument for keeping specifications at all.
“Speaking as someone who has helped implement PostScript from the specifications, I can assure you that published specifications are always incomplete. There is no possibility of specifying formats as complex as CAD or Word so carefully that a clean-room implementation will be perfect. Indeed, there are always minor incompatibilities (sometimes called enhancements, and sometimes called bugs) between different versions of the same code. And there is no possibility of digital preservation efforts being able to afford the investment to do high-quality clean-room implementations of these complex formats. Look at the investment represented by the Open Office suite, for example.

“On the other hand, note that Open Office and other open source office suites in practice do an excellent job of rendering MS formats, and their code thus represents a very high quality specification for these formats. Code is the best representation for preservation metadata.”
Colin Neilson from DCC SCARP wondered what the implications of incomplete specifications were for the concept of Representation Information in OAIS (RepInfo is often associated in examples with specifications).

He wrote:
“I am interested in implications for areas (such as CAD software) where proprietary (secret sauce) formats are historically the norm. Is the legacy of digital working always preservable within an OAIS framework? […] Are there some limits in using an OAIS model if some "specifications" are inadequate or information is not available?”
and in a later message
“Do we need to have "access software" preserved (long term) if the other representation information is less complete in the case where standards for proprietary file formats (say like Microsoft word DOC format) are to a degree incomplete, less adequate or not available (perhaps more so in the case of older versions of file formats)?”
Personally I think one of the advantages of Open Office is that it is not just Access Software, but Open Source Access Software. This should give it much greater longevity. But of course, such alternatives don’t exist in many areas, including many of the CAD formats Colin is concerned about.

Alan Morris from Morris and Ward asked the obvious question:
“Who would even consider utilizing WORD as a preservation format?”
… and got a surprising answer, from Peter Murray-Rust from the eponymous Cambridge research group!
“I would, and I argued this in my plenary lecture at OpenRepositories08. Not surprisingly it generated considerable discussion, from both sides.

“First the disclaimer. I receive research funding (though not personal funding) from Microsoft Research. Some of you may wish to stop reading now! But I don't think it colours my judgment.

“My argument was not that Word2007 should be the only format, but that it should be used in conjunction with formats such as PDF. We have a considerable amount of work on [depositing] born-digital theses and we have recommended that theses should be captured in their original format (OOXML, ODT, LaTeX, etc.) as well as the PDF.

“I am a scientist (chemist) but generally interested in all forms of STM data (for example we collaborated in part of the KIM project mentioned a few emails ago). If you believe that preservation only applied to the holy "fulltext", stop reading now. However I think many readers would agree that much of the essential information in STM work (experiments, data, protocols, code, etc.) is lost in the process of publication and reposition. Very frequently, however, the original born-digital work contains semantic information which can be retrieved. For example OOXML and ODT allow nearly 100% of chemical information (molecular structures) to be retrieved (in certain circumstances), whereas PDF allows 0% by default. (It is possible, though extremely difficult and extremely lossy, to turn PDF primitives back into chemistry)

“Note that we also work on Open Office documents and have a JISC-sponsored collaboration with Peter Sefton [of the Australian Digital Futures Institute of USQ in Australia] on his excellent ICE system. We are exploring how easy it is to author chemistry directly into an ODT document and by implication into any compound semantic document (note that XML is the only practical way of holding semantics). […]”

“We've looked into using PDF for archiving chemistry and found that current usage makes this almost impossible. So we work with imperfect material.

“Note that Word2007 can emit OOXML that can be interpreted with Open Source tools. The conversion is not 100%, but whatever is? […]”

“I wonder whether the all the detractors of OOXML have looked at it in detail. Yes, it is probably impossible to recreate all the minutiae of typesetting, but it preserves much of the embedded information that less semantic formats (PDF and even LaTeX) do not. If I have no commercial software and someone gives me a PDF of chemistry and someone else gives me OOXML I'd choose the OOXML. HTML is, in many cases, a better format than PDF.

“So my suggestion is simple. Use more than one document format. After all do we really know what future generations want from the preservation process. It costs almost nothing as we are going to have to address compound documents and packaging anyway.”
An anonymous contributor suggested that the appropriate course was to structure AIPs to contain both original source format and the preservation format. In the future, he asserted, better tools may exist to take the original source format and render a more completely accessible preservation format, particularly bearing in mind scientific notation.

Finally, Geoffrey Brown from Indiana also argued in favour of keeping the original (and against NARA policy):
“The Bush administration as well as various companies managed to embarrass themselves with inadvertently leaked information in the form of edit histories in word documents. Migration will likely (who knows ?) discard such information unless special care is taken in developing migration tools.

“I am uncomfortable with the assumption that we can abandon the original documents as NARA seems to be doing by requiring(?) agencies to submit documents in PDF). The edit histories are part of the historical record; however, it's safe to say that most patrons will be satisfied with the migrated document.

“Digital repositories have an obligation to figure out how to preserve access to documents in any format and not use format as a gatekeeper.”
So… running code is better than specs as representation information, and Open Source running code is better than proprietary running code. And, even if you migrate on ingest, keep BOTH the rich format and a desiccated format (like PDF/A). It won’t cost you much and may win you some silent thanks from your eventual users!

Tuesday, 1 July 2008

Responses to RAW versus TIFF: image-related

This post summarises image-related responses to the “RAW versus TIFF” post made originally by Dave Thompson of the Wellcome Library. The key elements of Dave’s post were whether we should be archiving images using RAW (which contained more camera and exposure-related data) or TIFF (which was more standardized, likely to be accessible for longer, and had more available utilities). A subsidiary question on whether we should archive both is greatly affected by cost; responses related to this element are summarized in a separate post. As I mentioned before, these responses came on the semi-closed DCC-Associates email list. [I forgot to mention that Maureen Pennock re-posted Dave's question to this blog with some remarks of her own, and we also received a useful response which you can view there.]

One comment [on the DCC-Associates list] made by many was that JPEG2000 should be considered. Alan Morris of Morris and Ward asked if it was an ISO standard, to which Shuichi Iwata (former President of CODATA and Professor at the University of Tokyo) responded:
“Yes, it is. Organized by Joint Photographic Experts Group of ISO. It is used widely for compression of images at different levels.

http://www.jpeg.org/”
James Reid of EDINA, University of Edinburgh, mentioned that “the DPC report on JPEG 2000 has relevance”. He also expressed concern:
“RAW images tend to have proprietary and sensor specific aspects that makes them less than ideal for a generic preservation format - they are of course lossless but there would be an implicit overhead in deriving sensor specific options to make preservation possible (and these may not be publicly available due to proprietary interests)...”
Larry Murray of the Public Record Office of Northern Ireland mentioned how essential RAW is to him as a keen amateur photographer, allowing him to manipulate images. He added:
“James is quite correct in his comment but personally I think the decision to save the RAW images is one of available resources. The RAW images give the data owner the opportunity to control so many of the characteristics of the original image without data loss or integrity and the ability to bulk process said changes means serious consideration must be given to their value.

For other reference on Open RAW formats see http://www.openraw.org/”
Chris Puttick, CIO of Oxford Archaeology asked:
“RAW is the original - would you store a scan of a document as the primary method of preserving the document? I think RAW + [a common image format] is the best option. It does of course require greater storage space, but storage is cheap and getting cheaper every day.

Software such as DigiKam has libraries that read pretty much all the RAW formats including rarer ones such as that produced by the Foveon sensor (Sigma, Polaroid). Any additional documentation needed for RAW should be easily prised out of the sensor manufacturers if the right pressure is brought to bear...”
Stuart Gordon of Compact Services (Solutions) Ltd agrees:
“In order to ensure that the images are viewable for posterity we must have the primary archive images in a universal lossless format, currently TIFF however as standards evolve this will inevitably change.

The camera RAW formats are proprietary and are therefore not suitable as the primary archival format; however, if space is available then do archive the RAW as well.

As with all questions it depends on what your goals are and in our case it is ensuring that the archival images are stored in a format that will ensure they are accessible for posterity.”
Paul Wheatley, Digital Preservation Manager at the British Library commented:
“As well as the extremes of the camera/manufacturer specific formats and then at the other end of the spectrum, formats like JPEG2000 and TIFF, we also have a number of emerging RAW file “standards”, of which Open RAW is one. I’ve not seen any independent technology watch work to assess all these options in a reasoned way and make recommendations. Perhaps this is a candidate for a DPC Tech Watch piece? If so, I would like to see authorship from a panel of experts to ensure a balanced view. Both an independent viewpoint and careful analysis of the objectives behind the use of different file formats were a little lacking in the last DPC report. As Larry is hinting at here, I’m sure there are different answers depending on exactly what you want to achieve from the storage and preservation. These aims need to be carefully captured as well as the recommendations to which they relate.”
[UPDATE: Rob Berentsen, of Deloitte Consulting B.V.] brings us back to preservation concerns:
“The discussion whether to use Raw, Tiff or both is one that depends on what the (future) user demands will be and the current preservation demands are. As you can see, Larry prefers Raw so he can use the image in an optimal manner as photographer/manipulator. James however might have a different view on the importance of certain aspects of images and might have different future demands in mind.

Looking at the OAIS model, the most raw image should leave all roads open to create a TIFF image (or others). This TIFF might be more accessible, and even the non-lossless format JPG might be good enough for some people to have [the] image easily accessible over the internet.

To cope with this, the OAIS's archival information package (AIP) offers room for more then just one 'manifestation'. You can see one (the Raw version) as the preservation format, and others are possible future preservation formats or accessible formats. The accessible formats are commonly the ones end-users can get (DIP-package).”
Marc Fresko of Serco suggested there are two questions here:
“One is the question addressed by respondents so far, namely what to do in principle. The answer is clear: best practice is to keep both original and preservation formats.

The second question is "what to do in a specific instance". It seems to me that this has not yet been addressed fully. This second question arises only if there is a problem (cost) associated with the best practice approach. If this is the case, then you have to look at the specifics of the situation, not just the general principles. Most importantly:
  • what will you lose by going from RAW to TIFF (probably nothing that matters for many archival applications, but you will lose something which would matter in other applications)?
  • over how long do you need to preserve access to the images, compared to the likely lifetime of the software you'll need to access the specific RAW format?
You can easily see that different answers to these two questions will lead you to prefer different formats.”
Marc’s comment brings us back to costs, and there is a further series of emails on this topic which I will include in another blog post.

Colin Neilson of the DCC SCARP Project took the discussion towards science data:
“If you generalise and treat your camera as a "Digital Image Acquisition System" :-) then do you keep the original data from the instrument (camera) or would you prefer only to keep the derived data in some form of standardised output?

I suspect in a scientific context you would want to keep both. The problem is that the steps taken in deriving the data are not standardised or documented and may not be repeatable at a future time e.g. when the particular make of instrument (camera) becomes obsolete. The software & algorithms for processing captured image data to standard format are embedded in the camera and function to adjust the data produced from the proprietary image processor (e.g. Canon's DIGIC chip). Some of the "improvements" produced in image quality may not be welcome in a scientific context (e.g. may have introduced artefacts). Some makes and models of camera offer the option save both the "raw" image and the derived image at the cost of memory and processing time.

You can seem some of the issues developed when looking at open microscopy environments where the issue is image capture with effective data management system for multidimensional images including spatial, temporal, spectral and lifetime dimensions. Slide 13 of this presentation by Kevin Eliceiri sets out the current situation. This is quite an exciting area!

Of course this may all be a bit academic (not to say "over the top") if your main concern is to take the best steps to protect you holiday pictures or personal photography efforts. I have found this article useful in an amateur photography context.”
Finally (for this post), Riccardo Ferrante, who is IT Archivist & Electronic Records Program Director for the Smithsonian Institution Archives agreed with Colin:
“We have scientists here that consider the raw data absolutely critical and whose integrity and authenticity must be preserved (archival package). However, the usefulness of the data comes only when the raw data is filtered. The filtered data (derivation) is also considered an archival package by these scientists since it is reused repeatedly in the future, cited in publications, etc. and therefore needs its authenticity, integrity, and provenance to be preserved. To use OAIS-Reference Model language - the DIP generated from the original AIP (raw data) is itself a derived AIP.

In contrast, we have artists who deal solely in photographs and do not consider the raw image to be an object of record, but more like an ingredient for the object that is finalized after post-processing occurs, i.e. after it is published.

To me, it seems the common point is what the creator considers the object of record and the purpose of the raw data in light of that - this should be a major factor in determining the object of record from a recordkeeping viewpoint.”
I thought this was an extremely useful conversation!

Thursday, 20 March 2008

Migration on Request: OpenOffice as a platform?

Following on from my previous post relating to legacy formats, I was thinking again about the problems of dealing with documents in those formats. For some, the answer lies in emulation and perpetual licences of those original software packages, but for me that just doesn't cut the mustard. I won't have access to those packages, but I might want access to the documents. Some of them for example, might be the PowerPoint 4 presentations created on a predecessor to the Macintosh that I use now, but which are un-readable with my current PowerPoint software (I CAN get at them by copying them to a colleague's Windows machine; her version of PowerPoint has input filters unavailable on my Mac).

So I want some form of migration. In the example above, this is known as "Save as"!

However, I know that every time I do migration I introduce some sort of errors. So if I migrate from those PowerPoint 4 files to today's PowerPoint, and then from today's to tomorrow's PowerPoint, and then from tomorrow's to the next great thing, I will introduce cumulative errors whose impact I will only be able to assess at some horribly cringe-making moment, like in the middle of a presentation using a host's machine. So the best way to do migration is to start from the original file and migrate to today's version. Always. It's nuts for Microsoft to drop old file format support from its software (at least from this pint of view).

This approach of migrating from original version to today's version is called Migration on Request, and was described in a paper by Mellor, Wheatley and Sergeant back in 2002 (I referred to it earlier), but the idea hasn't caught on much. They had some other great ideas, like writing the migration tool in a specially portable version of C with all the nasty bits removed, called C--.

I have wondered from time to time however, for that class of documents we call Office Documents (word processing, spreadsheets, presentations), whether tacking onto an open source project which has a strong developer community might be a better approach. Something like OpenOffice. I'm not sure how many file formats this already supports (always growing, I guess, but Chapter 3 of the "Getting Started" documentation lists the following:
Microsoft Word 6.0/95/97/2000/XP) (.doc and .dot)
Microsoft Word 2003 XML (.xml)
Microsoft WinWord 5 (.doc)
StarWriter formats (.sdw, .sgl, and .vor)
AportisDoc (Palm) (.pdb)
Pocket Word (.psw)
WordPerfect Document (.wpd)
WPS 2000/Office 1.0 (.wps)
DocBook (.xml)
Ichitaro 8/9/10/11 (.jtd and .jtt)
Hangul WP 97 (.hwp)
.rtf, .txt, and .csv
... which is not a bad list (just the word processing bit, too)... and maybe extended in more up to date versions. For interest their FAQs have a question "Why does OpenOffice not support the file format my application uses?"
"There may be several reasons, for example:
  • The file formats may not be open and available.
  • There may not be enough developers available to do the work (either paid or volunteer).
  • There may not be enough interest in it.
  • There may be reasonable, available workarounds."
Making legacy file formats more open was the subject of my previous post, and I guess we have to wait and see. But there are plenty of legacy word processing formats not on that list (Samna, for example, later to evolve into Lotus Word Pro, as well as formats for obsolete computers like the Atari, such as the German word processor SIGNUM, supposedly very good for mathematical formulae). What about earlier version of MS Word? Wikipedia lists a bunch of word processors; there must be many documents in obscure locations in these formats.

With a concerted effort, we could gradually build OpenOffice input filters for these obsolete document types, thus brining them into the preservable digital world. And this is an effort that could bring in that extraordinary community of enthusiasts who do so much to build document converters and other kinds of software, so much ignored by the digital preservation community!

Legacy document formats

On the O'Reill XML blog, which I always read with interest (particularly in relation to the shenanigins over OOXML and ODF standardisation), Rick Jelliffe writes An Open Letter to Microsoft, IBM/Lotus, Corel and others on Lodging Old File Formats with ISO. He points out that
"Corporations who were market leaders in the 1980s and 1990s for PC applications have a responsibility to make sure that documentation on their old formats are not lost. Especially for document formats before 1990, the benefits of the format as some kind of IP-embodying revenue generator will have lapsed now in 2008. However the responsibility for archiving remains.

"So I call on companies in this situation, in particular Microsoft, IBM/Lotus, Corel, Computer Associates, Fujitsu, Philips, as well as the current owners of past names such as Wang, and so on, to submit your legacy binary format documentation for documents (particularly home and office documents) and media, to ISO/IEC JTC1 for acceptance as Technical Specifications.[...] Handing over the documentation to ISO care can shift the responsibility for archiving and making available old documentation from individual companies, provide good public relations, and allow old projects to be tidied up and closed."
This is in principle a Good Idea. However, ISO documents are not Open Access; the specifications Rick refers to would benefit greatly from being Open. They would form vitally important parts of our effort to preserve digital documents. Instead of being deposited in ISO, they should be regarded as part of Representation Information for those file types, and deposited in a variety (more than one, for safety's sake) of services such as PRONOM at The National Archive in the UK, the proposed Harvard/Mellon Global Digital Format Registry, the Library of Congress Digital Preservation activity or the DCC's own Registry/Repository of Representation Information.

Tuesday, 17 July 2007

The National Archives and Microsoft join forces...

On 4 July, The National Archives and Microsoft announced a Memorandum of Understanding "ensuring preservation of the nation´s digital records from the past, present and into the future". Partly this relates to the standardisation of the Office Open XML format for Microsoft's products (I note that O'Reilly appear strongly in favour of this activity, seeing no conflict with the standardisation of Open Document Format). I had thought, through the rumour mill, that MS had provided (or was to provide) the specifications of obsolete file formats. This would fit with TNA's PRONOM file format registry, and their Seamless Flow programme. However, the key paragraph instead seems to be:
"Today´s announcement sees Microsoft provide The National Archives with access to previous versions of Microsoft´s Windows operating systems and Office applications powered by Microsoft Virtual PC 2007. Virtual PC 2007 enables people to run multiple operating systems at the same time on the same computer. This allows The National Archives to configure any combination of Windows and Office from one PC, thereby allowing access to practically any document based on legacy Microsoft file formats. It is estimated that The National Archives will have to manage many terabytes of data in these formats."
This sounds as if TNA must keep licensed versions of all older MS products, but then can run them under this emulation mode. This is a LOT better than nothing, but not as good as open access to the old file formats, with the ability to build additional tools that implies. But maybe there's more than is apparent in the press release?

Should data people care? Apart from the documentation and other information that many will have stored in obsolete proprietary formats, it turns out that many store their data that way as well. The excellent JISC-funded StORe project did a survey of several disciplines, and got over 350 responses. Although one respondent said “God preserve us from idiots who archive data in proprietary commercial formats (Excel spreadsheets and MS-word documents)!”, 220 said they kept source data in spreadsheets and the same number kept them in word processed documents. These were the largest categories except images (228)!

Let's hope they are keeping them somewhere else as well...