Tuesday, 1 April 2008

National Statistics no joke

Today the UK's new independent National Statistics Authority began work, replacing the Office for National Statistics. This is clearly an important agency; in the Today Programme this morning they were claiming that they would make a clear distinction between the statistics and political interpretation. I thought it was worth a look at their web site.

Two things initially discouraging: first, a Google search for National Statistics Agency [sic] produces a web page for the old ONS at http://www.statistics.gov.uk/default.asp; this is ALMOST but not quite the same URL as the new authority at http://www.statistics.gov.uk/... but what about the apparently similar but possibly different http://www.statisticsauthority.gov.uk/? Well, teething troubles no doubt. Second there's a prominent link on the (first of the above) home page that "ONS independence comes into effect on 1 April", and the link is broken. More teething...

A quick explore led me to the UK snapshot, with lots of interesting web pages summarising data. As an example, there is a page Headed "Acid Rain" under the "Environment" section; you get a graph and a few paragraphs of text, eg "Emissions of chemicals that can cause acid rain fell by 53.8 per cent between 1990 and 2005, from 6.9 million tonnes to 3.2 million tonnes." In fact, the page doesn't tell us whether rain became less acidic during this period, but again that's a quibble.

But I was looking for data, not summaries. I found some under "Time Series data", but had to go through a complicated sequence of selections to find an actual dataset. I selected Share Ownership, then "Total market value by sector of beneficial owner: end-2006", then "DEYQ, SRS: Ungrossed: Total Market Value:Individuals" before I got a download button. The 3 download options were:
  • View on Screen
  • Download CSV
  • Download Navidata
Where Navidata is a tool they make available. View on screen gives me a nicely formatted web page displaying a two-column table. Downloading the CSV gave me a cryptically named CSV file, which looks as follows (Crown Copyright, from the national Statistics Authority, Reproduced under the terms of the Click-Use Licence):
" 1998",154.8,
" 1999",163.3,
" 2000",181.0,
" 2001",148.4,
" 2002",104.0,
" 2003",136.0,
" 2004",122.3,
" 2005",..,
" 2006",155.8,

"DEYQ","SRS: Ungrossed: Total Market Value:Individuals"
,"Not seasonally adjusted"
,"Updated on 8/ 6/2007"
All fine, I think... except I could not find any way to do this automatically. Maybe they have an API I haven't found, maybe they have plans not yet come to fruition. Anyway, some good stuff here, but perhaps room for improvement?


  1. I know via Paribus (former mpow) that the ONS are working on an API to this data. They're slipping around on the SOAP at the moment, but hopefully they can be persuaded down the RESTful road.

    Paribus are also pushing for the click use license to be advertised at the point of download.

  2. Thanks Jim. Yeah, RESTful would be nice (or so I am led to believe, I'm not a programmer any more!).

    I thought afterwards that I should have mentioned the click through licence a bit more. It was a nightmare! Each page contained an enormous identical graphic, so I had to page down every page to find the bit to fill in. Although I read through everything very caerfully, the terms etc were very ambiguous, and a whole pile of stuff was specified by what it is not. Again, not very API-like! Grrrr...


Please note that this blog has a Creative Commons Attribution licence, and that by posting a comment you agree to your comment being published under this licence. You must be registered to comment, but I'm turning off moderation as an experiment.