Bathing Water Quality visualisation

Using the Environment Agency's bathing water quality data provided by DEFRA and UK Location, I've whipped up a very quick visualisation of my own - showing off indicators for faecal bacteria at the bathing sites.

The data source:

http://environment.data.gov.uk/doc/bathing-water.html?_view=basic&_properties=samplingPoint.lat,samplingPoint.long,latestSampleAssessment.faecalColiformCount,latestSampleAssessment.faecalStreptococciCount,latestSampleAssessment.totalColiformCount&_page=0&_sort=-latestSampleAssessment.faecalStreptococciCount

Notable parameters I used:

• view = "basic" view
• properties = lat, long, total coliform count, faecal coliform count, faecal streptococci count
• page size = 500
• sort = by faecal streptococci count (descending)

The steps to create the visualisation:

• Realised I was quite interested in seeing where the bathing sites were that contained the most faecal bacteria (Blackpool area by the looks of it)
• Tailored my own API call using instructions from the API documentation.
• Tried using Yahoo Pipes and other connecting API thingys to see what I could do within about 5-10 minutes.
• Decided to use Googles Fusion Tables as it can create maps from spreadsheet data
• The Enviroment API offers several formats, so I just changed my API call to include ".csv" intsead of ".html".
• Then, after spending the last half year developing the LinkedGov extension for Google Refine - I immediately thought of it as the first go-to tool to shape the data and make it fit for importing into some sort of mapping API.
• I used Google Refine's faceting & number range features to decide how to split the bacteria counts into low, medium and high.
• I exported the data from Refine as CSV to my computer.
• I uploaded the CSV into Fusion Tables - and all the hard work was done for me!
• Time taken = 15 minutes (I have experience with the Linked Data API - otherwise it would have taken me a little while longer to tailor the API call I wanted)

Please note: I have manually adjusted the banding points for the levels of bacteria so the visualisation showed a visually pleasing number of red, yellow and green markers. While a site may have a red marker - it could actually be of quite high water quality.

Total Coliform Count

Low-temperature electron micrograph of a cluster of E. coli bacteria, magnified 10,000 times. Each individual bacterium is oblong shaped. (Photo credit: Wikipedia)

Faecal Coliform Count

Faecal Streptococci Count

Gram-stained smear of streptococci (Photo credit: Wikipedia)

Government posts visualisation with Isotope

I found this visualisation on my server this morning - hadn't looked around the directory structure for ages and forgot I'd made it. I think it's quite cool, might think about carrying on with it in some way or another.



You can play with it here:

http://danpaulsmith.com/apps/postlist_new?dept=dft.

You can replace "dft" with a department acronym (e.g. co, dfe, hmrc...).

SVG-based, linked-data-driven chart widgets

In Feburary I was asked to have a pop at visualising some spending data for some local UK councils. The data was in linked data format (RDF/Turtle) - so stored in a RDF store somewhere and had the Linked Data API layered on top (the Puelia implementation).

The brief was to build an open-source, interactive, cross-browser dashboard of widgets; that would allow the comparison of council's spending data, say, per month.

After a little time spent researching, I came to the conclusion that RaphaelJS (an open-source JavaScript vector library) would fit the bill for this project nicely. The documentation wasn't great (I'm used to that though as I've been using theJIT library for previous visualisation work), but understandable enough to pick some of the demos apart and get the hang of how things worked within a few days.

The great thing about RaphaelJS is that it's simply a drawing library, so you can create a static SVG image of a banana or you can create a animated, multi-coloured, shape-shifting, real-time rotting banana, thanks to being able to manipulate and listen to events on the SVG DOM elements that form the vector image.

Linked Data API - JSON endpoints

For developers who are familiar and comfortable with JSON - I've gathered a list of linked-data JSON API endpoints, to provide shallow waters (that can still attract the most colourful of fish!) for linked-data beginners.

Installing memcache on Ubuntu 10.04

Another Linux guide, simply because I had to use a mixture of guides to get memcache working for me.

Slides and notes from Open Government Data Camp 2010

I didn't actually end up using these slides on the day, but here they are with notes & links.

Please note: The geniuses behind the Linked Data API are listed here and Jeni Tennison is also the genius behind the API's HTML Viewer, which makes API responses human friendly and a joy to develop with, transforming this into this!

More information about the "reference" data I'm using and examples of the Linked Data API & HTML Viewer can be found in the Linked Data section on the data.gov.uk website.

Government organogram work

I found an image on the Telegraph's website recently that helps me explain what exactly I'm working on with the data.gov.uk team right now.

The image is a collage of the government's departmental organisation structure charts - labelled "Cameron's biggest nightmare" - which Cameron requested from the departments towards the end of 2010 (along with salaries and other information) - in an effort to increase government transparency.

The data.gov.uk team (and others including Talis, Epimorphics and TSO) have been working hard on modelling, converting and producing all of this data in machine-readable formats. To cut a long story short about why machine-readable formats are being used - it's because a) relationships between data can be expressed - letting machines infer and conclude things about the data using higher levels of semantic information and b) because standardised formats increase interoperability between machines and applications allowing the data to be used - and linked to - more easily and more often.

Since July, I have been designing and implementing an interactive governmental organogram as a web-application that's powered by linked data (RDF/Turtle) - which, once every department has published their organogram data, will be a generic application that's able to visualise every department's structure. Each department will also be able to embed the web-application onto their website and customise the look and feel as they wish.

Each department roughly publishes three types of data - their organisation structure, junior posts & grades and salary information within the structure. As linked data is being used, connections and relationships can be established across each of these three datasets and the web-application is able to automatically display the connecting information in real-time - allowing the public to ask questions such as "who is earning more than their boss?". Without the use of linked data, asking this question would have required searching for the people and their salaries across three datasets (which may be in spreadsheet format for example). The benefits are tenfold once you start linking together more datasets each containing hundreds of thousands of data entries (i.e. the COINS dataset).

Work is still on-going for the web-application as I've just got my hands on the real-data which is slightly different to what I've been working with throughout the prototyping stages, but everything's working as planned and looking great!

* Update: Version 1 of the visualisation can be found at http:/labs.data.gov.uk/gov-structure