Slides and notes from Open Government Data Camp 2010

I didn't actually end up using these slides on the day, but here they are with notes & links.

Please note: The geniuses behind the Linked Data API are listed here and Jeni Tennison is also the genius behind the API's HTML Viewer, which makes API responses human friendly and a joy to develop with, transforming this into this!

More information about the "reference" data I'm using and examples of the Linked Data API & HTML Viewer can be found in the Linked Data section on the data.gov.uk website.

Slides

 

http://blogs.telegraph.co.uk/news/conradquiltyharper/100059122/the-chart-that-keeps-cameron-up-at-night/

• The Telegraph recently created a collage of all of the Whitehall department organograms, naming it Camerons Biggest Nightmare


• And you saw from earlier that this data is becoming available as linked data In the reference.data.gov.uk database (List of organograms from the Cabinet Office: http://www.cabinetoffice.gov.uk/newsroom/news_releases/2010/101015-structure-charts.aspx)


• In the last few months, Ive been trying to create an "organogram explorer" that people can use to browse around departments - seeing who reports to who as well as the general overview of a department and its units


• As the data is pulled in real-time from the reference.data.gov.uk database through the Linked Data API - whenever the data is updated, my visualisations will update automatically too - they are alive!


• The HTML Viewer and my visualisations are both direct representations of the data what you see is what you can get.

• Up until now, and more than often, following a hyper-link in search of linked data would lead you to a blank text box asking for a SPARQL query.


• Sure a SPARQL query can be entered by hand, but these endpoints (serving machine-readable data) are mainly for applications or APIs to latch on to


• The first time round, most people have no idea what data they want exactly - they just want to take a peek at whats on offer to get a better idea of the details.


• With the introduction of the Linked Data API and its HTML Viewer useful links can now be provided for "slices" of data, which can now be found on this page for example: http://data.gov.uk/linked-data

http://reference.data.gov.uk/doc/department

http://reference.data.gov.uk/doc/department/dfe/unit

http://reference.data.gov.uk/doc/department/dfe/unit/education-standards-directorate/post

• With a bit of previous linked data experience from university, I started tailoring my data needs thinking about what I needed in terms of things (helpful when using linked data)


• As I have been working together with the people generating the data, I was told specifically what data I could use and the types of properties and values contained in that data, so I didnt need to carry out much detective work myself. If this wasnt the case, then the Linked Data APIs HTML Viewer does a brilliant job at letting you explore and discover data for yourself


• As mentioned earlier, for each API call, several formats can be requested by simply adding their extension to the end of the URL (.rdf, .ttl, .xml, .json)

http://reference.data.gov.uk/doc/department.json?_view=minimal&_properties=unit.label,unit.post.label,unit.post.reportsTo&_pageSize=10

• The APIs flexibility is outstanding you can get your hands on almost anything in any shape


• For the initial API call for the government structure visualisation, I needed a list of departments with a path of connecting information: department > units > posts > reporting posts


• I can get this data using the /department endpoint shown above, and then by specifying an API parameter: _properties=unit,unit.post,unit.post.reportsTo


• .label means I want the human-friendly, readable "rdfs:label" property of that item - another alternative could be the "skos:prefLabel" using "unit.prefLabel"

• On the left is a list of departments in JSON format returned by the API (http://reference.data.gov.uk/doc/department.json)


• The top part of the response is useful metadata about the data such as the number of items per page and what formats are available.


• Typically a result will contain a list of items (for List Endpoints) and in this case the items are departments.


• Each department item returned by the API contains data such as the departments name and the units within that department.

• So the API returns a nested structure here (lists of units within a list of departments), and uses it's vocabulary to name the JSON items and their properties


• However, the visualisation plugin I wanted to use required a different JSON data structure (pictured on the right)


• I needed to convert the APIs JSON structure to a different nested JSON structure - as well as replacing the property names to match id,name,data and children, which was achieved using custom JavaScript functions.

http://labs.data.gov.uk/gov-structure

/gov-structure

• Aiming to be a journey-starter for the organogram visualisation, this is a treemap (great for large amounts of hierarchical data such as computer file systems or the UK government) allowing you to drill down into departments and units.


• Departments can be sized by the number of units or posts they contain, units are sized by the number of posts they contain and posts are sized by the number of posts that report to them. Clicking on a post will load its organogram.


• You can jump back to the /gov-structure visualisation by clicking on the breadcrumbs within the organogram visualisation


• Information about the data being used in the visualisation can be found by clicking on the data source items in the bottom right of the screen


• Note: this isnt the whole government visualised, but its what is available through reference.data.gov.uk right now. As new data is uploaded to the database, it will then appear within the visualisations

/organogram

• A concious effort has been made to link back to the data itself when possible the post information panel contains links to the APIs HTML Viewer for the post and its unit. If the user is fairly interested in that post or the post's unit, they are given the option to go off and explore the data surrounding those things using the HTML Viewer.


• Again, the data sources are present in the bottom right describing the API calls that have generated what the user can see in-front of them, where the data was retrieved from (an API endpoint) and the different data formats available