Banksy by rocor, some rights reserved.

Anyway when we started work on on the BBC’s Nature site we set out our development manifesto. I thought you might be interested in it:

1. Peristence — only mint a new URIs if one doesn’t already exist: once minted, never delete it
2. Linked open data — data and documents describe the real world; things in the real world are identified via HTTP URIs; links describe how those things are related to each other.
3. The website is the API
4. RESTful — the Web is stateless, work with this architecture, not against it.
5. One Web – one canonical URI for each resource (thing), dereferenced to the appropriate representation (HTML, JSON, RDF, etc.).
6. Fix the data don’t hack the code
7. Books have pages, the web has links
8. Do it right or don’t do it at all — don’t hack in quick fixes or ‘tactical solutions’ they are bad for users and bad for the code.
9. Release early, release often — small, incremental changes are easy to test and proof.

It’s worth noting that we didn’t always live up to these standards — but at least when we broke our rules we did so knowingly and had a chance of fixing them at a later date.

From a certain perspective this doesn’t represent a big change – after all we’re still publishing exclusive natural history news stories, video collections and video clips and information about: animals, plants, habitats (and the ancient earth’s habitats, such as Snowball Earth), adaptations & behaviours, places and ecozones, the geological time periods when they lived, the major mass extinction events, including the one that killed the dinosaurs, in fact lots of information on the history of life on earth and the fossil record. We even have a page about fish – and they don’t really exist!

However, from another perspective this is a really big change. It’s a big change because we’ve (hopefully) made everything so much simpler.

We’ve made it simpler by bringing everything together into one site and removed the various sub brands – if you love nature and natural history everything is now in one place: news stories, video clips from the archive, opinion pieces and more.

Bringing everything together has also allowed us to make a few additional changes which should help us more easily publish the content.

In addition to natural history news we have a features section where we can bring together articles and photo galleries (like this one) and a new blog Wonder Monkey written by Matt Walker. Matt has written a few posts so far including this one on the oddball midge that shouldn’t exist.

I really hope you like it. It represents the culmination of two years of work, during which time we launched and evolved both the site itself and the editorial proposition – there now are c.3,000 clips available online (many of which are available worldwide) about almost 900 animals (both prehistoric and living), 50 plants etc.

And of course wildlife data is for machines too.

However, after two years of development this represents the last major release, for a while at least. The site will continue to grow because we are continuing to create great new content as well as digging out the best bits from the archive – like this video collection looking back at David Attenborough’s Madagascar (starting with Zoo Quest 50 years ago). But there won’t be any major new features for a while, not that that’s a major problem – the site should offer a rich experience with amazing content.

As I said yesterday, I’m very proud of what we’ve produced and if I can marshal my thoughts I’ll try and write a post or two about how we went about building the site and the lessons I learnt on the way, until then enjoy the site.

The BBC has been a great place to work – I’ve worked with some amazing people, helped deliver some of the best work of my career and had the opportunity to speak at conferences around the world, including (amazingly) at the Web’s 20th birthday celebrations in CERN.

Day Three Hundred Sixty Five by Dustin Diaz, some rights reserved

The BBC can certainly be a challenging place to work but I’m very grateful to Dan Hill and Matt Wood for offering me a job in the first place. I just hope I’ve not let them down because for every challenge, gripe and frustration there have also been opportunities to learn new things, work with brilliant people and help deliver great stuff that has, I think, had an positive impact on what the BBC does online.

So what have I been up to since I’ve been here?

The first project I worked on was /programmes a site that means that every programme the BBC broadcasts now has a web presence – one that both humans and machines can enjoy. The site is sometimes criticized as being a card catalogue of BBC programme metadata but its worth remembering that until the site launched the vast majority of programmes had no URI, had no webpage of any kind; /programmes changed that at a stroke. It was also the first truly dynamic web site on bbc.co.uk and whatever people might say about the aesthetics the site has the prettiest URIs of any site I know (something to thanks Michael Smethurst for).

The music site was my other project while in the FM&T bit of Audio & Music. Building on Musicbrainz the idea was to create a rich graph, linking music programmes with artist pages (available as HTML and RDF etc.) via ‘clickable tracklistings‘.

After a couple of years I left Audio & Music and joined ‘BBC Vision’ – the bit of the the BBC that does the telly – and took on a project known internally as ‘BBC Earth‘. And pretty much tried to replicate the music work but for natural history content.

I say I tried to replicate the music work that’s not really true, or rather its only true to a point. The core underlying concepts where the same, but the manifestation is quite different. For starters we sought to digitise and make available the TV archive but we also created original content – this broke down into exclusive natural history news stories, stories from TV and Radio production teams on location and, curated video collections.

I wanted the nature site to help people discover, explore and understand the natural world through the BBC’s content, I hope we’ve achieve that to some extent. Personally, and I know I’m biased, I think the site is brilliant and one of the best looking and useful semantic web sites around (we publish the data as RDF).

The credit for the site, however, should go to the team that actually made it. I was lucky, the core of the team has remained on the project throughout its development and I’m indebted to those, more talented than me, for making it what it is.

As I’ve said, I think the site is brilliant and I think the editorial, technical and design knowledge and skills of the team shine through, the site is theirs not mine.

There’s much I could write about this work – but I should really do it a bit more justice than the space available here and so I’ll save what I have to say for another post. Also there’s one last thing to push live on the site, to round off its development and it feels wrong to preempt that.

So what now? Well I’m joining Nature Publishing Group as Head of Platform for nature.com. As a failed scientist I’m very excited by the opportunities – Nature is the leading weekly, international scientific journal with a mission to:

Helping achieve that mission on the Web is a really exciting prospect and I hope the next five years prove as productive as the last. Wish me luck.

First attempt at a science ontology

To give  a bit of context – the idea behind the ontology is to provide a relatively high level model to describe the scientific method so that organisations, such as the BBC, could structure their content (archive footage, news stories etc.) using the model.

Hopefully most of the terms used are self explanatory, but for those that might not be:

Hypothesis – to quote from Princeton University – an hypothesis is a “a tentative insight into the natural world; a concept that is not yet verified but that if true would explain certain facts or phenomena”. A non-scientist might call this a theory.

Theory – again to quote from Princeton University a theory is “a well-substantiated explanation of some aspect of the natural world; an organized system of accepted knowledge that applies in a variety of circumstances to explain a specific set of phenomena.”  A non-scientist might think of this as a ‘fact’.

Agent – adopting the FOAF:agent this is a person or organisation.

Observation – an observed phenomena in the natural world cf “data” an observation resulting from an experiment.

As others have already pointed out we’re missing a lot of detail around the role of the agent, we’re missing important predicates, such as other forms of influence other than publication. And looking at it again we’ve failed to include anything about the predictions a theory might make of the natural world. But what else are we missing?

You can now watch video clips and discover BBC news stories about prehistoric life on Earth. We’ve published the best bits from TV series such as Walking with Dinosaurs and Walking with Beasts as well as episodes from Horizon and of course we’re also linking to relevant radio programmes and news stories.

At the core of the new content are pages, URIs (and yes RDF XML), for prehistoric taxa – if you visit the reptile page you will see that in addition to snakes, lizards and crocodiles we also have pages on Dinosaurs and Pterosaurs; visit the page on Apes and you can discover all not only today’s extant species (Chimps, Gorillas and us) but also our long dead relatives.

But we wanted to put this information in more context, we wanted to give a sense of time. After all most of these species didn’t roam the Earth together, at the same time. Looking back through time it is clear that different species, different groups, have flourished before going extinct, others have survived – the so called ‘living fossils. Different species have evolved at different times throughout Earth’s history. We have therefore published pages on the major geological time periods since life of Earth first arose.

You can step through time, starting with the Archean era 3.8 billion years ago and ending with the current Holocene epoch, to discover what the world was like and what lived then.

Over the last 3.8 billion years there have been a series of mass extinctions – five really big ones and numerous smaller scale ones. The most famous of these is the Cretaceous-Tertiary (or K-T) extinction which wiped out the Dinosaurs (and Ammonites). Again we have pages for both the Big Five extinction events and causes of those events all interlinked.

Our hope if that we’ve produced an interesting, enjoyable browsable space, one that gives you access to the BBC archive in a new and engaging way. I hope you enjoy exploring the history of life on Earth.

For my part I presented the work we’ve been doing on Wildlife Finder – how we’re starting to publish and consume data on the web. Ed Summers has a great write up of what we’re doing I’ve also published my slides here:

I also joined Paul Miller, Jeni Tennison, Ian Davis and Timo Hannay on a panel session discussing Linked Data in the enterprise.

In terms of Wildlife Finder there are a few things that I wanted to highlight:

1. If you’re interested in the RDF and how we’re modelling the data we’ve documented the wildlife ontology here. In addition to the ontology itself we’ve also included some background on why we modelled the information in the way we have.
2. If you want to get you’re hands on the RDF/XML then either add .rdf to the end of most of our URLs (more on this later) or configure your client to request RDF/XML – we’ve implemented content negotiation so you’ll just get the data.
3. But… we’ve not implemented everything just yet. Specifically the adaptations aren’t published as RDF – this is because we’re making a few changes to the structure of this information and I didn’t want to publish the data and then change it. Nor have we published information on the species conservation status that’s simply because we’ve not finish yet (sorry).
4. It’s not all RDF – we are also marking-up our taxa pages with the species microformat which gives more structure to the common and scientific names.

Anyway I hope you find this useful.

URIs for habitats, taxa and adaptations

Over the last few months we’ve been plundering the NHU’s archive to find the best bits — segmenting the TV programmes, tagging them (with DBpedia terms) and then aggregating them around URIs for the key concepts within the natural history domain; so that you can discover those programme segments via both the originating programme and via concepts within the natural history domain — species, habitats, adaptations and the like.

The segments/ clips ‘belong’ to their originating programme — and as a result we’ve been adding information, about a bunch of programmes from the archive, to PIPs (the underlying database behind iPlayer and /programmes). The clip pages aren’t yet linked in with their owning episode, but they will be soon.

In addition to being able to discover these clips from within the context of the programme we are also providing URIs to aggregate information around the natural history domain, that is URIs for species, habitats, adaptations and ecozones.

URIs for species such as the Bush Elephant

Our hope is that by providing highly inter-linked, URIs we can help people gain a greater understanding of the natural world. For example, by being able to see the different animals and habitats that live within different ecozones you can gain an understanding of the diversity of of life in different parts of the world; or what different animals make up the Mammal or Bird Class; or more about a particular adaptation.

Ovoviviparous - what it is, what animals do it and BBC archived content about it

Of course we are doing more than providing access to programme segments, we have also plundered our sound archive so you can hear what the different habitats and species sound like (and obviously those sounds are separately addressable), we are then aggregating content from the other ‘BBC Earth’ projects — Earth News and Out of the Wild and elsewhere on the web.

It’s not just about BBC content.

You might have noticed that the slugs for our URIs (the last bit of the URL) are the same as those used by Wikipedia and DBpedia that’s because I believe in the simple joy of webscale identifiers, you will also see that much like the BBC’s music site we are transcluding the introductory text from Wikipedia to provide background information for most things. This also means that we are creating and editing Wikipedia articles where they need improving (of course you are also more than welcome to improve upon the articles).

We are also publishing data from bunch of other organisations. Information about habitats, ecozones and species distribution is provided by WWF’s Wildfinder; the species conservation status by IUCN’s Red List of Threatened Species and (where available) information about why a species is at threat coming for Zoological Society of London’s EDGE of Existence programme. Finally information about a species adaptations and behaviours are provided by Animal Diversity Web.

Adopting this approach means that we are able to contribute distinctive content to the Web while at the same time helping people find what is already there.

There is a lot more we need to do, including linking in with current programmes and making everything available as RDF, JSON and for mobile devices. That’s all on it’s way but in the meantime I hope you find what’s there useful, informative and entertaining.

Tim Berners-Lee. Photograph by Documentally, some rights reserved.

Unlike — well probably every other TV programme I’ve ever come across — each programme will be influenced and debated on the web during it’s production. Some of rushes and interviews will be made available on the web (under permissive terms) so that anyone can contribute to the debate, helping to shape the final programme.

To kick all this off the BBC hosted a debate chaired by Aleks Krotoski with Tim Berners-Lee, Bill Thompson, Susan Greenfield and Chris Anderson. The audience was almost as impressive as the folks up on stage a great mix of geeks and journalists, and luckily I managed to wangle an invite (probably because I’ve had a tiny, tiny role on the project).

Anyway… the presentations were very cool, and while I tweeted the best bits on the day I thought I would write up a short post summing it all up. You know, contributing to the debate and all that.

The thing that struck me most were the discussions and points made around the way in which the web has provided a platform for creativity, and the risks to it’s future because of governments’ failure to understand it (OK, the failure to understand it is my interpretation, not the view expressed by the speakers).

I’ve written previously about how the web’s generative nature has helped enable an eruption of creativity, spawning a new economy in it’s wake; and how governments have failed to grasp that it’s the people that use the medium that need policing not the medium itself. But as you might expect from such an illustrious bunch of people the panel managed to nail the point much better than I ever could.

To misquote TimBL: The web should be like paper. Government should be able to prosecute if you misuse it, but they shouldn’t limit what you are able to do with it. When you buy paper you aren’t limited in what can be written or drawn on it, the and like paper the Internet shouldn’t be set up in such a way as to constrain it’s use.

The reason this is important is because it helps to preserve the web’s generative nature. TimBL points out that people are creative, they simply need platform for that creativity, and if that platform is to be the Web then it needs to support everyone, anyone should be able to express that creativity and that means it needs to be open.

As an aside there was a discussion as to whether or not access to the Internet is a ‘human right’ — I’m not sure whether it is or not, but it’s worth considering whether or not if everyone had access to the Web whether it could be used to solve problems in the developing world. For example, by allowing communities to share information on how to dig wells and maintain irrigation systems, information on health care and generally providing educational material. It is very easy, for us in the West to think of the Web as synonymous with the content and services currently provided on it and whether they would be useful in developing countries. But the point really should be if anyone, anywhere in the world where able to create and share information what would they do with it? My hope would be that the services offered would reflect local needs — whether that be social networking in US colleges or water purification in East Africa.

Of course being open and free for all to use doesn’t mean that everything on the web will be wonderful, or indeed legal; no more so than paper ensures wonderful prose because it is open. Or as TimBL puts it:

Just because you can read everything out there doesn’t mean you should. If you found a piece of paper blowing in the wind you wouldn’t expect it to be edifying.

But what does open mean?

Personally I think that an open web is one that seeks to preserve it’s generative nature. But the discussion last Friday also focused on the implications for privacy and snooping.

Governments the world over, including to our shame the current UK Government, are seeking to limit the openness of the web; that is rather than addressing the specific activities that happen on the web, they are seeking to limit the very platform itself. ISPs around the world, at the behest of governments, are being asked to track and record what you do on the web, everything you do on the web. Elsewhere, content is being filtered, traffic shaped and sites blocked.

The sorts of information being collected can include your search terms (pinned to your IP address) and the sites you visit. Now for sure this might, sometime include a bunch of URIs that point to illegal and nefarious activity, but it might also include (indeed it’s more likely to include) URIs relating to a medical condition or legal advice or a hundred and one other, perfectly legal but equally personal bits of information.

Should a government, its agencies or an ISP be able to capture, store and analyses this data? Personally I think not. And should you think that I’m just being a scaremonger have a read of Bill’s post “The digital age of rights” about the French government’s HADOPI legislation.

On the day Bill Thompson (who, by the way, was on blinding form) summed up the reason why when he summed up his hopes for the web thus:

I hoped that the web would help us know our neighbours better, so that we didn’t go and kill them. That hasn’t happened but it does now mean it’s much harder to get away with it – the world will now know if you do kill them.

Governments know this, which is why some now try to lock down access to the Internet when there is civil unrest in their country. And it is also why the rest of the web tries to help them break though.

Few Western governments, would condone the activities of such Totalitarian states. But it is interesting to consider whether Western governments would support North Korea or Iran setting up the kinds of databases currently being debated in Europe and the States. Now they might point out that the comparison isn’t a fair one since they are nice, democratic governments not nasty oppressive ones. But isn’t that painfully myopic? How do they know who will be in power in the future? How do they know how future governments might seek to use the information they are gathering now?

Seeking to prevent snooping on the Internet aside there is another reason why the web should remain open, and it is the reason why it’s important to fight for One Web.

Susan Greenfield quite rightly pointed out that ‘Knowledge is to be found by creating context, links between facts; it’s the context that counts’. Although she was making the point in an attempt to take a swipe at the Web, trying to suggest that the web is no more than a collection of facts devoid of context, it seems to me that in fact the web is the ultimate context machine. (One sometimes wonders whether she has ever actually used any of the services she complains about, indeed I wonder if she uses the web at all).

The web is, as the name suggest, a set of interconnected links. Those URIs and the links between, as TimBL reminded us, are made by people, they are followed by people and as such you can legitimately think of the Web as humanity connected.

URIs are incredibly powerful, particularly when they are used to identify things in addition to documents. When they are used to identify things (dereferencing to the appropriate data or document format) they can lead to entirely new ways to access information. An example highlighted by TimBL is the impact they might have on TV channels and schedules.

He suggested that the concept of a TV channel was limited and that it would be replaced with complete random access. When anyone, anywhere in the world, can follow a URI to a persistent resource (note he didn’t say click on a link) then the TV channel as a means of discovery and recommendation will be replaced with a trust network. “My friends have watched this, most of them like it…” sort of thing.

Of course to get there we need to change the way we think about the web and the way in which we publish things. And here TimBL pointed to the history of the web, suggesting that the next digital revolution will operate in a similar fashion.

The web originally happened not because senior management thought it was a good idea – it happened because people who ‘got it’ thought it was cool, that it was the right thing and that they were lucky enough to have managers that didn’t get in the way. Indeed this is exactly what happened when TimBL wrote the first web server and client and then when the early web pioneers started publishing web pages. They didn’t do it because they were told to, they didn’t do it because there was any immediate benefit. They did it because they thought that by doing it it would enable cool things to happen. The last couple of years suggests that we are on the cusp of a similar revolution as people start to publish linked data which will in turn result in a new digital revolution.

"Semantic Web Rubik's Cube" by dullhunk. Some rights reserved.

TimBL is working with the UK Cabinet Office (as an advisor) to make our information more open and accessible on the web [cabinetoffice.gov.uk]
The blog states that he’s working on:

• overseeing the creation of a single online point of access and work with departments to make this part of their routine operations.
• helping to select and implement common standards for the release of public data
• developing Crown Copyright and ‘Crown Commons’ licenses and extending these to the wider public sector
• driving the use of the internet to improve consultation processes.
• working with the Government to engage with the leading experts internationally working on public data and standards

The Guardian has an article on the appointment.

Closer to home there have been a few interesting developments

Media Meets Semantic Web – How the BBC Uses DBpedia and Linked Data to Make Connections [pdf]
Our paper at this years European Semantic Web Conference (ESWC2009) looking at how the BBC has adopted semantic web technologies, including DBpedia, to help provide a better, more coherent user experience. For which we won best paper of the in-use track – congratulations to Silver and Georgie.

The BBC has announced a couple SPARQL endpoints, hosted by talis and openlink
Both platforms allow you to search and query the BBC data in a number of different ways, including SPARQL — the standard query language for semantic web data. If you’re not familiar with SPARQL, the Talis folk have published a tutorial that uses some NASA data.

A social semantic BBC?
Nice presentation from Simon and Ben on how social discovery of content could work… “show me the radio programmes my friends have listen to, show me the stuff my friends like that I’ve not seen” all built on people’s existing social graph. People meet content via activity.

PriceWaterhouseCooper’s spring technology forecast focuses on Linked Data [pwc.com]
“Linked Data is all about supply and demand. On the demand side, you gain access to the comprehensive data you need to make decisions. On the supply side, you share more of your internal data with partners, suppliers, and—yes—even the public in ways they can take the best advantage of. The Linked Data approach is about confronting your data silos and turning your information management efforts in a different direction for the sake of scalability. It is a component of the information mediation layer enterprises must create to bridge the gap between strategy and operations… The term “Semantic Web” says more about how the technology works than what it is. The goal is a data Web, a Web where not only documents but also individual data elements are linked.”

Including an interview with me!

You should also check out…

sameas.org a service to help link up equivalent URIs
It helps you to find co-references between different data sets. Interestingly it’s also licenced under CC0 which means all copyright and related or neighboring rights are waived.

Kakpo -- Out of the wild

The first site to go live, “Out of the Wild” aims to bring you a view on the natural world from the perspective of our crews while on location; a sort of “From our correspondent” for the natural world. The stories — a mix of short video clips, slideshows and text based stories — are all grouped around the expeditions, the people on location and the originating programmes. Our hope is that you will enjoy this more personal view of the natural world brought to you from some of the most amazing part of the world by the worlds best wildlife documentaries makers.

We then launched “Earth News” which does pretty much what is says on the tin — news about the natural world.

We’re both aggregating natural history news articles from elsewhere on the BBC news site as well as new articles (some unique) written for Earth News, such as the story of the adult king penguin which kidnapped a skua chick and then attempted to raise it.

The final part of BBC Earth will see us starting to open up the BBC archive in, what I hope, will be interesting and useful ways.

We then, of course, need to make all of this available in nice machine representation so that others can start to hack with the data.