Header: StructuredBlogging.org

Structured Blogging Roadmap 

Preamble

The first version of Structured Blogging was released to the world in December 2005 and was greeted as a positive development for the Web industry. But we're not done yet! This document outlines the roadmap for where we want to take Structured Blogging and how we're going to do it. In Part 1 of the Structured Blogging project, we did an initial pass at providing a toolset for microformats - with an starter set of code libraries and plugins. Now we want to expand the base of libraries and plugins, add APIs and build out some compelling end-user examples.

In Part 2 of the Structured Blogging project, we want to enable people to DO things with microformats. Structured Blogging is all about providing end-users with solutions, enabling them to use microformats and microcontent. We're in the era of the 'Live Web' (or Web 2.0), which is about content applications and services that utilize the Web platform. But to make the Live Web work, we need compatible schemas and APIs to mesh all our apps and services together. There are a lot of missing links currently in the Live Web, which is where Structured Blogging can help. 

We’ve identified the following technologies that we need to provide, to continue our roll-out of Structured Blogging:

  • More discovery mechanisms
  • Namespaces
  • Shared xml servers
  • APIs

Structured Blogging Part 2 and beyond will ensure we fill in the blanks and the holes that are missing in the microformats ecosystem. It's also imperative that we continue to build partner relationships and support others in the ecosystem. Microformats are a great way to define microcontent schemas. Structured Blogging then is about taking Microformats and doing something with them - focusing on end-user benefits. 

One issue is that there are a lot of schemas coming from a lot of different places. The role of Structured Blogging is to superset schemas and make sure that (wherever possible) all data fields are covered in these 'super-schemas'. Our role is also to support existing schemas, such as Apple’s iTunes and Yahoo’s mRSS. We will let the marketplace decide which schemas survive in the long-run and we'll let the technologists solve the incompatibilities of schemas. Note that we will only ever invent our own schemas and APIs when they do not already exist in the ecosystem.

So to re-state our position: we love microformats, but that's geeky stuff. What we're doing with Structured Blogging is putting the emphasis on solutions. We believe that only so many people will add tags to their pages. And when tools vendors implement microformats, they often ask: "where are the subscription formats?" or "where is the autodiscovery?". It’s our job to fill in these holes and provide a comprehensive solution to the world, to help foster the uptake of microcontent and microformats.

As Bob Wyman has stated, it is important to remember that Structured Blogging is a thing you do and is independent of formats. On the other hand, "microformats" is, as it's name implies, simply one of many approachs to defining the under-the-covers formats that are used by people who are doing Structured Blogging. The two concepts are orthogonal. They don't compete. They can't compete. Verbs don't compete with nouns.

We've divided up the roadmap into three parts: 

Part 1: immediate; short term
Part 2: medium term
Part 3 & beyond: long term

Note that Parts 1 & 2 are primarily devoted to starting products and getting initial versions out there. But it won't be until Part 3 that we ship refined Structured Blogging products.

Publishing data shouldn't be an absolute requirement to using Structured data. We should be focusing on tools, use cases, and making publishing easier for authors - not just formats.

We’ve already started to refine the web site, including adding tutorials and an easy-to-follow install/config explanation. 

Finally, StructuredBlogging.org is becoming a non-profit and we're looking for people to fund it. PubSub and Broadband Mechanics have agreed to initially fund the project.

Part 2

End-user benefits

The end user benefits of Structured Blogging were not fully articulated in the announcement of Structured Blogging. We want key participants to talk about WHY microformats matter, which we're still not seeing happen. We’ll have a section on the site summarizing the end user benefits, but we need to make sure solutions get built to show the benefit of microformats and Structured Blogging. We can’t rely upon developers and the blogging community for this key ingredient - it's up to us to lead.

To that end, following are 6 different end-user benefits of Structured Blogging which we’ll focus on in Part 2 and beyond. We’d love to hear from you about any other end-user benefits you see arising out of widespread support of microformats and Structured Blogging. 

The plan is to open source all the code, provide plug-ins for WP and MT and build an output to blog objects as well. We recommend (to start off with) PHP and Perl, WP and MT. Note that an aggregator may run as a standalone site rather than inside a blog, so it could just be PHP - in which case we will scrape and aggregate events, then output an aggregate RSS feed which will be added into the blog.

1) Reviews Aggregator

The first kind of end-user benefit we’d like to build is some generic XML server solutions, in particular a Reviews Aggregator or server – modeled after Phil Pearson’s coffee review site or what Alf Eaton did years ago. There aren't many decent Reviews Aggregators on the market currently, so this is a gap we can fill. Google is one of the few Reviews Aggregators that is pulling in data from external sources - yet it only uses a few mainstream sources. Yahoo and Allconsuming.net are centralized, as are most of the others. So there is an opportunity for us to build an open source microcontent aggregator for reviews - starting with, say, music and expanding from there.

To make this happen, we need to build a server to read review schema(s) and provide a UI around the public page that the server generates. We will provide APIs and the server will become one of the ‘destinations’ that OutputThis will support. Fancier UIs for browsing through the reviews can be built later, along with things such as building in mobile posting. Ideally in the future people will ‘podcast’ their reviews directly from their cell phones. That itself is an app and service that we should provide.

In Part 2 we can create a product that will enable users to send content to it via outputthis (similar to how the Topic Exchange works). Alternatively it can poll RSS feeds and aggregate content that way, like how k-collector works. 

We’re hoping that PubSub and others will provide feeds for ALL the reviews they find, which will then be collected by our Reviews aggregator. One suggestion is for PubSub to parse microformats, or allow matching on HTML class attributes. If a user could subscribe to something like CLASS:hreview and have PubSub notify them whenever it encountered HTML like <div class="hreview">, we could create a feed that will output all reviews marked up with the hReview microformat.

2) Event Aggregator

The second end user product we propose is an event aggregator. This would bring to life the scenario Salim mapped out when we released Structured Blogging.

Note that this can be part of a bigger plan to build at least 2-3 shared XML servers, that are general purpose and do nothing but support our APIs and store microcontent. Building this Event Aggregator will require us to produce conversion routines, basic ‘MyEvents’ structures and the mechanisms to move/connect/subscribe to all this stuff. Web 2.0 calendars are already becoming a commodity, so we can contribute to the world by building an ‘open source’ one. We also need to get data into and out of Outlook.

The best proof that Structured Blogging is useful even without aggregators is the simple fact that we have people using Structured Blogging plugins today even though there are virtually no aggregators. SB isn't just about formats and it isn't just about aggregators. However, for most people, there will be great value in publishing the data. If for no other reason than that doing so will make it possible for them to get indexed by semantic search tools, like a future Google Base or a microformats base.

Here's the design:

Here's a high level overview of a development plan:

We should start off with something down and dirty, which doesn't try to compete with any of the event aggregators and new kind of calendar apps out there already. This means we only work with (for example) the online version of Outlook at Live.com – and take in subscription feeds from various sources such as UpComing, Eventful, Zvents. We can then expand to the client Outlook and scrape a decent number of social networks later. Then functionality like what Tantek showed recently, where he scraped conference agendas out of a conference web page if it’s been hCal enabled. It would be nice to support hCal in an RSS feed. Indeed this could be the main way of getting events in - the Structured Blogging plugins output hCal in RSS feeds and then eventually other sites will too.

The real purpose of this aggregator (besides showing a compelling end-user benefit) is to justify creating conversion routines, namespaces and any other building blocks missing from our current code base – which will be required to build this sort of service. By giving all the code away for free, it becomes a framework for others to use. Needless to say we should use any scraping, blog object building, parsers or other routines we can find. Make it a mashup unto itself. Then the issue becomes – once again – which tools does it work with and what code libraries are available?

3) Shared XML Servers

Another example of a microcontent aggregator. We will produce generic servers – in multiple language bases – which can support any of our schemas and APIs for access. We will support PHP, Ruby, Java, ASP.NET, Python, Perl - and perhaps Plone and Drupal as well.

4) Conversion routines

We've spoken before of ‘conversion’ routines, which enable people to convert from (for example) a large XML file into microformats. Currently there is the issue of importing/exporting into the world of legacy applications, which aren't covered by microformats. We really need conversion routines and we need them to flesh out specific end-user benefit scenarios.

Particularly useful would be conversion of microformatted data into Structured Blogging XML, so we could import microformatted posts into the SB plugin. So 'conversion' becomes a verb that the APIs support.

5) Enterprise usage of microformats and structured blogging

[PubSub to do]

6) End-user benefit showing off our new AUTO_DISCOVERY code 

(see below)

Summary

This is just the beginning and we're looking for more examples from the 'real world'.

CODE – Libraries and additional plug-ins

These are the things which we need to get out to the community. We only have 3000 downloads so far, which isn't good enough. A lot of that had to do with our website, which we're in the process of fixing right now. Our aim is to make the whole process of people installing and setting up the code much better, so we're working on making the documentation and tutorials better. 

We also need kick-ass code and we’ll continue to improve that too. The specific technical issues are drilled into below, but basically we need additional libraries in different languages and we need to support additional tools and platforms. For example Gus has officially requested Python and Java libraries of everything. We should also support ASP.NET and Ruby.

Open APIs

This needs more discussion. But at the very least we need to be eating our own dogfood and go beyond just schemas. Put and Get are the most obvious APIs to provide. Plus (as mentioned above) conversion routines.

Once we start talking about Open APIs and media as a kind of microcontent, then the issue becomes: “where are the Open APIs for media?” It is important that we take this responsibility on.

We suggest that StructuredBlogging.org embrace and evolve into an Open Media API standardization initiative. YouTube and Flickr (amongst others) have established APIs for video and photos. There are no podcasting APIs yet. However we think Yahoo and Google will also release APIs for their media efforts. So while there are others out there releasing media APIs, we need to provide a superset of that and do our ‘compatibility box’ - wrap our arms around them all.

In short, media APIs are really important because API implies that there's a place/database where items are stored. Whether it be a generic place, a virtual file system or directory structure - there's a lot of work to do on the implementation. We need shared xml servers, APIs to call up an event or review, and so forth.

Marketing

As Google Base and Edgeio fight it out and Microsoft, Yahoo and AOL decide to play – Structured Blogging has a huge opportunity to position itself as the objective 3rd party, standing on the side rooting for everyone. And that means all search engines as well. This would make us 'The User's advocate', in that we don't have any biases towards the others and so we're in the best position to tie everything together. 

We need to go out and position ourselves with microformats, focus on end user benefits, do roadshows and bring it to the people. Plus we need to work with the big companies such as Microsoft and Yahoo. There is a ton of stuff we can and should do to help market and sell Structured Blogging. It’s not all limited by budget either, because politics and developer uptake are a key aspect of marketing as well.

Some marketing suggestions:

1. We need to circle back to AOL, SixApart, Microsoft and Yahoo and get their support.

2. Ideally we’ll do a series of speeches and presentations, plus a roadshow. Based upon standardized .ppts, a comprehensible rap and open strategy. Anyone could then take these presentation materials and use them - for example at their local BarCamp or user-group meetup, or for their bosses and influencers.

3. Marc has friends at a company called Qumana, so we’ll be using them as an early stage developer (in addition to Reger.) We believe we can recruit at least 5 other small startups to directly contribute and help. Candidates include Edgeio, Plum, Iceflakes, MyToday and NetVibes. All others who wish to participate and encouraged to do so. For example edgeio would be very happy if we would add a 'listing' microcontent type that would output the tags that edgeio picks up on. Some people are already working on an hListing microformat and they plan to support that, so we should get involved in that.

4. We also need to focus on getting our current fans supporting the code and hone in on key aggregator developers.

5. Also get the support of people like Mary Hodder, Euan Semple, Kevin Burton, Niall Kennedy, the Atom folks, Danny Ayers, and many others.

CSS style sheets

One of the biggest and most important things we need to see happen with SB.org (besides APIs) is adding the ability to simply and easily define a unique CSS style sheet for each post type. We have some prototypes of that, for PeopleAggregator, but they’re just a start.

What we have right now are programmer layouts, which are basically an ugly vertical stack of schema elements. But those UIs can be polished up a lot. We can certainly do better than what we have right now. Once microcontent has unique borders, frames, background images, wrappers to delineate them “from the rest” – that's when the meme of microcontent will start to flow.

Schemas

We’re not done yet – that’s for sure! We need to do a comprehensive review of our schemas and add on a few more. Remember much of our work is done by leveraging the work of others, so it’s about research and contacting the key players – and supersetting their existing work. For example, SixApart has recently submitted their ‘Lists’ schemas.

Some suggested schemas:

1. Recipes (Part 2)

2. Items/Listings (as in “things that get bought”) (Part 2)

3. Jobs

4. Fine Arts

5. Press Releases

All of these new schemas need to get baked into our code. Note that some of these can be done in Part 2, others can be done in Part 3 and beyond.

International support

A request has come onto the list recently for International support. Currently that only means the title of the kind of microcontent, but we should also translate all of the English text in the plugin and the microcontent description files - so a native-language UI can be presented. 

We need basic UTF-8 support and other International standards should be built-in.

Bugs – support

Obviously we need to clean up any and all bugs, get our content to validate and most importantly support our end-users. Some of the issues we need to clean up are:

  • categories don't work correctly
  • error handling - e.g. if you don't configure an upload directory in MT, things silently fail
  • a few changes have been made in the WP plugin, but not yet ported to the MT plugin

We suggest using a Wiki and Bugzilla to manage bugs support. This will be an on-going cost. Mail lists can only do so much – and special emphasis should be put upon developers of complementary or supporting products.

Broadband Mechanics would like to train 2 or 3 of our Indian resources to handle on-going support – over an extended period of time. This will significantly reduce our costs, while still providing us excellent quality work. But we will need a contract and commitment for some decent period of time, before we can hire and train these resources.

Fundraising

For a status update, we are now a corporation. We are filing for non-profit status, which is a lengthy process. We do not however have to wait until this happens to start raising funds. We need to see how much it will cost to get through the “must haves” find out who is willing to kick in the dough and who is willing to kick in the developers. Marc mentioned about $500,000 of work here. We need to get that number down by about half. We can do that by delaying some of the work, or getting other companies or individuals in the community to kick in some developers. Jon's preference is for the latter. We are encouraging everyone to contribute what they can. Whether it is fund or developers, it is all of us working together as a community that will make this successful.

Parts 3 and 4

CODE – Libraries and additional plug-ins

Ruby libraries and .Net code are coming in Part 3 and beyond. This will mark the point where we extend beyond just code and begin to develop Structured Blogging frameworks. Examples of existing frameworks to emulate in scope are Laszlo, Plone and the construction kit framework of Drupal.

We have almost finished a Structured Blogging plugin for Drupal and we’d like to see plug-ins for other leading tool vendors too. 

There will also be a number of dashboard platforms released and developed over the next year, which SB.org will support. Modules/plugins are going to be big this year - already active in this space are AOL, Microsoft, Yahoo, Google, NetVibes, and many others. What this means is that Structured Blogging needs to have both of the following:

a) output displays, for example for ‘Reviews’ and ‘Events’; and

b) full fledged blog/site editors.

Note that we should offer the microcontent libraries as separate downloads, with documentation.

A Javascript library would be nice to have as well. For example, this could be how we do the cocomment.com-like 'instant SB' system.

New Technical Features

Lookups etc

We'd like to get the existing libraries to figure out more metadata by themselves. Currently we have lookup for books, movies etc, but if you upload a media file you have to manually put in data like the length, format, width, height, and so on. There's a fair bit of work involved, but it's a requirement if we want the media content to take off. 

Media Uploader

Another option is a media uploader. A standalone Windows / Mac application that will let you upload an image, audio or video file straight to your blog - or to any SB-enabled blog. Right now PHP provides fairly restrictive limits on the size of files you can upload, and of course anyone on a slow or unreliable connection will have great difficulty uploading more than a megabyte or two of audio or video.

Pervasive bookmarklet/plugin

One suggestion is a feature like cocomment.com. It's implemented by adding a second "post" button to almost any comment system (basically any html form). When you hit the "Submit" button, the content gets posted both to the original html form and to the cocomment.com server. The interesting thing is that it's totally ajax based and implemented via a scriplet, meaning that you don't need any code running on the server side. What if we could implement something similar for blogging (and generally publishing) tools? A universal SB client, which would allow us to instantly support all blogging platforms.

Microcontent creation tool

Related to all of this, perhaps we should put up a hosted microcontent creation tool on structuredblogging.org. Fundamentally this means creating a stand alone version of our plug-ins with its own CMS, for anyone not using WP or MT. It would let users create SB content and hook through outputthis.org to send it to peoples blogs. This would immediately enable people not on Wordpress or MT - and people who can't install plugins - to get at least part of the benefits of the SB plugin. Note that edgeio has just done this, with their 'hosted blog' solution.

Structured Blogging as web service 

A further extension of the microcontent creation tool is to enable users to enter structured content into structuredblogging.org - and then post it back to their blogs, if they tell us their login/password. We could make this part of outputthis.org, or include the website wrapper in the microcontent library distribution as an integration example. 

SB plugins on WP.com and Typepad

Another way to enable microcontent creation for the masses is to work with wordpress.com and Typepad (and anyone else we can find) to get the SB plugins installed on their sites. 

Auto-tagging

Edgeio has just had added a feature which enables bloggers to add their posts to Edgeio, without needing to tag them in their own blog authoring tool. Users simply enter their blog URL into edgeio's 'Get listings' textbox (on the homepage), click the button and a list of  their latest posts display. 

On the topic of edgeio and other commercial aggregators, our concerns are getting compatible data out - but they can of course do their own thing.

Wordpress 2.0 integration 

The current Wordpress SB plugin looks and feels good in WP 1.5, but doesn't take advantage of any of the new features in WP 2.0. We need to finish off the work to get rid of the duplicated post.php code - and make SB run inside the existing editing form, rather than providing its own.

Namespaces – File Formats

When we sit down and really analyze what we’re trying to do with microformats, the issue that keeps jumping out is ‘namespaces’. This is what Phil is doing to ‘correct’ the format issues caused by Bob’s <script> inclusion in his format. We need to do the following:

1. Get all the microcontent flowing as a RSS namespace, which will make it trivial to become an rdf namespace. Note that Phil is not 100% sure about RDF, because the "transform model" we have at the moment works well for that. But for RSS and Atom, Phil suggests adding a namespaced element that holds the XML that is currently encoded inside the <script> tags. That will make the feeds a little more palatable to XML purists.

2. ‘File formats’ for microcontent - Marc proposes we use OPML for that.

Another part of the effort to move away from inclusion via <script> is to move the structured XML out of rendered HTML and into separate files, which will perhaps fulfill this goal.

3. Marc has talked with Dave Winer about creating specific extensions for nodes, to support saving and loading microcontent. We’re ready for this and it just needs to get done.

International support

For Part 3, full translation of all materials into multiple languages.

What was discussed on the mailing list was translating the edit forms into different languages. The standard way to do translations is with gettext (see http://www.onlamp.com/pub/a/php/2002/06/13/php.html), which works in the same sort of way - although that article suggests it's not always included by default with PHP. If it isn't, Phil suggested making a simple XML format to store translated strings (or using an existing one, if one exists). So we would have the MCD XML file to describe a piece of microcontent and a bunch of XML files that specify translations for various strings. It seems like the community might be able to help with the actual translation.

Also we should work towards recognition of cultural norms.

Summary

Part 1 of Structured Blogging has provided us with a good base of code and tools. Now the challenge is to build on that and provide compelling solutions to end-users, in Parts 2 and beyond. At the end of the day this is a community effort and are looking forward to its endless and valuable possibilities.
For Developers
- Resources
- Formats & Support

Supporters:

For information about joining the initiative contact: info@structuredblogging.org