On Saturday April 24th 2010, at Open Data Ottawa Hackfest, a team built the ‘OttGuide’ application (here showing 6000+ bus stops on the Ottawa public bus transit network), built on the Open Data Application Framework using Silverlight and Bing Maps, accessing an OData service built with .NET 3.5 and hosted on Windows Azure and SQL Azure.  For more on open data and cross-platform togetherness, consider for the Make Web Not War conference on May 27th 2010 in Montreal.

This passed Saturday Apr 24th 2010, the very successful Open Data Ottawa Hackfest took place for the first time in Ottawa City Hall organised by Edward Ocampo-Gooding and friends.  A large number of people (enough to exhaust the free t-shirt supply) attended for 4+ hours from different fields (developers, designers, librarians, statisticians. media and city officials).  The event included opening presentations, a hack fest where attendees worked on applications showing use of Ottawa data and closed with a series of presentations showing the fruits of the attendees’ labour.

Prior to the event, the City of Ottawa (unlike cities like Vancouver and Edmonton) hadn’t officially published municipal data or a data publication policy.  This meant that attendees to the hackfest were using various interesting methods to get data for their application from screen scraping to manual data re-enty.

I had the opportunity to talk at length with the city’s CIO, Guy Michaud, and discovered that the city is in fact on the verge of agreeing a policy and then immediately publishing a few initial sets of satic data, most likely sometime in May.  It’s quite possible that the hackfest event helped push this action ahead, but Guy is clearly a guy with forward-looking ideas on the issue of data publishing.

I was told about the event a few weeks ago by friend and Microsoft Canada IT Pro Advisor Rick Claus who was planning on attending.  So, in those weeks, I took a look at some of the work that has been done to obtain Ottawa municipal data and at open data work in general.  In particular I looked in 3 things:

• Friend and Microsoft Canada Senior Developer Advisor, Christian Beauclair, introduced me to the Open Data Application Framework (ODAF).  ODAF is a framework that allows cities to easily map existing civic Open Data landmarks, and allow users to rate, comment, and tag them, as well as create new landmarks.  An example instance of this in action, is the VanGuide application which feeds off Vancouver’s open data catalogue and is hosted on the Windows Azure Platform.  Here, it’s showing the local bus stops.

• Craig Davey had done extensive work on taking Google-compatible data published on the City’s OCTranspo site, augmenting it, converting it to GTFS format and publishing it with updates in a feed.
• I had been familiar for some time with Microsoft’s open protocol for providing access to queryable and updateable data sets in a RESTful way – OData.  I did further research into this.  OData builds on ADO.NET Data Services and is fully incorporated into WCF Data Services within the recently released .NET 4 framework.  OData is designed to be consumed (or produced) by many different clients on many platforms. There are client libraries available or in the works for .NET (3.5 with an update & 4.0), Silverlight 4.0, iPhone, Javascript, Windows Phone 7, PHP & Java.  In fact, any client that can make web requests and handle XML can access OData-based services.

The night before the event, on a total whim, I decided to knock up an OData service exposing OCTranpo data, so I: 

• started with Craig’s GTFS-format data provided as CSV files in a ZIP file. 
• created an ADO.NET Entity Framework Model in a new .NET 3.5 Class Library project in Visual Studio 2010 to represent the GTFS data with entities and relationships
• used the designer tools to generate DDL script to create the database schema
• generated the schema using the script
• created a database in my local SQL Server 2008 server
• wrote code to import the CSVs and use the ADO.NET Entity Framework classes to import the data which was taking 3+ hours
• wrote new code to import the CSVs with the SQL using bulk copy class bypassing the model which did the import of over 2 million bus stop time rows in a few minutes
• added a WCF Service project and removed the default service from the project template
• added a WCF Data Service item, setting it up to use the class of the entity model I created and providing read access to all entities
• copied the connection string from my entity model class library app.config into the WCF Service web.config
• tested my OData service using the OData URL query syntax, here showing a list of all stops… but this could be queried to find things like a nearby stop (from GPS coordinates), the bus times for a stop, the trips (e.g. ‘saturday service to ABC on route x’) and routes (i.e. bus number) going through a stop, find the stops on a trip, when a specific service gets somewhere, or even for full journey planning, etc.

• re-opened by solution as administrator so I could use the Azure Cloud tools
• added a Windows Azure cloud project with no roles and added my existing WCF Service project to the roles
• tested OData access running in the local Azure Development Fabric connecting to my local SQL Server database
• created a SQL Azure server and database in the cloud using the SQL Azure portal
• connected to SQL Azure using SQL Server Management Studio and created my database schema using a modified version of my existing DDL script (removing USE statements)
• used the bcp utility to export tables from my local database to local files and then again to import those up to SQL Azure
• updated the connection string in my web.config to use the SQL Azure database
• tested OData access running in the local Azure Development Fabric connecting to the SQL Azure database
• published by cloud project in Visual Studio, created a Hosted Service on the Windows Azure portal and deployed the application to Azure using the portal
• tested OData access running in on Windows Azure connecting to the SQL Azure database

So I had an OData service exposing the OCTranspo bus routes, trips, stops and trip stop times – sweet!

When I got to the event at 3:15pm, I met up with Christian, Rick & John Weigelt (Microsoft Canada’s National Technology Officer).  They were looking at the VanGuide application and potential Ottawa data sources to hook up to it as a demonstration.  Demonstrations were due at 4pm.  Of course, as I’ve stated, good data sources were hard to find because the City of Ottawa hadn’t published any official open data sets.

OData service to the rescue…?

We set to work on integrating my sample OCTranspo OData service (which was already running the cloud), into the the Open Data Application Framework, transforming the VanGuide application into an ‘OttGuide’ application.  The framework includes a Silverlight application project that includes dynamic handling of web-based data sources (the dynamic list of landmark sources on the left of the UI) and asynchronous data retrieval and handling code (that takes KML and other format data and adds data points as map points on a Bing Maps Silverlight control).

We wanted to make this Ottawa specific, wo we went ahead and:

• modified the data source list to show an entry for OCTranpo and removed the other ones
• removed other Vancouver-specific map region options
• added custom code to handle OCTranpo GTFS entities in a feed (our OData feed) using Linq to XML to create Landmark instances that the framework already understands
• made the application zoom in on Ottawa
• made some other cosmetic changes for Ottawa

The result is the application you see at the start of this blog entry.  For full disclosure, we got the full bus stop retrieval and map point creation working literally 1 minute after the public presentation section finished, but Christian managed to demonstrate the VanGuide version and the OData feed.

The ODAF framework means that an OttGuide application user can select bus stops, rate them, add comments, etc.  It would be relatively easy to add datasets for other ‘landmarks’ (e.g. water foundations, tourist spots, parks and even pot holes).

The same OData service could be used on clients like the new Windows Phone 7 platform to give bus services users valuable travel information :).  Perhaps more on that later…

Open data and open web technologies are current hot topics and OData is a great protocol to bring things together on many platforms.  Silverlight is a great technology to deliver rich user experiences on many platforms using data sources from a variety of platforms. 

Microsoft and Open Source Communities are sponsoring the Make Web Not War conference on May 27th 2010, showcasing the latest cross-platform techniques and technologies, including presentations, panels, workshops, a codefest, the FTW coding competition and a party!  Open Data, HTML 5, PHP, JQuery, Mobile, SEO and the Cloud, are amongst the topics being covered.  Speakers and panelists attending include myself, Microsoft Canada team members and many open source experts.