Bringing Azure+OData+Silverlight Goodness to Open Data Apps in Ottawa

On Saturday April 24th 2010, at Open Data Ottawa Hackfest, a team built the ‘OttGuide’ application (here showing 6000+ bus stops on the Ottawa public bus transit network), built on the Open Data Application Framework using Silverlight and Bing Maps, accessing an OData service built with .NET 3.5 and hosted on Windows Azure and SQL Azure.  For more on open data and cross-platform togetherness, consider for the Make Web Not War conference on May 27th 2010 in Montreal.

odata1

This passed Saturday Apr 24th 2010, the very successful Open Data Ottawa Hackfest took place for the first time in Ottawa City Hall organised by Edward Ocampo-Gooding and friends.  A large number of people (enough to exhaust the free t-shirt supply) attended for 4+ hours from different fields (developers, designers, librarians, statisticians. media and city officials).  The event included opening presentations, a hack fest where attendees worked on applications showing use of Ottawa data and closed with a series of presentations showing the fruits of the attendees’ labour.

Prior to the event, the City of Ottawa (unlike cities like Vancouver and Edmonton) hadn’t officially published municipal data or a data publication policy.  This meant that attendees to the hackfest were using various interesting methods to get data for their application from screen scraping to manual data re-enty.

I had the opportunity to talk at length with the city’s CIO, Guy Michaud, and discovered that the city is in fact on the verge of agreeing a policy and then immediately publishing a few initial sets of satic data, most likely sometime in May.  It’s quite possible that the hackfest event helped push this action ahead, but Guy is clearly a guy with forward-looking ideas on the issue of data publishing.

I was told about the event a few weeks ago by friend and Microsoft Canada IT Pro Advisor Rick Claus who was planning on attending.  So, in those weeks, I took a look at some of the work that has been done to obtain Ottawa municipal data and at open data work in general.  In particular I looked in 3 things:

odata2

  • Craig Davey had done extensive work on taking Google-compatible data published on the City’s OCTranspo site, augmenting it, converting it to GTFS format and publishing it with updates in a feed.
  • I had been familiar for some time with Microsoft’s open protocol for providing access to queryable and updateable data sets in a RESTful way – OData.  I did further research into this.  OData builds on ADO.NET Data Services and is fully incorporated into WCF Data Services within the recently released .NET 4 framework.  OData is designed to be consumed (or produced) by many different clients on many platforms. There are client libraries available or in the works for .NET (3.5 with an update & 4.0), Silverlight 4.0, iPhone, Javascript, Windows Phone 7, PHP & Java.  In fact, any client that can make web requests and handle XML can access OData-based services.

The night before the event, on a total whim, I decided to knock up an OData service exposing OCTranpo data, so I: 

  • started with Craig’s GTFS-format data provided as CSV files in a ZIP file. 
  • created an ADO.NET Entity Framework Model in a new .NET 3.5 Class Library project in Visual Studio 2010 to represent the GTFS data with entities and relationships
  • used the designer tools to generate DDL script to create the database schema
  • generated the schema using the script
  • created a database in my local SQL Server 2008 server
  • wrote code to import the CSVs and use the ADO.NET Entity Framework classes to import the data which was taking 3+ hours
  • wrote new code to import the CSVs with the SQL using bulk copy class bypassing the model which did the import of over 2 million bus stop time rows in a few minutes
  • added a WCF Service project and removed the default service from the project template
  • added a WCF Data Service item, setting it up to use the class of the entity model I created and providing read access to all entities
  • copied the connection string from my entity model class library app.config into the WCF Service web.config
  • tested my OData service using the OData URL query syntax, here showing a list of all stops… but this could be queried to find things like a nearby stop (from GPS coordinates), the bus times for a stop, the trips (e.g. ‘saturday service to ABC on route x’) and routes (i.e. bus number) going through a stop, find the stops on a trip, when a specific service gets somewhere, or even for full journey planning, etc.

odata3

  • re-opened by solution as administrator so I could use the Azure Cloud tools
  • added a Windows Azure cloud project with no roles and added my existing WCF Service project to the roles
  • tested OData access running in the local Azure Development Fabric connecting to my local SQL Server database
  • created a SQL Azure server and database in the cloud using the SQL Azure portal
  • connected to SQL Azure using SQL Server Management Studio and created my database schema using a modified version of my existing DDL script (removing USE statements)
  • used the bcp utility to export tables from my local database to local files and then again to import those up to SQL Azure
  • updated the connection string in my web.config to use the SQL Azure database
  • tested OData access running in the local Azure Development Fabric connecting to the SQL Azure database
  • published by cloud project in Visual Studio, created a Hosted Service on the Windows Azure portal and deployed the application to Azure using the portal
  • tested OData access running in on Windows Azure connecting to the SQL Azure database

So I had an OData service exposing the OCTranspo bus routes, trips, stops and trip stop times – sweet!

When I got to the event at 3:15pm, I met up with Christian, Rick & John Weigelt (Microsoft Canada’s National Technology Officer).  They were looking at the VanGuide application and potential Ottawa data sources to hook up to it as a demonstration.  Demonstrations were due at 4pm.  Of course, as I’ve stated, good data sources were hard to find because the City of Ottawa hadn’t published any official open data sets.

OData service to the rescue…?

We set to work on integrating my sample OCTranspo OData service (which was already running the cloud), into the the Open Data Application Framework, transforming the VanGuide application into an ‘OttGuide’ application.  The framework includes a Silverlight application project that includes dynamic handling of web-based data sources (the dynamic list of landmark sources on the left of the UI) and asynchronous data retrieval and handling code (that takes KML and other format data and adds data points as map points on a Bing Maps Silverlight control).

We wanted to make this Ottawa specific, wo we went ahead and:

  • modified the data source list to show an entry for OCTranpo and removed the other ones
  • removed other Vancouver-specific map region options
  • added custom code to handle OCTranpo GTFS entities in a feed (our OData feed) using Linq to XML to create Landmark instances that the framework already understands
  • made the application zoom in on Ottawa
  • made some other cosmetic changes for Ottawa

The result is the application you see at the start of this blog entry.  For full disclosure, we got the full bus stop retrieval and map point creation working literally 1 minute after the public presentation section finished, but Christian managed to demonstrate the VanGuide version and the OData feed.

The ODAF framework means that an OttGuide application user can select bus stops, rate them, add comments, etc.  It would be relatively easy to add datasets for other ‘landmarks’ (e.g. water foundations, tourist spots, parks and even pot holes).

The same OData service could be used on clients like the new Windows Phone 7 platform to give bus services users valuable travel information :).  Perhaps more on that later…

Open data and open web technologies are current hot topics and OData is a great protocol to bring things together on many platforms.  Silverlight is a great technology to deliver rich user experiences on many platforms using data sources from a variety of platforms. 

Microsoft and Open Source Communities are sponsoring the Make Web Not War conference on May 27th 2010, showcasing the latest cross-platform techniques and technologies, including presentations, panels, workshops, a codefest, the FTW coding competition and a party!  Open Data, HTML 5, PHP, JQuery, Mobile, SEO and the Cloud, are amongst the topics being covered.  Speakers and panelists attending include myself, Microsoft Canada team members and many open source experts.

Make Web Not War

Advertisements

Add a ‘Windows Live PC for Xbox’ to Your Live Mesh

In this series I’ve been talking about the possible strategy that Microsoft could be unleashing with Live Mesh and associated Microsoft technologies.  I’ve framed it as a trojan strategy because it is not the offering being talked about, but just like Silverlight (the trojan RIA platform onto other platforms), Live Mesh could quickly spring into something dramatic – the Microsoft ‘Live PC’ concept that I predict in the last post.

Live Mesh provides a Web-based Live Desktop which is currently just a 5GB file store with a Windows-Explorer styled web interface.  Add Windows Server 2008 Server RemoteApp into the mix, or should I say mesh, and you get the ability to run Windows anywhere you can run Remote Desktop.

Remote Desktop uses the Remote Desktop Protocol.  So for a client device to provide a virtual Windows experience it more or less just needs to support a graphic blitting display, keyboard & mouse (or similar), TCP/IP and some cryptography for security.

So how basic could such a device be?  Well that doesn’t matter because that xbox 360 is more than powerful enough and guess who sells that.  That’s right, you may already have a device in one or more rooms in your house that could be the PC of your future.  Remember that the RD protocol isn’t great for remoting intense A/V or graphics.  That’s OK, because you would play games locally using the full local power of the xbox, and Microsoft has already mastered the Xbox Live multi-player service.

So you could have a Microsoft ‘Live PC’ which you access from any Xbox without any software installation.  Xbox already does this kind of trick and even with HD video when it acts as an extender for a local Windows Media Center (running on XP Media Center or various Vista versions).

WPF makes it easier.  The RD protocol does things to optimise the transfer of the virtual desktop image on the remote physical machine.  When Xbox 360 is used as an extender it talks to the Media Center service on a local PC with a higher-than-pixel-level protocol to optimise the data.  WPF provides a high level of retained descriptive UI too.  I can see the RD protocol optimised (if it hasn’t been already) for remotely WPF applications.  Microsoft would then encourage ISV to create more WPF-based apps that would be inherently optimised for a ‘Live PC’ experience.

Let’s not forget that Windows Home Server that was quietly (relatively) released last year.  That server could start providing a LAN-based RemoteApp service for those things that can’t be run well over an Internet connection.  A virtual ‘Home Office 201x Service’ perhaps?

Death of Hal

Arthur C Clarke has died at age 90.  He co-authored 2001: A Space Odyssey (based on one of his short stories) and envisaged the idea of the communications satellite 20 years before one was launched.

Some people may not know that there are sequel books to 2001 (along with one sequel movie) including 3001: The Final Odyssey.

I’m just getting started on the 2nd of his Time Odyssey series that he co-authored with Stephen Baxter.  The 3rd book of the series was published just last December.

The ultimate Horrific Question

Warning: the article referred to, deals with with war-crimes.

In reading this CNN article about a war crimes trial (only for the non-squeamish!) I came across what seems like one of the worst questions of all time – you’ll know which one it is if you read it.

I thought twice about posting this.  Not much surprises me and I generally post about technology surprises or disappointments.  While I was aware that the subject matter dealt with has been known to happen in human history, I was surprised that the defense attorney would ask such a question.

Mix08 Day 2 Keynote with The Steve Ballmer – Waiting…

It’s starting at 13:00 PT.  I wonder what will be announced.

So far we’ve not really heard anything new.  Sure, the Silverlight 2.0 Beta 1 runtime, tools and sdk have been released (along with a disappointing roadmap for Silverlight on Mobile, save for the Nokia involvement as some unspecified point) along with a developer/standards ie8 beta 1.

If there’s any kind of announcement, I’d expect it would be in relation to a cloud-based service.

I’d like to hear about something both cool and surprising and preferably fully-baked to download today.

Microsoft platform technologies are so community-previewed these days – using the conversation to fine tune the product – that there’s nothing that surprising any more.

I’d like to hear about something both cool and surprising and preferably fully-baked to download today.  Please?

Let’s see…

Mix 08 Keynote With Scott Guthrie – Part 8

WPF Apps

Casting line of business application for Cirque Du Soleil

1000 artists nightly in 15 shows (6 resident and 9 touring).

22000 artists in database.

Field-based WPF-based casting system on tablet with camera strapped on.  Scott juggles for an in-the-field-based casting session.  Offline cached database.  Visual form entry and comparisons.

Upcoming WPF improvements in the summer

  • More controls
  • Streamlined Setup
  • Startup
  • Graphics layer

Demo of image or HD video with effects:

  • Hardware acceleration by default
  • Super low CPU usage because efforts on GPU
  • Custom effects API for hardware shading