Bringing Azure+OData+Silverlight Goodness to Open Data Apps in Ottawa

On Saturday April 24th 2010, at Open Data Ottawa Hackfest, a team built the ‘OttGuide’ application (here showing 6000+ bus stops on the Ottawa public bus transit network), built on the Open Data Application Framework using Silverlight and Bing Maps, accessing an OData service built with .NET 3.5 and hosted on Windows Azure and SQL Azure.  For more on open data and cross-platform togetherness, consider for the Make Web Not War conference on May 27th 2010 in Montreal.

odata1

This passed Saturday Apr 24th 2010, the very successful Open Data Ottawa Hackfest took place for the first time in Ottawa City Hall organised by Edward Ocampo-Gooding and friends.  A large number of people (enough to exhaust the free t-shirt supply) attended for 4+ hours from different fields (developers, designers, librarians, statisticians. media and city officials).  The event included opening presentations, a hack fest where attendees worked on applications showing use of Ottawa data and closed with a series of presentations showing the fruits of the attendees’ labour.

Prior to the event, the City of Ottawa (unlike cities like Vancouver and Edmonton) hadn’t officially published municipal data or a data publication policy.  This meant that attendees to the hackfest were using various interesting methods to get data for their application from screen scraping to manual data re-enty.

I had the opportunity to talk at length with the city’s CIO, Guy Michaud, and discovered that the city is in fact on the verge of agreeing a policy and then immediately publishing a few initial sets of satic data, most likely sometime in May.  It’s quite possible that the hackfest event helped push this action ahead, but Guy is clearly a guy with forward-looking ideas on the issue of data publishing.

I was told about the event a few weeks ago by friend and Microsoft Canada IT Pro Advisor Rick Claus who was planning on attending.  So, in those weeks, I took a look at some of the work that has been done to obtain Ottawa municipal data and at open data work in general.  In particular I looked in 3 things:

odata2

  • Craig Davey had done extensive work on taking Google-compatible data published on the City’s OCTranspo site, augmenting it, converting it to GTFS format and publishing it with updates in a feed.
  • I had been familiar for some time with Microsoft’s open protocol for providing access to queryable and updateable data sets in a RESTful way – OData.  I did further research into this.  OData builds on ADO.NET Data Services and is fully incorporated into WCF Data Services within the recently released .NET 4 framework.  OData is designed to be consumed (or produced) by many different clients on many platforms. There are client libraries available or in the works for .NET (3.5 with an update & 4.0), Silverlight 4.0, iPhone, Javascript, Windows Phone 7, PHP & Java.  In fact, any client that can make web requests and handle XML can access OData-based services.

The night before the event, on a total whim, I decided to knock up an OData service exposing OCTranpo data, so I: 

  • started with Craig’s GTFS-format data provided as CSV files in a ZIP file. 
  • created an ADO.NET Entity Framework Model in a new .NET 3.5 Class Library project in Visual Studio 2010 to represent the GTFS data with entities and relationships
  • used the designer tools to generate DDL script to create the database schema
  • generated the schema using the script
  • created a database in my local SQL Server 2008 server
  • wrote code to import the CSVs and use the ADO.NET Entity Framework classes to import the data which was taking 3+ hours
  • wrote new code to import the CSVs with the SQL using bulk copy class bypassing the model which did the import of over 2 million bus stop time rows in a few minutes
  • added a WCF Service project and removed the default service from the project template
  • added a WCF Data Service item, setting it up to use the class of the entity model I created and providing read access to all entities
  • copied the connection string from my entity model class library app.config into the WCF Service web.config
  • tested my OData service using the OData URL query syntax, here showing a list of all stops… but this could be queried to find things like a nearby stop (from GPS coordinates), the bus times for a stop, the trips (e.g. ‘saturday service to ABC on route x’) and routes (i.e. bus number) going through a stop, find the stops on a trip, when a specific service gets somewhere, or even for full journey planning, etc.

odata3

  • re-opened by solution as administrator so I could use the Azure Cloud tools
  • added a Windows Azure cloud project with no roles and added my existing WCF Service project to the roles
  • tested OData access running in the local Azure Development Fabric connecting to my local SQL Server database
  • created a SQL Azure server and database in the cloud using the SQL Azure portal
  • connected to SQL Azure using SQL Server Management Studio and created my database schema using a modified version of my existing DDL script (removing USE statements)
  • used the bcp utility to export tables from my local database to local files and then again to import those up to SQL Azure
  • updated the connection string in my web.config to use the SQL Azure database
  • tested OData access running in the local Azure Development Fabric connecting to the SQL Azure database
  • published by cloud project in Visual Studio, created a Hosted Service on the Windows Azure portal and deployed the application to Azure using the portal
  • tested OData access running in on Windows Azure connecting to the SQL Azure database

So I had an OData service exposing the OCTranspo bus routes, trips, stops and trip stop times – sweet!

When I got to the event at 3:15pm, I met up with Christian, Rick & John Weigelt (Microsoft Canada’s National Technology Officer).  They were looking at the VanGuide application and potential Ottawa data sources to hook up to it as a demonstration.  Demonstrations were due at 4pm.  Of course, as I’ve stated, good data sources were hard to find because the City of Ottawa hadn’t published any official open data sets.

OData service to the rescue…?

We set to work on integrating my sample OCTranspo OData service (which was already running the cloud), into the the Open Data Application Framework, transforming the VanGuide application into an ‘OttGuide’ application.  The framework includes a Silverlight application project that includes dynamic handling of web-based data sources (the dynamic list of landmark sources on the left of the UI) and asynchronous data retrieval and handling code (that takes KML and other format data and adds data points as map points on a Bing Maps Silverlight control).

We wanted to make this Ottawa specific, wo we went ahead and:

  • modified the data source list to show an entry for OCTranpo and removed the other ones
  • removed other Vancouver-specific map region options
  • added custom code to handle OCTranpo GTFS entities in a feed (our OData feed) using Linq to XML to create Landmark instances that the framework already understands
  • made the application zoom in on Ottawa
  • made some other cosmetic changes for Ottawa

The result is the application you see at the start of this blog entry.  For full disclosure, we got the full bus stop retrieval and map point creation working literally 1 minute after the public presentation section finished, but Christian managed to demonstrate the VanGuide version and the OData feed.

The ODAF framework means that an OttGuide application user can select bus stops, rate them, add comments, etc.  It would be relatively easy to add datasets for other ‘landmarks’ (e.g. water foundations, tourist spots, parks and even pot holes).

The same OData service could be used on clients like the new Windows Phone 7 platform to give bus services users valuable travel information :).  Perhaps more on that later…

Open data and open web technologies are current hot topics and OData is a great protocol to bring things together on many platforms.  Silverlight is a great technology to deliver rich user experiences on many platforms using data sources from a variety of platforms. 

Microsoft and Open Source Communities are sponsoring the Make Web Not War conference on May 27th 2010, showcasing the latest cross-platform techniques and technologies, including presentations, panels, workshops, a codefest, the FTW coding competition and a party!  Open Data, HTML 5, PHP, JQuery, Mobile, SEO and the Cloud, are amongst the topics being covered.  Speakers and panelists attending include myself, Microsoft Canada team members and many open source experts.

Make Web Not War

All I want for Mix-mas this March at Microsoft Mix10

So the seasonal gift giving has now passed, but I’m hoping for a few good prezzies for Microsoft’s Mix conference.

  • Windows Mobile 7 rocking a .NET ‘Mobile’ (not compact) Framework programmable Silverlight 3+ interface, Xbox mobile games (like Zune games programmable with the free XNA Game Studio), Zune Interface, Media Center extender abilities, and a marketplace with no publishing gauntlet for music, games and apps, and real devices available by Oct 2010.  A full marketplace in Canada is a must!
  • ‘Visual Mobile 2010 Express Edition’ – I don’t need this since I’ll have VS 2010 Premium/Ultimate, but Microsoft needs to release a free mobile development tool (with Silverlight designer support presumably) that will launch 100,000 apps
  • Decent replacement for the Live Framework (which was withdrawn) including free Bing Maps API usage up to a decent ceiling
  • Consistent mobile and desktop Live Services and Framework experience
  • Live Mesh Release (with better-than-hopeless user documentation)
  • Better free and on-ramp deal pricing for the Windows Azure Platform
  • Microformat & Live Clipboard SDK for IE 9.  Ray Ozzie blogged and tinkered (including a subsequent SDK and runtime) about ‘Wiring the Web’ with a ‘Live Clipboard’ using microformats back in 2006.  Skip to the present day and go to this Microsoft Canada blog post this week promoting my Mix session proposals (thanks Joey) and look at the Mix-powered Microformat tool that appears at the top left ;-)  Accelerators for IE8 just didn’t do it.
  • Oh, and of course for one or more of my Mix sessions (see below) to be selected by the public through the voting that ends tonight! 🙂

Not too much to ask for right?

 

Looking to learn more about the Windows Azure Platform, Silverlight, Windows Touch or Windows Identify Foundation?

If you’d like to see these sessions in person at Microsoft’s Mix 2010 conference or the recordings that will likely be made available for free later on, please vote for the sessions before January 15th 2010, by going to the site, adding the 3 sessions to you ballot and submitting it

Quick Tip to Retain ASP.NET dynamic Recompile Changes During Azure Debugging

One of the major benefits of the Windows Azure SDK and Windows Azure Tools for Microsoft Visual Studio is the ability to operate an Azure Development Fabric and Azure Development Storage emulation on your local machine.  Pressing F5 in a Visual Studio Azure project deploys and runs your application in an environment emulating the real Windows Azure.  This is somewhat similar to the ASP.NET Development Server that starts up and somewhat emulates IIS (or other ASP.NET hosting environment) when you debug an ASP.NET application.

This is great, but can also create an efficiency bottleneck during development.

If you are debugging a regular ASP.NET application and you need to make changes, in many cases you can stay in debug mode;  you can edit an ASPX page or other assets (including CSS) and simply refresh the already running web browser instance to see you changes.  In the case of the ASPX page having changed, it is dynamically recompiled when you next access it.  This works because the ASP.NET Development Server is pointing at your development files.  This efficient cycle is lost when you start debugging with the Windows Development Fabric…

When you debug (or run) an Azure project in Visual Studio, your application is built into a package that is ‘deployed’ to the local Azure emulation.  The emulation therefore uses its own copy of the files.  Making changes to the file in Visual Studio while debugging, only updates the copies under Visual Studio’s control – they are neither repackaged nor redeployed.  The Azure Development Fabric has no idea that they’ve changed.  So, you have to stop debugging, re-launch, wait for the new package to build and deploy, and then get back to the same place in the application.

Here’s what likely amount to a satisfactory workaround before you are ready for the emulation stage:  create a mock for the Data Access Layer that deals with Windows Azure Storage and debug your project outside of the Azure Development emulation.  You lose role instance configuration, but you can go back to using the ASP.NET Development Server.  You may also lose the Azure diagnostic logging – but you can mock that too.

Note:  If you are using SQL Azure, for the most part you can use a regular local SQL Server in place.

I’m not suggesting you mock the interfaces to the Azure APIs directly.  Your application may well have a Data Access Layer which (when following best practices), will likely have an interface.  I suggest you create an object to mock this using the same interface or perhaps use one of the available mocking frameworks.

To facilitate the selection of the polymorphic DAL objects (real or mock) in one code-base you’ll likely want to make your application smart enough to know if it’s running in the Azure environment (Microsoft or local emulation) or not.  My previous post on the topic explains this idea.

You’ll need an interface that suits the operations of your DAL such as this example (where WallMessage is defined elsewhere)…

image

a mock object (such as the example shown below) and a real object (not shown) that both implement the interface…

image

and then calling code that first selects the mock or real object to use based on environment detection……

image

after which you go on to use properties and methods on the selected interface oblivious to the implementation being used.

This is just a basic example and as mentioned, there are some very good Mocking frameworks as well as Dependency Injection frameworks available that ease this kind of setup in large or complex solutions.

The code shown is from the sample Silverlight application I provide (including source code) in my Azure Deployment Guide, already used by 100s of people to successfully deploy their first Azure application.

I hope you found this tip useful 🙂 If so, and you’re reading this on January 15th 2010 then please vote for my Mix 2010 conference submissions today! :)  Thank you.

Looking to learn more about the Windows Azure Platform, Silverlight, Windows Touch or Windows Identify Foundation?

If you’d like to see these sessions in person at Microsoft’s Mix 2010 conference or the recordings that will likely be made available for free later on, please vote for the sessions before January 15th 2010, by going to the site, adding the 3 sessions to you ballot and submitting it

Quick Tip for Detecting Azure Environment During Your Development Cycle

Given the Azure tools and portal for Visual Studio 2008, there are 4 likely modes of execution during your development cycle:

  1. Testing in Visual Studio (with no Azure environment)
  2. Testing in Visual Studio using the local Azure Development Fabric/Storage
  3. Staged on the Azure portal
  4. Deployed on the Azure portal

#1 mostly applies when you have an application that you are porting, or you are development components that may become part of the application.

At times you may find it useful to know if you are running inside or outside of the Azure environment (i.e. 1 vs. 2/3/4).

image

Providing you have included the appropriate assembly reference, you can use this API to detect your environment and take appropriate action while developing and testing your components at the stages above.

Looking to learn more about the Windows Azure Platform, Silverlight, Windows Touch or Windows Identify Foundation?

If you’d like to see these sessions in person at Microsoft’s Mix 2010 conference or the recordings that will likely be made available for free later on, please vote for the sessions before January 15th 2010, by going to the site, adding the 3 sessions to you ballot and submitting it

Just 2 days Left to Vote for Mix 2010 sessions on Azure, Silverlight, Windows Touch and WIF

Voting ends on Friday for sessions at Microsoft’s Mix 2010 conference.

I’ve submitted 3 session proposals – see the details – around Windows Touch, Azure & Silverlight which were all accepted into the voting list.  My session on Windows Identity Foundation has already been scheduled for the renowned high-level DevTeach 2010 conference at Microsoft’s Canadian HQ this March.

If you’d like to see my sessions in person at Mix or the recordings that will likely be made available for free then please vote for them.

You can find a list of all sessions available for voting  (a list that periodically shuffles its order), or go directly to my sessions here.  You can pick a total of 5 sessions to vote for.

Quick Tip for Hosting Services for Silverlight on Windows Azure

The Silverlight and Azure Tools for Visual Studio 2008 SP1 both provide convenient means to get going with these respective technologies.

Windows Azure is a good place to host services that your Silverlight application may call, as well as the web application that contains the Silverlight application itself.

However, if you’ve tried to get this to work, then you may have encountered as issue.

Let’s say you’ve added a Web Role to your Azure application to host the Silverlight application, and you want to add a Silverlight-enabled WCF Service to the website.  You may have tried using the Add New Item dialog to select the “Silverlight-enabled WCF Service” like this…

image

Unfortunately (for reasons I may go into in the future), the Azure platform isn’t going to correctly publish this WCF service in the web role, and your Silverlight app may be able to talk to the service just fine in your development environment, but not when you deploy to Azure proper.

The easiest workaround for this is to create a WCF Role, instead of a Web Role and then add your website files and Silverlight application to that.  The WCF Role exposes a service that is compatible with Silverlight on Azure.

If you check out my Guest Wall application (including source code), you’ll see this in action as well as the code :).

Looking to learn more about the Windows Azure Platform, Silverlight, Windows Touch or Windows Identify Foundation?

If you’d like to see these sessions in person at Microsoft’s Mix 2010 conference or the recordings that will likely be made available for free later on, please vote for the sessions before January 15th 2010, by going to the site, adding the 3 sessions to you ballot and submitting it

Microsoft Announcements at Steve Ballmer’s CES 2010 Keynote

This post was updated live during the keynote on Wed Jan 6th 2010.

All the juicy information on the keynote is below, but first, I’d appreciate 30 seconds of your time to support my session submissions for the Microsoft Mix 2010 conference by voting (follow the link, select a session, add to ballot, back up to repeat for other sessions, then submit the ballot)…

Looking to learn more about the Windows Azure Platform, Silverlight, Windows Touch or Windows Identify Foundation?

If you’d like to see these sessions in person at Microsoft’s Mix 2010 conference or the recordings that will likely be made available for free later on, please vote for the sessions before January 15th 2010, by going to the site, adding the 3 sessions to you ballot and submitting it

Background

Microsoft’s focus for the event

  • Screens woven into the fabric of our lives, specifically the PC and TV experience
  • Cloud – approach that combines power of client and cloud
  • Natural User Interface

Product/Service Announcements

  • Bing will be default search engine and MSN default homepage for HP PCs in 42 countries
  • HTC HD2 with Windows Mobile 6.5 exclusively through T-Mobile in the USA (more to say at phones at Mobile World Congress in Feb 2010)
  • Mediaroom 2.0 – IPTV solution for service providers such as AT&T U-verse (US) – live & on-demand content to the TV, Xbox, PCs (Windows 7 Media Center) & Windows Phones, enabled with Silverlight and IIS Smooth Streaming.
  • New slate form-factor PCs with Windows 7 coming in 2010 – will anyone notice once Apple releases their tablet?
  • Lots of cool games coming to Xbox in 2010 inc. Halo Reach (prequel to Halo 1), Crackdown 2, Fable III, COD MW2 content packs & Alan Wake (… a psychological action thriller delivered in episodes)
  • Xbox Game Room will have 1000+ old-style arcade games (with collaborative showdown) released over next 3 years starting in the spring.
  • Xbox Project Natal scheduled to be available for the 2010 holiday season – no new demos however

Demos/Information

  • Video of Seth Meyers from Saturday Night Live ‘thanking technology for everything is does for us’ (with comic sarcasm) – not sure there was a point to that…
  • There are 39 Million Xbox consoles around the world and 500M+ games sold ($20B+)
  • 11 Million users brought to Bing
  • Windows 7 had 3000 engineers, 50,000 partners and 8M beta testers
  • PC sales jumped nearly 50% the week Windows 7 launched
  • In 2009 33% more PCs sold at US ‘Black Friday’ (traditional retail sale day) than in 2008
  • Windows 7 All-in-ones shown: Lenovo A300 (thinnest on market), Medion Touch, Sony Vaio L
  • Windows 7 Laptops shown: Asus, MSI, Dell Adamo XPS (thinnest on market with very fast wake-up), Asus NX90
  • Windows 7 (‘full version’) Netbooks shown:  Lenovo, Acer, etc.
  • Windows 7 Gaming machines shown: HP NV15, Toshiba X505
  • Windows 7 3D Gaming machine shown: ASUS G51
  • Windows 7 on small-form-factor PCs shown: Acer Aspire Revo, Dell Xeno HD
  • Demo of digital reader with interaction and text to voice – Blio (1M+ books)
  • Demo of SkyDrive, Device Stage, Windows Live & Office 2010
  • Demo of Bing Maps zooming in with StreetSide
  • Demo of Ceton CableCard in Windows 7 Media Center – recording 4 HD channels at one time
  • Demo of what else you can do on the PC while watching TV on it (playing in background) – with horrible audio ‘stuck-record’ repeating glitch ;).
  • Demo of Windows 7 Media Center front-end to CBS (US TV Network)
  • Windows 7 slate form-factors coming in 2010 shown: Pegatron, Arcos & HP, with a video of the HP prototype specifically, with Kindle software running
  • Another Seth Meyes video on the ‘history of things that have happened in technology’… yawn…
  • Between Xmas and new year Xbox LIVE had 2.2M concurrent members online – busiest week ever
  • 20M+ Xbox Live users
  • A new member each second last week
  • A 1/3 of Xbox Live time is on non-gaming activities like buying extra songs, avatar clothing, etc.
  • 20M people have logged into Facebook, Twitter & last.fm, via Xbox Live
  • Demo of Xbox doing HD movie rental with instant streaming

Get further basic details from the MS press release, as well as a further press release about Natural User Interfaces (Voice, Touch and Beyond)

Azure Platform Billing On-Ramp

Here’s the timeline for the ramp-up of Azure Platform Services billing:

  • Jan 4 2010 – CTP accounts can be upgraded to commercial accounts in the countries listed below.
  • Feb 1 2010 – Billing starts for upgraded accounts and non-upgraded accounts are disabled with Windows Azure Storage going read-only and no new database creation in SQL Azure. 
  • Mar 1 2010 – non-upgraded SQL Azure databases will be deleted
  • Apr 1 2010 – non-upgraded Windows Azure Storage will be deleted

Available Azure Platform Services billing locations from Jan 2010

  • Austria
  • Belgium
  • Canada
  • Denmark
  • Finland
  • France
  • Germany
  • Ireland
  • India
  • Italy
  • Japan
  • Netherlands
  • New Zealand
  • Norway
  • Portugal
  • Singapore
  • Spain
  • Sweden
  • Switzerland
  • United Kingdom
  • United States