Released: ReliabilityPatterns – a circuit breaker implementation for .NET

How to get it

Library: Install-Package ReliabilityPatterns (if you’re not using NuGet already, start today)

Source code: hg.tath.am/reliability-patterns

What it solves

In our homes, we use circuit breakers to quickly isolate an electrical circuit when there’s a known fault.

Michael T. Nygard introduces this concept as a programming pattern in his book Release It!: Design and Deploy Production-Ready Software.

The essence of the pattern is that when one of your dependencies stops responding, you need to stop calling it for a little while. A file system that has exhausted its operation queue is not going to recover while you keep hammering it with new requests. A remote web service is not going to come back any faster if you keep opening new TCP connections and mindlessly waiting for the 30 second timeout. Worse yet, if your application normally expects that web service to respond in 100ms, suddenly starting to block for 30s is likely to deteriorate the performance of your own application and trigger a cascading failure.

Electrical circuit breakers ‘trip’ when a high current condition occurs. They then need to be manually ‘reset’ to close the circuit again.

Our programmatic circuit breaker will trip after an operation has more consecutive failures than a predetermined threshold. While the circuit breaker is open, operations will fail immediately without even attempting to be executed. After a reset timeout has elapsed, the circuit breaker will enter a half-open state. In this state, only the next call will be allowed to execute. If it fails, the circuit breaker will go straight back to the open state and the reset timer will be restarted. Once the service has recovered, calls will start flowing normally again.

Writing all this extra management code would be painful. This library manages it for you instead.

How to use it

Taking advantage of the library is as simple as wrapping your outgoing service call with circuitBreaker.Execute:

// Note: you'll need to keep this instance around
var breaker = new CircuitBreaker();

var client = new SmtpClient();
var message = new MailMessage();
breaker.Execute(() => client.SendEmail(message));

The only caveat is that you need to manage the lifetime of the circuit breaker(s). You should create one instance for each distinct dependency, then keep this instance around for the life of your application. Do not create different instances for different operations that occur on the same system.

(Managing multiple circuit breakers via a container can be a bit tricky. I’ve published a separate example for how to do it with Autofac.)

It’s generally safe to add this pattern to existing code because it will only throw an exception in a scenario where your existing code would anyway.

You can also take advantage of built-in retry logic:

breaker.ExecuteWithRetries(() => client.SendEmail(message), 10, TimeSpan.FromSeconds(20));

Why is the package named ReliabilityPatterns instead of CircuitBreaker?

Because I hope to add more useful patterns in the future.

This blog post in picture form

Sequence diagram

Released: SnowMaker – a unique id generator for Azure (or any other cloud hosting environment)

What it solves

Imagine you’re building an e-commerce site on Azure.

You need to generate order numbers, and they absolutely must be unique.

A few options come to mind initially:

  • Let SQL Azure generate the numbers for you. The downside to this approach is that you’re now serializing all of your writes down to a single thread, and throwing away all of the possible benefits from something like a queuing architecture. (Sidenote: on my current project we’re using a NoSQL graph DB with eventual consistency between nodes, so this wouldn’t work for us anyway.)
  • Use a GUID. These are far from human friendly. Seriously, can you imagine seeing an order form with a GUID on the top?
  • Prefix numbers with some form of machine specific identifier. This now requires some way to uniquely identify each node, which isn’t very cloud-like.

As you can see, this gets complex quickly.

SnowMaker is here to help.

What it does

SnowMaker generates unique ids for you in a highly distributed and highly performant way.

  • Ids are guaranteed to be unique, even if your web/worker role crashes.
  • Ids are longs (and thus human readable).
  • It requires absolutely no node-specific configuration.
  • Most id generation doesn’t even require any off-box communication.

How to get it

Library: Install-Package SnowMaker

(if you’re not using NuGet already, start today)

Source code: hg.tath.am/snowmaker or github.com/tathamoddie/snowmaker

How to use it

var generator = new UniqueIdGenerator(cloudStorageAccount);
var orderNumber = generator.NextId("orderNumbers");

The only caveat not shown here is that you need to take responsibility for the lifecycle of the generator. You should only have one instance of the generator per app domain. This can easily be done via an IoC container or a basic singleton. (Multiple instances still won’t generate duplicates, you’ll just see wasted ids and reduced performance.) Don’t create a new instance every time you want an id.

Other interesting tidbits

The name is inspired by Twitter’s id generator, snowflake. (Theirs is more scalable because it is completely distributed, but in doing so it requires node-specific configuration.)

Typical id generation doesn’t even use any locks, let alone off-box communication. It will only lock and talk to blob storage when the id pool has been exhausted. You can control how often this happens by tweaking the batch size (a property on the generator). For example, if you are generating 200 order ids per server per second, set the batch size to 2000 and it’ll only lock every 10 seconds.

Node synchronisation is done via Azure blob storage. Other than that, it can run anywhere. You could quite easily use this library from AppHarbor or on premise hosting too, you’d just wear the cost of slightly higher latency when acquiring new ids batches.

The data persistence is swappable. Feel free to build your own against S3, Ninefold Storage, or any other blob storage API you can dream up.

The original architecture and code came from an excellent MSDN article by Josh Twist. We’ve brushed it off, packaged it up for NuGet and made it production ready.

Under the covers

SnowMaker allocates batches of ids to each running instance. Azure Blob Storage is used to coordinate these batches. It’s particularly good for this because it has optimistic concurrency checks supported via standard HTTP headers. At a persistence level, we just create a small text file for each id scope. (eg, the contents of /unique-ids/some-id-scope would just be “4”.)

One issue worth noting is that not all ids will always be used. Once a batch is checked out, none of the ids in it can ever be reallocated by SnowMaker. If a batch is checked out, only one id is used, then the process terminates, the remaining ids in that batch will be lost forever.

Here’s a sequence diagram for one client:

SequenceDiagram

Here’s a more complex sequence diagram that shows two clients interacting with the store, each using a different batch size:

Multiple clients

Tearing down the tents (and moving them closer together)

Being fairly focused on Microsoft technologies myself, I see a lot of the “us vs. them” mentality where you either use Microsoft technologies, or you’re part of “the other group”. Seeing Lachlan Hardy at Microsoft Remix was awesome – he was a Java dude talking about web standards at a Microsoft event. The more we can focus on the ideas rather than which camp you’re from, the more we’ll develop the inter-camp relationships and eventually destroy this segmentation. Sure, we’ll still group up and debate the superfluous crap like which language is better (we’re nerds – we’ll always do that) but at least these will be debates between the sub-camps of one big happy web family. (It’s not as cheesy as it sounds – I hope.)

What’s the first step in making this happen? Meet people from “the other group”!

The boys and girls at Gruden and Straker Interactive have put together Web on the Piste for the second year running. It’s a vendor neutral conference about rich internet technologies – so you’ll see presentations about Adobe Flex and Microsoft Silverlight at the same event (among lots of other cool technologies of course). These types of events are a perfect way to meet some really interesting people and cross pollinate some sweet ideas.

It’s coming up at the end of August, and I understand that both conference tickets and accommodation are getting tight so I’d encourage you to get in soon if you’re interested (Queenstown is crazy at this time of year).

And of course, yours truly will be there evangelising the delights of Windows Live as well as ASP.NET AJAX to our Flash using, “fush and chups” eating friends. 🙂

Will you be there?

Location, Location, Location: My plan for location awareness, and the GeographicLocationProvider object

I know where I am. My phone knows where it is. Why doesn’t the webpage know where I am?

Think about these scenarios which will become more and more prominent as “the mobile web” starts to prevail:

  1. I visit a cinema website on my mobile. Rather than having to choose my cinema first, the website already knows which suburb I’m in so it defaults to the nearest cinema first.
  2. I need a taxi. I don’t know where I am, but my phone does. I want to be able to book a taxi and have the website discover my location automatically.

The key idea that I’m exploring here is the ability for a webpage to access this information through the standard browser interface.

I have a plan for making this a reality.

Windows Mobile has already taken a step towards baking location awareness into the OS with their GPS Intermediate Driver. The idea is that the operating system handles communication with the GPS unit, including all the various protocols. Applications then have a unified API for accessing GPS data. This proxy effect also facilitates non-exclusive access to the GPS.

But this doesn’t go far enough. Even with this unified API, very few applications are actually location aware. More importantly, I don’t want to have to download and install a piece of software on my device just to be able to see movie times. It’s just not going to happen.

We’ve also been making the assumption that location data comes from a GPS. Enter GeoPriv.

With the continuing rollout of VOIP, there are obvious challenges about the loss of location awareness. The current analog network makes call tracing relatively easy. It’s a fixed line of copper and the phone company knows where it terminates. This is a legal requirement for emergency call routing, as well as being immensely useful for scenarios such as a national number auto-routing to your nearest local store. Both of these scenarios become immensely difficult when you can’t even rely on there being a physical phone anymore – a piece of software with a network connection is now a fully fledged communication device that needs to support these scenarios somehow.

There’s an IETF working group tasked to solve this exact problem. The privacy impacts of sharing location data are so important that it’s in the name. They are the “Geographic Location/Privacy working group”, or “GeoPriv“. The best part is, they are living in a reality and delivering useful technology – and fast.

There are a number of key concepts they have identified:

  • We can’t go jamming a GPS chip in every single device we manufacture. We need to be able to capitalize on the ecosystem surrounding our existing devices to surface the information we already have.
  • Privacy is a critical element of any solution expecting wide spread adoption and trust.
  • There are two possible types of location you could need:
    • civic location (level, street number, street, suburb, etc)
    • geographic location (latitude, longitude, elevation)

Lets step away from mobile devices briefly and consider the laptop I’m writing this post on. My laptop doesn’t know where it is. Neither does my WiFi router, or my DSL modem. My ISP does though.

At some stage in the future, my modem will start receiving an extra DHCP option. In the same  way that my ISP supplies me with network settings like DNS when I connect, they will also start pushing out the address of their Location Information Server. My DSL modem will then push this setting out across my network. Finally, my laptop will be able to query this service to find out my current civic and/or geographic location. The privacy controls around this are beyond the scope of this post.

By asking the service provider for the information, these same techniques also works for mobile devices, 3G data connections, and all those other wonderful wireless technologies. Cell-based triangulation is already in use by phone companies around the world, including our national carrier here in Australia, however the interfaces are in no way standardized. The Location Information Server (LIS) and the HTTP Enabled Location Delivery protocol (HELD) solve this problem.

Now that our device is capitalising on the network ecosystem, getting it into the browser is the easy part. All that’s left is a thin veneer of JavaScript.

Location awareness is only becoming an increasing demand. I want to start the process of rolling in the JS layer now, so that as the supporting technologies come to fruition, we have the access layer to make them useful.

Inline with the XMLHttpRequest object that we’ve all come to know and love, I’ve started writing a spec for a GeographicLocationProvider object.

With XMLHttpRequest, we can write code like this:

var client = new XMLHttpRequest();
client.onreadystatechange = function()
{
    if(this.readyState == 4 && this.status == 200)
        alert(this.responseXML);
}
client.open("GET", "http://myurl.com/path");
client.send();

I want to be able to write code like this:

var provider = new GeographicLocationProvider();
provider.onreadystatechange = function()
{
    if(this.readyState == 2)
        alert(this.geographic.latitude);
}
provider.start();

Again, usage is conceptually similar to the XMLHttpRequest object:

  1. Initialize an instance of the object
  2. Subscribe to the state change event
  3. Set it free

The potential states are:

  • STOPPED. This is the state the the object is initialized in, and the state that it returns to if stop() is called.
  • RESOLVING. The object has been started, but not location information is available yet. In this state the browser could be:
    • prompting the user for permission,
    • searching for location sources (like GPS hardware, or an LIS endpoint), or
    • waiting for the location source to initialize (like connecting to satellites, or talking to the LIS)
  • TRACKING. A location source has been found and location data is ready to be queried from the provider.
  • UNAVAILABLE. No location data is available, and none is likely to become available. The user may have denied a privacy prompt, their security settings may have automatically denied the request, or there may be no location sources available. It is possible for the provider to return to the RESOLVING state if a location source become available later.

In more complex scenarios, the provider can be primed with a specific request to aid in evaluation of privacy policies and selection of location sources. For example, browsers may choose to hand over state-level civic location data without a privacy prompt. This data could also be obtained from an LIS, without needing to boot up a GPS unit. If the webpage requested highly accurate geographic location data, the browser would generally trigger a privacy prompt and boot up the most accurate location source available.

While we’ve now simplified the developer experience, the complexity of the browser implementation has mushroom clouded. How do we reign this in so that it’s attractive and feasible enough for browser implementers? How do we demonstrate value today?

You might have noticed that in my discussion of the JS layer I drifted away from the GeoPriv set of technologies. While any implementation should be harmonious with the concepts developed by the GeoPriv working group, we aren’t dependent upon their technology to start delivering browser-integrated location awareness today.

There are numerous location sources which can be used:

  • Statically configured location – with the network fingerprinting technology already in Vista, it would be relatively easy to prompt users for their civic location the first time location data is needed on a particular network.
  • GPS Intermediate Driver – already rolled into the Windows Mobile platform.
  • Location Information Servers – can be added to the mix later as LIS deployments become prevalent. This is the only one that is GeoPriv dependant.

The civic and geographic schemas have already been delivered by the working group as RFC 4119. There has been an incredible amount of discussion involved in developing a unified schema that can represent civic addresses for anywhere in the world, and this schema should be adopted for consistency. (Do you know the difference between states, regions, provinces, prefectures, counties, parishes, guns, districs, cities, townships, shis, divisions, boroughs, wards, chous and neighbourhoods? They do.)

Who is responsible for delivering this unified location layer?

I keep talking about the browser being responsible for managing all these location sources. Other than the JS layer, all of this infrastructure is client independent, so why don’t we just make the browser a dumb proxy to a unified location service. This service should be a component of the operating system, accessible by software clients (like Skype) and webpages via the browser proxy.

Windows Mobile has already started in the right direction with their GPS Intermediate Driver, however this is only one element of a wider solution.

What do I want?

  1. I want to see a “Location” icon in the settings page of my mobile device, the Control Panel of my Vista laptop and the System Preferences panel of my Mac.
  2. I want the browsers to expose the GeopgraphicLocationProvider object for JS clients. (The start of this specification is online now already.)
  3. I want the browsers to proxy location requests to the operating system store, along with hints like which zone the website is in.
  4. I want the operating system store to be extensible, implementing its own provider model which allows 3rd party developers to supply their own location data from software or hardware services as new discovery mechanisms are developed. We shouldn’t have to wait for widespread adoption before sources are surfaced into the store, and this should be open to software developers as well as driver developers.
  5. I want the operating system store to be accessible by any application, including 3rd party browsers.

How am I going to make this happen?

Dunno.

Right now, I’m documenting the start of what will hopefully be a fruitful conversation. Participate in the conversation.

Managed Wrappers for Windows Live Data

Windows Live Data is the API you use for delegated access to your user’s personal data like contacts. It’s a pretty simple API, however that hasn’t stopped me writing some components for it! Today, I’m releasing them publicly.

Not only do these components make it easier to work with the API, but they also provide an abstraction layer so that as the API develops your application doesn’t necessarily have to.

(Note: This post assumes an understanding for the Windows Live Data API. If you’ve never touched it before, read this first.)

First up is the PermissionRequestHyperLink control. Placing this on your page gives you a nice designer experience for building those yucky URLs and setting all the right flags.

A basic request looks something like this:

<live:PermissionRequestHyperLink id=”PermissionRequestHyperLink1″ runat=”server” Permission=”LiveContacts_ReadOnly” PrivacyUrl=”~/WindowsLive/PrivacyPolicy.aspx” ReturnUrl=”~/WindowsLive/ResponseHandler.ashx”>Permission Request</live:permissionrequesthyperlink>

And gives you designer experience like this:

image

image

Next up is the response handler base class. Start by adding a new ‘generic handler’ to your project:

image

Change the generated class to inherit from PermissionResponseHandler instead of IHttpHandler, then implement the ProcessResponse and ProcessFailure methods like so:

public class ResponseHandler : PermissionResponseHandler
{
    protected override void ProcessResponse(HttpContext context, PermissionResponse response)
    {
        //Do something here like storing the token for future use
        //response.DomainAuthenticationToken
        //response.OwnerHandle
    }

    protected override void ProcessFailure(HttpContext context, PermissionResponseCode responseCode)
    {
        //Perform some nice handling here
        //responseCode
    }
}

How easy is that!

You can grab the code from http://svn.fueladvance2.com/FuelAdvance.Components/trunk/ (username: anonymous). You’ll find the components in the FuelAdvance.Components.Web.WindowsLive namespace.

If you’re using Subversion yourself, remember that you can configure this as an svn:external and then you’ll always be running the latest version.

Next up, I’ll probably be releasing some managed wrappers for the Windows Live Contacts API.

Update 9/8/07: Change SVN link to point at the solution instead of the project.