Released: SnowMaker – a unique id generator for Azure (or any other cloud hosting environment)

What it solves

Imagine you’re building an e-commerce site on Azure.

You need to generate order numbers, and they absolutely must be unique.

A few options come to mind initially:

  • Let SQL Azure generate the numbers for you. The downside to this approach is that you’re now serializing all of your writes down to a single thread, and throwing away all of the possible benefits from something like a queuing architecture. (Sidenote: on my current project we’re using a NoSQL graph DB with eventual consistency between nodes, so this wouldn’t work for us anyway.)
  • Use a GUID. These are far from human friendly. Seriously, can you imagine seeing an order form with a GUID on the top?
  • Prefix numbers with some form of machine specific identifier. This now requires some way to uniquely identify each node, which isn’t very cloud-like.

As you can see, this gets complex quickly.

SnowMaker is here to help.

What it does

SnowMaker generates unique ids for you in a highly distributed and highly performant way.

  • Ids are guaranteed to be unique, even if your web/worker role crashes.
  • Ids are longs (and thus human readable).
  • It requires absolutely no node-specific configuration.
  • Most id generation doesn’t even require any off-box communication.

How to get it

Library: Install-Package SnowMaker

(if you’re not using NuGet already, start today)

Source code: hg.tath.am/snowmaker or github.com/tathamoddie/snowmaker

How to use it

var generator = new UniqueIdGenerator(cloudStorageAccount);
var orderNumber = generator.NextId("orderNumbers");

The only caveat not shown here is that you need to take responsibility for the lifecycle of the generator. You should only have one instance of the generator per app domain. This can easily be done via an IoC container or a basic singleton. (Multiple instances still won’t generate duplicates, you’ll just see wasted ids and reduced performance.) Don’t create a new instance every time you want an id.

Other interesting tidbits

The name is inspired by Twitter’s id generator, snowflake. (Theirs is more scalable because it is completely distributed, but in doing so it requires node-specific configuration.)

Typical id generation doesn’t even use any locks, let alone off-box communication. It will only lock and talk to blob storage when the id pool has been exhausted. You can control how often this happens by tweaking the batch size (a property on the generator). For example, if you are generating 200 order ids per server per second, set the batch size to 2000 and it’ll only lock every 10 seconds.

Node synchronisation is done via Azure blob storage. Other than that, it can run anywhere. You could quite easily use this library from AppHarbor or on premise hosting too, you’d just wear the cost of slightly higher latency when acquiring new ids batches.

The data persistence is swappable. Feel free to build your own against S3, Ninefold Storage, or any other blob storage API you can dream up.

The original architecture and code came from an excellent MSDN article by Josh Twist. We’ve brushed it off, packaged it up for NuGet and made it production ready.

Under the covers

SnowMaker allocates batches of ids to each running instance. Azure Blob Storage is used to coordinate these batches. It’s particularly good for this because it has optimistic concurrency checks supported via standard HTTP headers. At a persistence level, we just create a small text file for each id scope. (eg, the contents of /unique-ids/some-id-scope would just be “4”.)

One issue worth noting is that not all ids will always be used. Once a batch is checked out, none of the ids in it can ever be reallocated by SnowMaker. If a batch is checked out, only one id is used, then the process terminates, the remaining ids in that batch will be lost forever.

Here’s a sequence diagram for one client:

SequenceDiagram

Here’s a more complex sequence diagram that shows two clients interacting with the store, each using a different batch size:

Multiple clients

Released: FormsAuthenticationExtensions

What it does

Think about a common user table. You probably have a GUID for each user, but you want to show their full name and maybe their email address in the header of each page. This commonly ends up being an extra DB hit (albeit hopefully cached).

There is a better way though! A little known gem of the forms authentication infrastructure in .NET is that it lets you embed your own arbitrary data in the ticket. Unfortunately, setting this is quite hard – upwards of 15 lines of rather undiscoverable code.

Sounds like a perfect opportunity for another NuGet package.

How to get it

Library: Install-Package FormsAuthenticationExtensions

(if you’re not using NuGet already, start today)

Source code: formsauthext.codeplex.com

How to use it

Using this library, all you need to do is add:

 using FormsAuthenticationExtensions; 

then change:

 FormsAuthentication.SetAuthCookie(user.UserId, true); 

to:

 var ticketData = new NameValueCollection {
    { "name", user.FullName },
    { "emailAddress", user.EmailAddress }
 };
new FormsAuthentication().SetAuthCookie(user.UserId, true, ticketData);

Those values will now be encoded and persisted into the authentication ticket itself. No need to store it in any form of session state, custom cookies or extra DB calls.

To read the data out at a later time:

 var ticketData = ((FormsIdentity) User.Identity).Ticket.GetStructuredUserData();
var name = ticketData["name"];
var emailAddress = ticketData["emailAddress"];

If you want something even simpler, you can also just pass a string in:

 new FormsAuthentication().SetAuthCookie(user.UserId, true, "arbitrary string here"); 

and read it back via:

 var userData = ((FormsIdentity) User.Identity).Ticket.UserData; 

Things to Consider

Any information you store this way will live for as long as the ticket.

That can be quite a while if users are active on your application for long periods of time, or if you give out long-term persistent sessions.

Whenever one of the values stored in the ticket needs to change, all you need to do is call SetAuthCookie again with the new data and the cookie will be updated accordingly. In our user name / email address example, this is actually quite advantageous. If the user was to update their display name or email address, we’d just update the ticket with new values. This updated ticket would then be supplied for future requests. In web farm environments this is about as perfect as it gets – we don’t need to go back to the DB to load this information for each request, yet we don’t need to worry about invalidating the cache across machines. (Any form of shared, invalidatable cache in a web farm is generally bad.)

Size always matters.

The information you store this way is embedded in the forms ticket, which is then encrypted and sent back to the users browser. On every single request after this, that entire cookie gets sent back up the wire and decrypted. Storing any significant amount of data here is obviously going to be an issue. Keep it to absolutely no more than a few simple values.

Web Forms Model-View-Presenter on Hanselminutes

Over the last few months Damian Edwards and myself have been spending quite a bit of time building out a Model-View-Presenter framework for ASP.NET Web Forms.

Until now we’ve been pretty quiet about it all on our blogs because we were busy polishing off v1 and trying to get all the documentation in order. Nevertheless, the word has definitely started to spread as Scott Hanselman interviewed me about the library on this week’s Hanselminutes episode.

Listen to the podcast

Learn more about the library

Video: Building Fast, Public Websites

Following up from my last post about the ASP.NET MVC vs ASP.NET WebForms debate, we’ve had a second TechTalk posted, also from TechEd Australia. In this video, Michael Kordahi, Damian Edwards and I sat down to discuss building fast, public websites. It was a bit of a teaser for our breakout session at the conference, which will be available online as a screencast in the next week or two.

If you’re interested in learning more about building large public websites on ASP.NET, remember that the full video from our recent REMIX session is still available online too.

Building Fast, Public Websites

Watch Online or Download

Video: ASP.NET MVC vs ASP.NET WebForms – Will WebForms be replaced by MVC?

At the recent TechEd Australia conference, Paul Glavich, Damian Edwards and myself sat down to discuss what we thought about the current MVC vs WebForms debate. Our TechTalk has now been posted on the TechEd Online site, and available for anyone to watch.

Check it out, and feel free to continue the debate with any of us. 🙂

ASP.NET MVC vs ASP.NET WebForms – Will WebForms be replaced by MVC? 

Watch Online or Download

Accessing ASP.NET Page Controls During PreInit

If you’ve read my previous post explaining a common pitfall with view state, I’d hope you’re preparing all your controls in the Init event of the page/control lifecycle.

Even if I’m not reusing them through my application much, I like to factor elements like a drop down list of countries into their own control. This centralizes their logic and allows us to write clear, succinct markup like this:

<tat:CountriesDropDownList ID="AddressCountry" runat="server" />

The code for a control like this is quite simple:

[ToolboxData("<{0}:CountriesDropDownList runat=\"server\" />")]
public class CountriesDropDownList : DropDownList
{
    protected override void OnInit(EventArgs e)
    {
        DataSource = Countries;
        DataBind();

        base.OnInit(e);
    }
}

The Problem

Once you start using this encapsulation technique, it won’t be long until you want to pass in a parameter that affects the data you load. Before we do, we need to be aware that the Init event is fired in reverse order. That is, the child controls have their Init event fired before that event is fired at the parent. As such, the Page.Init event is too late for us to set any properties on the controls.

The natural solution is to try and use the Page.PreInit event, however when you do you’ll often find that your control references are all null. This happens when your page is implemented using a master page, and it relates to how master pages are implemented. The <asp:ContentPlaceHolder /> controls in a master page use the ITemplate interface to build their contents. This content (child controls) is not usually prepared until the Init event is called, which means the control references are not available. For us, this represents a problem.

The Solution

The fix is remarkably simple; all we need to do is touch the Master property on our Page and it will cause the controls to become available. If we are using nested master pages, we need to touch each master page in the chain.

I often create a file called PageExtensions.cs in my web project and add this code:

public static class PageExtensions
{
    /// <summary>
    /// Can be called during the Page.PreInit stage to make child controls available.
    /// Needed when a master page is applied.
    /// </summary>
    /// <remarks>
    /// This is needed to fire the getter on the top level Master property, which in turn
    /// causes the ITemplates to be instantiated for the content placeholders, which
    /// in turn makes our controls accessible so that we can make the calls below.
    /// </remarks>
    public static void PrepareChildControlsDuringPreInit(this Page page)
    {
        // Walk up the master page chain and tickle the getter on each one
        MasterPage master = page.Master;
        while (master != null) master = master.Master;
    }
}

This adds an extension method to the Page class, which then allows us to write code like the following:

protected override void OnPreInit(EventArgs e)
{
    this.PrepareChildControlsDuringPreInit();

    MyCustomDropDown.MyProperty = "my value";

    base.OnPreInit(e);
}

Without the call to the extension method, we would have received a NullReferenceException when trying to set the property value on the MyCustomDropDown control.

You now have one less excuse for preparing your controls during the Load event. 🙂

How I Learned to Stop Worrying and Love the View State

(If you don’t get the title reference, Wikipedia can explain. A more direct title could be: Understanding and Respecting the ASP.NET Page Lifecycle.)

This whole article needs a technical review. Parts of it are misleading. I’ll get back to you Barry.

Page lifecycle in ASP.NET is a finicky and rarely understood beast. Unfortunately, it’s something that we all need to get a handle on.

A common mishap that I see is code like this:

protected void Page_Load(object sender, EventArgs e)
{
    if (!Page.IsPostBack)
    {
        AddressCountryDropDown.DataSource = CountriesList;
        AddressCountryDropDown.DataBind();
    }
}

The problem here is that we’re clogging our page’s view state. Think of view state as one of a page’s core arteries, then think of data like cholesterol. A little bit is all right, but too much is crippling.

To understand the problem, lets investigate the lifecycle that’s in play here:

  1. The Page.Init event is being fired, however we are not subscribed to that.
  2. Immediately after the Init event has fired, view state starts tracking. This means that any changes me make from now on will be saved down to the browser and re-uploaded on the next post back.
  3. The Page.Load event is being fired in which we are setting the contents of the drop down list. Because we are doing this after the view state has started tracking, every single entry in the drop down is being written to both the HTML and the view state.

There’s yet another problem here as well. By the time the Page.Load event is fired, all of the post back data has been loaded and processed.

To investigate the second problem, let’s investigate the lifecycle that’s in play during a post back of this same page:

  1. The user triggers the post back from their browser and all of the post back data and view state is uploaded to the server.
  2. The Page.Init event is fired, however we are not subscribed to that.
  3. Immediately after the Init event has fired, view state starts tracking. This means that any changes me make from now on will be saved down to the browser and re-uploaded on the next post back.
  4. The view state data is loaded for all controls. For our drop down list example, this means the Items collection is refilled using the view state that was uploaded from the browser.
  5. Post back data is processed. In our example, this means the selected item is set on the drop down list.
  6. The Page.Load event is fired however nothing happens because the developer is checking the Page.IsPostBack property. Usually, they say this is a “performance improvement” however it is also required in this scenario otherwise we would lose the selected item when we rebound the list.
  7. The contents of the drop down list are once again written to both the HTML and the view state.

How do we do this better? Removing the IsPostBack check and placing the binding code into the Init event is all we need to do:

protected override void OnInit(EventArgs e)
{
    AddressCountryDropDown.DataSource = CountriesList;
    AddressCountryDropDown.DataBind();

    base.OnInit(e);
}

What does this achieve?

  • We are filling the contents of the drop down before the Init event is fired; therefore a redundant copy of its contents is not written to the view state.
  • We are filling the contents of the drop down before the postback data is processed, so our item selection is successfully loaded without it being overridden later.
  • We have significantly reduced the size of the page’s view state.

This simple change is something that all ASP.NET developers need to be aware of. Unfortunately so many developers jumped in and wrote their first ASP.NET page using the Page_Load event (including myself). I think this is largely because it’s the one and only event handler presented to us when we create a new ASPX page in Visual Studio. While this makes the platform appear to work straight away, it produces appalling results.

Video: The best web stack in the world

This is the demo I made for the Microsoft Demos Happen Here competition. The criteria was that it had to be under 10 minutes, and show one or more of the features from the launch wave technologies – Visual Studio 2008, SQL Server 2008 and Windows Server 2008.
I chose to demonstrate PHP running on IIS7 using FastCGI, then integrating this with an ASP.NET application before finally load balancing the whole application between two servers in a high availability, automatic failover cluster.
I managed to do all that with 2 minutes to spare, so I think it’s pretty clear that Windows Server 2008 is the best web stack in the world.

http://vimeo.com/1439786

Update 5-Sep-08: I won. 🙂 

Tearing down the tents (and moving them closer together)

Being fairly focused on Microsoft technologies myself, I see a lot of the “us vs. them” mentality where you either use Microsoft technologies, or you’re part of “the other group”. Seeing Lachlan Hardy at Microsoft Remix was awesome – he was a Java dude talking about web standards at a Microsoft event. The more we can focus on the ideas rather than which camp you’re from, the more we’ll develop the inter-camp relationships and eventually destroy this segmentation. Sure, we’ll still group up and debate the superfluous crap like which language is better (we’re nerds – we’ll always do that) but at least these will be debates between the sub-camps of one big happy web family. (It’s not as cheesy as it sounds – I hope.)

What’s the first step in making this happen? Meet people from “the other group”!

The boys and girls at Gruden and Straker Interactive have put together Web on the Piste for the second year running. It’s a vendor neutral conference about rich internet technologies – so you’ll see presentations about Adobe Flex and Microsoft Silverlight at the same event (among lots of other cool technologies of course). These types of events are a perfect way to meet some really interesting people and cross pollinate some sweet ideas.

It’s coming up at the end of August, and I understand that both conference tickets and accommodation are getting tight so I’d encourage you to get in soon if you’re interested (Queenstown is crazy at this time of year).

And of course, yours truly will be there evangelising the delights of Windows Live as well as ASP.NET AJAX to our Flash using, “fush and chups” eating friends. 🙂

Will you be there?