Effective software delivery–Tips for keeping your releases predictable

January 29th, 2012

In my last post I illustrated the advantages of short consistent release cycles but in order to achieve this, you often need to build your software in a way that helps ensure that releases cannot be missed.

  1. Support enabling / disabling features without deployment or rollback

    Having atomic feature configuration for new or risky features reduces the need for a rollback, helping safely deploy new or risky changes. This prevents the friction caused when a deployment containing 5 new features fails because of an undiscovered bug being found in a single feature. Rather than rolling back the code, costing you business confidence and a disrupted development flow, instead prefer disabling the feature and taking a story into your current iteration to fix the problem. You’d be doing this anyway, but rather deliver value with the other 4 features while you’re diagnosing the problems.

  2. Use tiny, frequently integrated feature branches to isolate development of features.

    Any form of long running feature branching should definitely be considered an anti-pattern because any un-integrated changes that diverge from your trunk can potentially cause problems with other features. However, small and short lived feature branches consisting of a day or half a days work on a single atomic feature can help you compose a build of integrated features excluding one that is causing problems in QA.

  3. Stick to MVP: Always cut features, never change your release date

    The temptation to fire fight a broken feature in a release is always strong, but it’s often the core reason of a delayed release. Frequently developers and product managers will want their new shiny feature in the release it was expected and will operate under the belief that “holding the release for an extra day” is the right thing to do. It’s never the right thing to do.

    By doing this, you prevent any other features in the release from delivering business value on time, and this is always the wrong decision. It’ll hurt, but you should always disable the feature and release, or compose a new build without that feature integrated to keep the steady canter of your release and development cycle.

    If you don’t do this, your next iteration will slip by the time it takes to fix the issue you’re currently facing. Rather, prefer pulling the feature and adding a story to fix it at the top of the next iteration.

    It’s a much more transparent and accountable way of dealing with slippage, otherwise people will expect a larger delivery than is possible come your *next* release, costing you trust.

  4. Deployment as a first class citizen: Automate your deployment and rollback

    As you tighten your development and release cycles, any form of manual deployment will become a chore on your development and operations team, possibly counterbalancing the benefits of a tighter focus on releasing. You should always treat deployment and installation as a first class citizen. All your software, from day one, should support continuous integration and automated deployment. If you configure software with these characteristics when it’s new and inert you can prevent any large arduous tasks configuring it later.

  5. Configure old and new behaviour side by side

    If you’re touching particularly sensitive or high traffic portions of your software, ensure you can switch the behaviour between the two implementations trivially and without rollback. Remove the configuration when the new code is “trusted” to prevent costly rollbacks and patches.

  6. De-scope existing bugs in order to ship

    If an existing bug is threatening your release, de-scope the bug fix to the next release or hotfix it later. Always ship if you’re adding value rather than wait for unconfirmed fixes.

Always prefer de-risking releases for the sake of consistency, it’s a more honest way to deliver software on time.

Effective software delivery–Short, consistent release cycles

January 29th, 2012

One of the greatest challenges facing any software company is the risk associated with new releases. As a responsible software development team, you should put just as much emphasis on delivery of the software as you do on development.

The most direct and deliberate way to mitigate the risks associated with software delivery is to stick to the mantra of “Release early, release often”. The surest way to accomplish this is to repeatedly build *and* deploy the minimum viable product and the smallest possible change. Doing this mitigates risk by reducing the impact of a rollback to “just a single feature”.

The reason that this is so important is that it allows you to stick to short and simple release dates, which encourages and enforces predictability into your software development. Your release cycles should be small and take place at the smallest sensible interval on a sliding scale between “as soon as the feature is QA’d” (which represents the least possible risk and the smallest possible change) to “as soon as the current iteration or sprint is complete” (which is often a fair compromise allowing a more coherent shipped change).

Ensuring that your releases happen on simple and consistent dates (for example “Every Tuesday”, “Every day at 7pm”, “Every time QA signs off a feature”) maximises transparency for the rest of your organisation and helps build trust in too-often-opaque software development departments. In addition to transparency you will also gain a tight appreciation for scripting your release processes and increasing your comfort in your deployments due to the increased and and consistent nature of installing software. This focus will quickly surface any points of friction in your software release process that need improvement.

iTunes Persistent Id Cloner – Sync one iPhone / iPod with many computers

January 25th, 2012

I frequently have trouble trying to copy music onto my iPod due to iTunes’s draconian binding and synchronising rules. These rules exist to ensure that only “authorised” files can be copied onto any given device. Apple implement this by binding your device to a specific instance of iTunes using a randomly generated Persistent Id that’s registered on your computer and on your device.

I have 3 computers that I use regularly, and I regularly rip CDs in different places and want to put that music on my iPod for travelling imminently. By default, iTunes restricts me from being able to do this.

Tonight I hacked together a little command line app to clone your persistent Id’s across multiple machines.

In order to use it, you need a copy of iTunes bound to your device installed on a machine. Once you have that, you run the program in –extract mode, which will show you the persistent Id for that machine. You can then use the program on other computers to patch any installed iTunes libraries to believe that they have the same persistent Id as the “master” machine. It’s a pretty trivial process.

If it breaks anything, don’t come crying, but “works on my machines”. Happy listening.

Source code / detailed instructions: https://github.com/davidwhitney/itunes-persistent-id-cloner
Executable: https://github.com/downloads/davidwhitney/itunes-persistent-id-cloner/4f7dd551998d515108d937d5b39f28a7b2a058ff.zip

The iTunes Persistent Id Cloner!

Small command line app to:

1) Extract the persistent Id from an iTunes library (this is the thing that your device is bound to)
2) Patch up an iTunes library with a persistent Id you supply (I bet you can see where this is going).

Usage:

Compile.

1) Quit iTunes on machine that you want to be considered your "master" persistent Id.
    I'd suggest the one you've already got your iPod / iPhone registered on.

2) Run itunes-persistent-id-cloner.exe -extract
    Write down the Persistent Id it gives you.

3) Install a fresh copy of iTunes on another machine
    Run it once, and quit iTunes.

3) Run itunes-persistent-id-cloner.exe -patch on another machine
    When prompted, enter the persistent Id supplied by the other Id.
    Ensure iTunes isn't running.
    Hit enter.

No friendly error messages or error handling at the moment.
Don't make typos, things might crash. Worse case, you can run / patch a second time.

JustGiving Direct Debit SDI–Y U NO WORK?!

December 11th, 2011

JustGiving SDIs, ExitUrl redirection and monthly direct debit donations. Y U NO WORK?!

When we first implemented SDI, it worked for everything everywhere. At that time, we accepted single donations via credit card and PayPal, and monthly donations via a recurring credit card mandate where we manually re-requested authorisation on a schedule until the end user stopped it from their control panel. Nice and easy.

SDI operates on a single principle: If you specify an ExitUrl, we guarantee that if the user completes the donation flow, regardless of the success of the donation, you’ll end up with that user hitting the exit Url. This allows people to integrate, validating the donation result using the API, and keeping the user on their website for anything they want to message after. It also hooks in to a few iPhone and Android apps, where that ExitUrl re-launches the application through some nifty JavaScript hacks.

Several months later, we implemented what we internally call “IPDD”, internet paperless direct debit. Which is… kind of cool? Depending on what you think about the awesomeness of lack thereof, of Direct Debits. Suffice to say, charities were begging for it, because recurring donations are really good for them (largely because people forget to ever cancel them, and they’re a solid source of predictable income for financial planning). We switched out our credit card mandate for direct debit processing.

Part of the legal requirements surrounding Direct Debits is that you have to show some *very* specific information in a *very* specific format when the user creates one. Basically, the user needs to know their rights, they need to know how to cancel, and these things are all the more important when you’re just pulling money right out of their account, as a result, we were literally forced to show the success page that you’ll have seen if you’ve set up an IPDD on JustGiving. It’s not pretty, but in short, there’s a reason that all direct debit forms look the same.

Jumping back to SDI, the way we ensure that a user is correctly redirected, is that we don’t give them a choice, this is also the reason that the users don’t see the “leave a message” page during SDI, and that we implemented a default message in the SDI url schema. We hook on to our processing page and immediately send them back to the integrator after we have some kind of response (or send them regardless after 13 seconds). Unfortunately, our direct debit confirmation page comes *after* the processing page (because by definition, it comes after the “donate” click) in the process.

So, long story short: we’re required to show a direct debit confirmation, and SDI needs to hook in before that point and as a result, SDI broke for direct donations when we integrated IPDD and we had to make the choice to leave it broken. Furthermore, the nature of the IPDD produce means that (I believe) we are legally required to supply the user with certain information.

So, rock and a hard place right?

Honestly, at the moment this hasn’t been the most high priority thing to look at, as it’s not a huge use case (this is also why I’m writing this on a Sunday afternoon) but I feel like we need to do *something* about it, but my options are fairly limited.

Here are a few ideas that have been suggested internally and externally with some pros and cons.

· Do a server side call to the return Url to confirm a donation

o Pro: You’ll still be aware of a donation and can sync your data
o Con: Not the anticipated user flow, this kind of various will confuse integrators
o Con: Useless for UI based logic at integrators end

· Move the direct debit “confirmation” information to before the “confirm” button click

o Pro: Almost a perfect solution, as we could then re-enable SDI
o Con: Technically wouldn’t be a “confirmation” page, so may have to get some special sign off from external parties

· Don’t show the confirmation at all, just send the user an email

o Pro: Again, ideal from an integration perspective, may lead to some users to be confused and not realise it’s actually been “done”, so may lead to support issues
o Con: Not even sure if we’re allowed to do this. Would definitely need external sign off

Does any have any other ideas about how they’d prefer this to work? Regardless, I hope this answers some questions related to this problem.

For my part, I’d love to fix it, but it’s not super high priority right now. HOWEVER, if there’s something we can do practically with a small amount of effort, and the demand is there, I’m sure we’ll get it looked at much sooner.

(Cross posted to Google groups and my blog for seachability)

Configuring Windows Server 2008 R2 as a non-domain file server

November 28th, 2011

Oh dear lord, I’ve just lost two days of my life trying to find the exact incantation to persuade a fresh install of Windows Server 2008 R2 to act as a file server on my home network. I did this last two and a half years ago, and it’s taken me two days to regain the knowledge.

Here’s the list of things you need to do to make it play nicely, both with anonymous / legacy accounts and operating systems, and to allow access control for protected shares.

  1. Add the file server role
  2. Modify the Local Security Policy –> Local Policies –> Security Options
    1. Network access: Allow anonymous SID/name translation = Enabled
    2. Network access: Do not allow anonymous enumeration of SAM accounts = Disabled
    3. Network access: Do not allow anonymous enumeration of SAM accounts and shares = Disabled
    4. Network access: Let Everyone permissions apply to anonymous users = Enabled
    5. Network access: Restrict anonymous access to Named Pipes and Shares = Disabled
    6. Network access: Sharing and security model for local accounts = Classical
    7. Network security: Do not store LAN Manager hash value on next password change = Enabled
    8. Network security: LAN Manager authentication level = Send LM & NTLM – use NTLMv2 session security if negotiated
  3. Flush your group policy like a boss (gpupdate /force)
  4. Control Panel\Network and Internet\Network and Sharing Center\Advanced sharing settings
    1. Turn on network discovery
    2. Turn on file and printer sharing
    3. Turn on public folder sharing
    4. Turn off password protected sharing (else nobody will be able to browse to machine names)
  5. Turn on the Guest user account
  6. Create any user accounts you want to give specific access to.

Now: Anonymous users will authenticate as the guest account, so “Everyone” network sharing will work fine, and you can specifically log on to the server as SERVERNAME\user for specific access rights on mapped network shares.

It’s just that easy.

Yeah.

CouchDb using C# in 15 minutes

March 14th, 2011

I’ve spent quite a lot of time experimenting with document databases at the moment, almost exclusively RavenDb of late.  We’re currently looking at sweeping architecture changes at work, and unfortunately it doesn’t look like Raven is going to be suitable (nothing to do with the product, which is excellent, just a target audience with a specific ambivalence to Windows server exclusive products).  Knowing that a lot of Ravens design decisions were based around using CouchDb, and knowing that CouchDb might be a better fit for the scenario I have in mind, I set out to see how easy Couch is to use on Windows.

Short answer? Very very easy, so here’s a few quick steps for someone that wants to get up and running quickly.

First, go grab a copy of your platform-specific installer from CouchBase and install it. This is a 2-3 click process, and as the installer exits your browser pops up into “Futon” the Couch web UI.

Then go to https://github.com/foretagsplatsen/Divan , clone / download the Divan source code and build it. Divan is a handy open source project aimed at wrapping the Couch API (which is RESTful, so if you want to roll your own client library feel free). Divan is quite bare to the metal and relies on Json.Net for de/serializing documents.

Launch Visual Studio and new up a project of your choice. I’ve chucked together a few utility classes while experimenting to deal with persistence in order to get a syntax that looks like this:

   1: const string host = "localhost";

   2: const int port = 5984;

   3: var server = new CouchServer(host, port);

   4: var db = server.GetDatabase("myfirstdatabase");

   5:  

   6: var document = new SomeRandomAggregateRoot

   7:                    {This = "Is cool!", Subclass = new SomeRandomClass {IsAwesome = "hell yeah!"}};

   8:  

   9: using (var couchRepo = new CouchRepository<SomeRandomAggregateRoot>(db))

  10: {

  11:     var id = couchRepo.Save(document);

  12:     var returnedCouchDoc = couchRepo.Retrieve(id);

  13:     ViewData["Message"] = returnedCouchDoc._id;

  14: }

  15:  

  16: return View();

Which I’m pretty happy with as it seems very usable and maps pretty simply from the underlying Json documents.

I have a base class and an interface at the bottom of the object graph that look like this

   1:  

   2:     public class CouchAggregateRoot : ICouchAggregateRoot

   3:     {

   4:         public string _id { get; set; }

   5:         public string _rev { get; set; }

   6:     }

   7:  

   8:     public interface ICouchAggregateRoot

   9:     {

  10:         string _id { get; set; }

  11:         string _rev { get; set; }

  12:     }

and two little repository classes that look like this:

   1: public class CouchRepository<T> : CouchRepository where T : CouchAggregateRoot

   2: {

   3:     public CouchRepository(ICouchDatabase couchDatabase):base(couchDatabase)

   4:     {

   5:     }

   6:  

   7:     public T Retrieve(string id)

   8:     {

   9:         return Retrieve<T>(id);

  10:     }

  11:  

  12:     public string Save(T @object)

  13:     {

  14:         return Save<T>(@object);

  15:     }

  16: }

  17:  

  18: public class CouchRepository: IDisposable

  19: {

  20:     protected readonly ICouchDatabase CouchDatabase;

  21:  

  22:     public CouchRepository(ICouchDatabase couchDatabase)

  23:     {

  24:         CouchDatabase = couchDatabase;

  25:     }

  26:  

  27:     public string Save<T>(T @object) where T : CouchAggregateRoot

  28:     {

  29:         var couchDocument = new CouchDocumentWrapper<T>(@object);

  30:         CouchDatabase.SaveDocument(couchDocument);

  31:         return couchDocument.Id;

  32:     }

  33:  

  34:     public T Retrieve<T>(string id) where T : CouchAggregateRoot

  35:     {

  36:         var rawDocument = CouchDatabase.GetDocument(id);

  37:         var @object = JsonConvert.DeserializeObject<T>(rawDocument.ToString());

  38:         @object._id = rawDocument.Id;

  39:         @object._rev = rawDocument.Rev;

  40:         return @object;

  41:     }

  42:  

  43:     public void Dispose()

  44:     {

  45:         // Should probably get round to implementing this...

  46:     }

  47: }

They’re obviously not entirely fleshed out, and the Divan project obviously encapsulates more of Couches functionality than those two methods alone.

When you execute the code snippet above (which for the sake of this example was just in a random ASP.NET MVC Controller’s Get method) you’ll see a new database called “myfirstdatabase” created in Futon with your newly created document inside it.

This is obviously just scratching at the surface, but I’ve been exceptionally pleased at how easy it is to get a File –> New Project up and running using readily available tools.

Code in full on GitHub for posterity

The perils of outsourcing software development

November 29th, 2010

I’m frequently asked for my opinion on outsourcing software projects and components and in the vast majority of cases I always end up explaining my distaste for outsourcing when you work in the software industry. It just doesn’t work very well at all.

People often decide that outsourcing could be a great idea for a specific project because they see an opportunity to bring something to market very quickly and don’t want to disrupt their existing development resource or interrupt on-going project work. Unfortunately, this decision is often based around the fundamental mistruth that “outsourcing, on a sufficiently small scale, is both cheap and quick and as such harmless.”

The train of thought that leads people to outsourcing is often derailed by a quick experiment in discovering what your company is really about and what your company really does. I have one rule:

“Never outsource your core competences or anything that leads to your competitive advantage”

It’s that simple. If you outsource something that gives your business a real competitive advantage, there is absolutely nothing to stop any or all of your competitors doing exactly the same thing. In this scenario, the only people that win are the agencies that produce the outsourced software.

I’ve mentioned this to people in the past who have been surprised by what they were about to do, and just hadn’t identified the product they were about to outsource as a competitive advantage. What this really highlighted to me was how people often struggle to identify what their company is really about and what their core competencies and products really are. Effectively what the “essence” of the company is and what the product really is.

A little experiment that I encourage people to do, is to take what you perceive your “core” product to be and then mentally remove everything else from your business in a “virtual outsourcing” thought experiment. Take what’s left, the nucleus of your product and look at it’s characteristics. Those characteristics are what your company is really about, and like it or not, if you attempt to outsource any of those core characteristics, you are deciding to side-line your own creative advantage.

This experiment is especially good at highlighting when a company’s’ core asset is technology or a technology product, while they perceive themselves to be more of a media company or non-tech company. If your core product is technology, you are a technology company regardless of whether that’s what you set out to be. This often applies to online business’ who fail to grasp that if their product is a website, you can never take the technology away, and at it’s very core, that is the value proposition of their business.

Understanding your core competency is not the only place that outsourcing and software development fall apart, it just illustrates how businesses reach these conclusions in the first place. The original software outsourcing model was a natural evolution of the idea that software development process could be modelled after manufacturing in some sort of glorified “software assets” process where you built a bunch of components that just plugged together to make a system. This idea rode on the back of the general good practice of reuse in software development and the boom in “4GL” programmer-less languages in the 90s. Well, that fad never really worked out, but it took people awhile to realise that the core assumption that software is like manufacturing was fundamentally flawed.

“Software isn’t a manufacturing process, it’s a design process, and as such, should not be outsourced as if it were mass production or duplication”

Outsourcing the manufacture of plastic cups works really well. We all know what a cup looks like, and even if you make a particularly fancy cup, you’ll get your designers to come up with the patterns, you’ll ship your pattern to a manufacturing plant who specialise in duplicating plastics and you’ll ask them to make a few thousand. Which part of producing a few thousand of your plastic fancy cups was the important part? The design that you handed over that made your cup different to the others was the important part, not the duplication. Software works in exactly the same way.

When software was first offshored, there was this fundamental misunderstanding that the development of the software was akin to pressing CDs in a plant, or manufacturing a thousand cups, when actually it was more like when Apple make products “Designed in California”. They’re produced en mass in China, but that’s not the important part, the design is. Treating the design of your software as if it were replication will only ensure low quality of the resulting product.

If you’re attempting to outsource components of a bigger system, you’re also going to face a whole second set of problems all related to trying to make your outsourced components consistent and maintainable in the context of a greater software product. You’re likely to receive software that’s outside of any “house-style” in your code (with different conventions), the quality will be unmonitored and variable and you’ll find it very difficult to prescribe patterns, practices and QA processes. Once you’ve dealt with that collection of known unknowns, you’ll then have to integrate the outsource components into the system with nothing more than a contract in place to ensure that they work. Don’t let that contract fool you though, it’s not security, it just means you’re allowed to get mad when the software doesn’t work. You’re not going to get your wasted time back. Despite the best quotes and claims of third parties, never underestimate the effort of integration. I worked on a particular project which touted a “4 hour” integration process. All things considered, with 3 days of meetings and the trailing two weeks of implementation, it took roughly 300x the quoted length of time.

People are often driven to outsourcing as a way to circumvent resource restriction but all that happens when this is attempted is you fall foul of the traps highlighted by Fred Brooks in The Mythical Man-Month – namely, the communication overhead of coordinating outsourcing and your internal development and the effort that it requires vastly outweighs any perceived benefits of the outsourcing exercise to start with. This is all compounded by the fact that humans are notoriously bad at writing software specifications, and your best attempt at outsourcing is only as good as your best attempt at communicating your ideas in way that the other party thoroughly understands. But as soon as you specify succinctly, you loose any kind of adaptability and end up, in the best case, with the product you thought you needed at the start of the process, rather than the product you know you need by the end because of changing business requirements.

Rework in conjunction with outsourcing is difficult and costly and if you end up down that route, you’re going to want to avoid it at all costs, though, ironically, if the outsourcing venture is successful, you’re likely to want to bring the project in house anyway to curb high maintenance costs and loosing your competitive advantage. Maintaining outsourced software is expensive for a very obvious reason: agency culture dictates that they must charge an appropriate amount to offset their risk of having no work. Bringing outsourced or offshored projects back in house is going to cause you yet more problems, especially if your technology stack doesn’t exactly match that of the outsourcing partner. That great PHP system they wrote? Doesn’t work so well with your ASP.NET software or your Ruby on Rails app. You’re then faced with the horrible position of having to re-write to bring the software back in house, or maintain a whole separate set of systems to support this product, raising it’s total cost of ownership significantly. This might not be a problem if you make hybrid technology choices deliberately, but at this point they’re effectively forced upon you.

So is outsourcing ever a good idea? Well, probably not, all things considered. You’re more likely to get more consistent results by spending the money you’d allocated for that killer new outsourced project training your staff. In the long run it’s better value. It’s also sometimes more prudent to skip what you perceive to be that “killer opportunity” when you don’t have time or resource and revisit it later with a quality product that maintains your own creative vision and competitive advantage. There are a few well established large agencies with a known track record for delivery and professionalism. You’ll find these agencies are more likely to work alongside you and your staff, rather than dropping a boxed product on your lap, and this co-working gives them a distinct advantage over any other form of outsourcing as they are more likely to work well inside of your internal processes.

If you’re considering outsourcing a software project, you should always ensure that you factor the effort required to bring that project back in house as part of the estimate of cost. If you’re considering outsourcing part or all of your core product, you’ve probably already lost.

Pluralization (Pluralisation!) in C#

November 15th, 2010

So today at work we were spitting some league table-ish (shh! unreleased feature!) data onto a screen and I realised I was going to have to manually pluralise “0 donations”, “1 donation”, “2 donations” depending on the number of donations having been made. I felt a little bit of the crazy eye coming on, because I’ve solved this problem a hundred times before.

By “solved” I mean, “I’ve written a shitty if(..) statement to solve this problem a hundred times before”, which totally sucks.  And I exclaimed “but surely this is a problem everybody that has ever displayed text has faced, it can’t possibly be that we all solve this every day”.

Turns out, that this has pretty much been the case until recently. Rails has Pluralize methods, and most ORMs clearly have this functionality to some extent but I’d never happened across a general purpose .NET library to do the job.  After sitting and expressing my utter disbelieve that this isn’t (at least in it’s simplest form) a solved problem, I actually used some GoogleFu to discover that it was.

So, in .NET4, it looks like Microsoft actually got round to implementing this in the framework, I’d not happened across it before, but it’s pretty freaking cool.

So let me introduce, if I may, the System.Data.Entity.Design.PluralizationServices namespace!

The more observant of you might notice that it appears to be in a namespace with “Entity” in the title. I suspect it was cooked up as part of EF (boo! hiss!) but luckily, you don’t need to actually reference the main System.Data.Entity assembly as it’s been separated out into a satellite assembly.  It also looks like localization is hiding behind one giant NotImplementedException(); currently, but the usage is pretty simple and calmed me down in no time.

var _pluralizationService = PluralizationService.CreateService(Thread.CurrentThread.CurrentCulture);
var plural = _pluralizationService.Pluralize(singular);

and that’s it!

Almost too good to be true, but it actually works, managing “donation”=>”donations”, “sheep”=>”sheep” and as esoterically pointed out on twitter, even “cardex”=>”cardecies”. Which is really pretty cool.

We took it a little further and added support to it in our suite of extension methods for the ASP.NET MVC HtmlHelper class, so in our MVC views we can do this:

<%@ Import Namespace="Extensions" %>
<%=Html.Language().CorrectTenseForQuantity(Model.Quantity, "word") %>

Which I thought was pretty tasty (and at the very least, gives us the same thing the Rails guys have, just with slightly more explicit syntax (my personal preference).

I’ve put our extension implementation up as a Gist on GitHub here if anyone wants to see the full implementation.

Pretty cool.

JustGiving <3 CharityHack 2010

September 9th, 2010

So this year @JustGiving we’re going to be making a little bit of a big deal at Charity Hack 2010 by releasing the first round of public JustGiving APIs.

The new JustGiving APIs are a suite of RESTful resources covering a few key areas of the site, debuting at Charity Hack 2010 after closed testing as the tech powering the JustGiving iPhone, Android and Facebook applications.

Headlines: JustGiving. RESTful APIs debuting at Charity Hack 2010. Fully documented and with code examples in major languages.

We’re covering four key areas: End user accounts, Fundraising Pages, Search and Donation Integration.  Don’t worry, this post isn’t the documentation, instead, it’s a bit of a teaser.

Account

  • Account Registration

Fundraising

  • Get Fundraising Page Details (key page details)
  • Get Fundraising Page Donations (with pagination)
  • Get Fundraising Pages (retrieving all pages owned by the authenticated user)
  • Fundraising Page creation (automate page creation)
  • Update Fundraising Page (to modify details and user stories)
  • Upload Fundraising Page Image (upload new FRP photos to the gallery)
  • Fundraising Page Url Availability Check (check the availability of JG Page Urls, useful paired with creation)

Search

  • Charity Search

Donation

  • Get Donation Status (for a given donation Id, check the processed state of the donation)

Simple Donation Integration (SDI)

SDI is similar to PayPal’s Express Checkout allowing partners to funnel their users through the JustGiving donation process as part of the flow of other / existing applications.  It’s designed to be partnered with the Donation Status API methods.  The idea is, a user is redirected / pushed through to a web browser directly into the JustGiving donation process.  The user makes a donation and is then redirected back to a call-back Url on the originators domain.  This call-back will optionally pass the donation Id of the users donation on the query string, allowing the destination site to query the status of the users donation.

This may be of extra interest to mobile application developers, as combined with registering protocols for your mobile app in iOS or Android, this will effectively enable you to take donations in your mobile applications.  SDI powers the recently released Facebook JustGiving Charity Gifts application.

I’m going to be there on the day with a few of my colleagues to babysit the APIs and answer questions, but more importantly, I’m going to be there as a hacker. Obviously we can’t really compete in any of the "hack offs", but we might well end up cooking up something cool or fun on the show floor.

The APIs we’re going out with are RESTful and use XML as the content-type (boo!), but if you dig deep enough, we might well just be respecting other content types (in an unsupported, on the quiet, kind of way) if your client asks nicely (yay!).

Happy hacking!

Serving different views for mobile devices in ASP.NET MVC

May 3rd, 2010

As browsing becomes more commonplace on phones, sub-notebooks and (within the next year or so) Tablet PCs, there’s an increased appetite to tailor your user experience for people using these “non-desktop” devices.  You can leverage your existing application infrastructure without having to create costly or outsource applications for specific (*cough* iPhone) platforms that have lots of market share, to the exclusion of others (*cough* iPhone).

There are some excellent examples of this in the wild, Facebook and the BBC do a great job of this already.

So what about us ASP.NET MVC types?  Well thankfully, there’s a lot of stuff built in to the framework to allow us to do this with relative ease.  First you’re going to need a few things..

1) Go grab the latest Mobile Device Browser File from http://mdbf.codeplex.com.  If you’re familiar with the old “Browser Caps”, it’s the same sort of thing.  A regularly updated collection of device data.  It’s pretty interesting on it’s own, breaking down incoming devices by user agent / headers and offering you various stats (touch enabled, screen resolution etc).  You’ll probably want to keep this up to date periodically.

2) A new ViewEngine!  Thankfully, ASP.NET MVC has a pretty flexible pipeline when it comes to slotting in new ViewEngines.  The most transparent way to switch out which view you’re serving for any given request is to override some of the logic in the default ViewEngine to look elsewhere when a View is requested by the MVC framework.

3) A way to test this stuff.  I normally break out the “User Agent Switcher” FireFox plugin which you can grab here: https://addons.mozilla.org/en-US/firefox/addon/59

 

The View Engine

In order to keep this as close to vanilla MVC as possible, I’m going to extend the regular WebFormsViewEngine to add mobile device detection and view overriding.  What I’m going to do is check for a mobile browser when a request reaches the view engine, and if one is found, add some extra paths to look for the view files in at the top of the list of locations searched.  By doing this, we can prioritise the mobile edition of a website if the user is visiting from a phone, while degrading gracefully, allowing the regular version of the website to be served if a mobile version of a given page isn’t available.  This actually allows us to support mobile views piece by piece rather than forcing us to support the entire site out of the box.

Unfortunately, a few of the methods surrounding view resolution are marked as private in the WebFormsViewEngine and as such are inaccessible.  I’ve had to reflect in and copy a couple of methods to get around this.  Ideally the access modifier on these could be changed in later versions of the framework.

The key method we have to work with is

public override ViewEngineResult FindView(ControllerContext controllerContext, string viewName, string masterName, bool useCache)
{ … }

What we’re going to do is add a few extra locations based on the browser type.  With the mobile device browser file installed, this is really simple:

public override ViewEngineResult FindView(ControllerContext controllerContext, string viewName, string masterName, bool useCache)
{
    if (controllerContext == null)
    {
        throw new ArgumentNullException("controllerContext");
    }
    if (String.IsNullOrEmpty(viewName))
    {
        throw new ArgumentException("viewName");
    }

    List<string> viewLocationsSearched;
    List<string> masterLocationsSearched;

    string[] viewLocationsToSearch = ViewLocationFormats;
    string[] masterLocationsToSearch = MasterLocationFormats;

    viewLocationsToSearch = AddMobileViewLocations(controllerContext, viewLocationsToSearch, MobileViewLocationFormats);
    masterLocationsToSearch = AddMobileViewLocations(controllerContext, masterLocationsToSearch, MobileMasterLocationFormats);

    string controllerName = controllerContext.RouteData.GetRequiredString("controller");
    string viewPath = GetPath(controllerContext, viewLocationsToSearch, viewName, controllerName, CacheKeyPrefixView, useCache, out viewLocationsSearched);
    string masterPath = GetPath(controllerContext, masterLocationsToSearch, masterName, controllerName, CacheKeyPrefixMaster, useCache, out masterLocationsSearched);

    if (String.IsNullOrEmpty(viewPath)
        || (String.IsNullOrEmpty(masterPath)
        && !String.IsNullOrEmpty(masterName)))
    {
        return new ViewEngineResult(viewLocationsSearched.Union(masterLocationsSearched));
    }

    return new ViewEngineResult(CreateView(controllerContext, viewPath, masterPath), this);
}

You’ll notice there’s a few methods that get called in there.  The most important of which is “AddMobileViewLocations”.  This really is where all the legwork is done, and looks like this

public class SwitchingViewEngine : WebFormViewEngine
{
    private const string CacheKeyFormat = ":ViewCacheEntry:{0}:{1}:{2}:{3}:";
    private const string CacheKeyPrefixMaster = "Master";
    private const string CacheKeyPrefixView = "View";
    private static readonly List<string> EmptyLocations = new List<string>();

    protected string[] MobileViewLocationFormats { get; private set; }
    protected string[] MobileMasterLocationFormats { get; private set; }

    public SwitchingViewEngine()
    {
        ViewLocationFormats = new[]
                                  {
                                      "~/Views/{1}/{0}.aspx", "~/Views/{1}/{0}.ascx", "~/Views/Shared/{0}.aspx",
                                      "~/Views/Shared/{0}.ascx"
                                  };

        MobileViewLocationFormats = new[]
               &
#160;                        {
                                            "~/Views/{1}/{0}.mobile.aspx", "~/Views/{1}/{0}.mobile.ascx",
                                            "~/Views/Shared/{0}.mobile.aspx",
                                            "~/Views/Shared/{0}.mobile.ascx"
                                        };

        MasterLocationFormats = new[] {"~/Views/{1}/{0}.master", "~/Views/Shared/{0}.master"};
        MobileMasterLocationFormats = new[] {"~/Views/{1}/{0}.mobile.master", "~/Views/Shared/{0}.mobile.master"};
    }

    public override ViewEngineResult FindView(ControllerContext controllerContext, string viewName, string masterName, bool useCache)
    {   …    }

    private static string[] AddMobileViewLocations(ControllerContext controllerContext,
                                                   string[] viewLocationsToSearch,
                                                   IEnumerable<string> mobileViewLocations)
    {
        if (controllerContext == null
            || controllerContext.HttpContext == null
            || controllerContext.HttpContext.Request == null
            || controllerContext.HttpContext.Request.Browser == null
            || viewLocationsToSearch == null
            || viewLocationsToSearch.Length == 0
            || mobileViewLocations == null
            || mobileViewLocations.ToList().Count == 0
            || !controllerContext.HttpContext.Request.Browser.IsMobileDevice)
        {
            return viewLocationsToSearch;
        }

        var mobileViews = viewLocationsToSearch.ToList();
        foreach (var view in mobileViewLocations.Reverse())
        {
            mobileViews.Insert(0, view);
        }

        viewLocationsToSearch = mobileViews.ToArray();

        return viewLocationsToSearch;
    }

This method takes the current ViewLocations that are defined at the top of the class, does a bunch of guard checks (to prevent mobile switching crashing your request.. call me paranoid), then verifies that controllerContext.HttpContext.Request.Browser.IsMobileDevice is true.  This check makes use of the browser device file.  If all the checks pass, it inserts the new “mobile view paths” at the very top of the list of paths to be searched when resolving the location of a view file.  The calling method then subsequently calls the method “GetPath” (which is one of the private methods I’ve had to reflect out of the framework source code).  GetPath searches the supplied list of potential view locations, and returns as soon as it finds a match.  In our case, if the browser is mobile, and a view file with the extension mobile.aspx is found, this will be the first view resolved and returned.

The full code listing for the view engine (don’t worry, there’s a link at the bottom to grab all of this in one archive):

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Web.Mvc;

namespace MultipleViewMvcExample.DemoCode
{
    public class SwitchingViewEngine : WebFormViewEngine
    {
        private const string CacheKeyFormat = ":ViewCacheEntry:{0}:{1}:{2}:{3}:";
        private const string CacheKeyPrefixMaster = "Master";
        private const string CacheKeyPrefixView = "View";
        private static readonly List<string> EmptyLocations = new List<string>();

        protected string[] MobileViewLocationFormats { get; private set; }
        protected string[] MobileMasterLocationFormats { get; private set; }

        public SwitchingViewEngine()
        {
            ViewLocationFormats = new[]
                                      {
                                          "~/Views/{1}/{0}.aspx", "~/Views/{1}/{0}.ascx", "~/Views/Shared/{0}.aspx",
                                          "~/Views/Shared/{0}.ascx"
              
60;                       };

            MobileViewLocationFormats = new[]
                                            {
                                                "~/Views/{1}/{0}.mobile.aspx", "~/Views/{1}/{0}.mobile.ascx",
                                                "~/Views/Shared/{0}.mobile.aspx",
                                                "~/Views/Shared/{0}.mobile.ascx"
                                            };

            MasterLocationFormats = new[] {"~/Views/{1}/{0}.master", "~/Views/Shared/{0}.master"};
            MobileMasterLocationFormats = new[] {"~/Views/{1}/{0}.mobile.master", "~/Views/Shared/{0}.mobile.master"};
        }

        public override ViewEngineResult FindView(ControllerContext controllerContext, string viewName, string masterName, bool useCache)
        {
            if (controllerContext == null)
            {
                throw new ArgumentNullException("controllerContext");
            }
            if (String.IsNullOrEmpty(viewName))
            {
                throw new ArgumentException("viewName");
            }

            List<string> viewLocationsSearched;
            List<string> masterLocationsSearched;

            string[] viewLocationsToSearch = ViewLocationFormats;
            string[] masterLocationsToSearch = MasterLocationFormats;

            viewLocationsToSearch = AddMobileViewLocations(controllerContext, viewLocationsToSearch, MobileViewLocationFormats);
            masterLocationsToSearch = AddMobileViewLocations(controllerContext, masterLocationsToSearch, MobileMasterLocationFormats);

            string controllerName = controllerContext.RouteData.GetRequiredString("controller");
            string viewPath = GetPath(controllerContext, viewLocationsToSearch, viewName, controllerName, CacheKeyPrefixView, useCache, out viewLocationsSearched);
            string masterPath = GetPath(controllerContext, masterLocationsToSearch, masterName, controllerName, CacheKeyPrefixMaster, useCache, out masterLocationsSearched);

            if (String.IsNullOrEmpty(viewPath)
                || (String.IsNullOrEmpty(masterPath)
                && !String.IsNullOrEmpty(masterName)))
            {
                return new ViewEngineResult(viewLocationsSearched.Union(masterLocationsSearched));
            }

            return new ViewEngineResult(CreateView(controllerContext, viewPath, masterPath), this);
        }

        private static string[] AddMobileViewLocations(ControllerContext controllerContext,
                                                       string[] viewLocationsToSearch,
                                                       IEnumerable<string> mobileViewLocations)
        {
            if (controllerContext == null
                || controllerContext.HttpContext == null
                || controllerContext.HttpContext.Request == null
                || controllerContext.HttpContext.Request.Browser == null
                || viewLocationsToSearch == null
                || viewLocationsToSearch.Length == 0
                || mobileViewLocations == null
                || mobileViewLocations.ToList().Count == 0
                || !controllerContext.HttpContext.Request.Browser.IsMobileDevice)
        &#
160;   {
                return viewLocationsToSearch;
            }

            var mobileViews = viewLocationsToSearch.ToList();
            foreach (var view in mobileViewLocations.Reverse())
            {
                mobileViews.Insert(0, view);
            }

            viewLocationsToSearch = mobileViews.ToArray();

            return viewLocationsToSearch;
        }

        private string GetPath(ControllerContext controllerContext, string[] locations, string name,
                               string controllerName, string cacheKeyPrefix, bool useCache,
                               out List<string> searchedLocations)
        {
            searchedLocations = EmptyLocations;
            if (string.IsNullOrEmpty(name))
            {
                return string.Empty;
            }
            if ((locations == null) || (locations.Length == 0))
            {
                throw new InvalidOperationException("Property cannot be null or empty.");
            }
            bool flag = IsSpecificPath(name);
            string key = CreateCacheKey(cacheKeyPrefix, name, flag ? string.Empty : controllerName);
            if (useCache)
            {
                string viewLocation = ViewLocationCache.GetViewLocation(controllerContext.HttpContext, key);
                if (viewLocation != null)
                {
                    return viewLocation;
                }
            }
            if (!flag)
            {
                return GetPathFromGeneralName(controllerContext, locations, name, controllerName, key,ref searchedLocations);
            }
            return GetPathFromSpecificName(controllerContext, name, key, ref searchedLocations);
        }
        private static bool IsSpecificPath(string name)
        {
            char ch = name[0];
            if (ch != ‘~’)
            {
                return (ch == ‘/’);
            }
            return true;
        }

        private string GetPathFromSpecificName(ControllerContext controllerContext, string name, string cacheKey,
                                               ref List<string> searchedLocations)
        {
            string virtualPath = name;
            if (!FileExists(controllerContext, name))
            {
                virtualPath = string.Empty;
                searchedLocations = new List<string> {name};
            }
            ViewLocationCache.InsertViewLocation(controllerContext.HttpContext, cacheKey, virtualPath);
            return virtualPath;
        }

        private string GetPathFromGeneralName(ControllerContext controllerContext, string[] locations, string name,
                                              string controllerName, string cacheKey, ref List<string> searchedLocations)
        {
            string virtualPath = string.Empty;
            searchedLocations = new List<string>();
            for (int i = 0; i < locations.Length; i++)
            {
                string str2 = string.Format(CultureInfo.InvariantCulture, locations[i],
       
                                     new object[] {name, controllerName});
                if (FileExists(controllerContext, str2))
                {
                    searchedLocations = EmptyLocations;
                    virtualPath = str2;
                    ViewLocationCache.InsertViewLocation(controllerContext.HttpContext, cacheKey, virtualPath);
                    return virtualPath;
                }
                searchedLocations[i] = str2;
            }
            return virtualPath;
        }

        private string CreateCacheKey(string prefix, string name, string controllerName)
        {
            return String.Format(CultureInfo.InvariantCulture, CacheKeyFormat,
                                 GetType().AssemblyQualifiedName, prefix, name, controllerName);
        }
    }
}

 

The Wiring

Now you have your view engine, you need to register it as the default view engine in your MVC application.  Easy!  Open up your Global.aspx.cs file and add the following to ApplicationStart

ViewEngines.Engines.Clear();
ViewEngines.Engines.Add(new SwitchingViewEngine());

Done!

Now you need to actually add the browser detection file to your application.  Presuming you downloaded the latest archive from the codeplex url at the top of this article, all you need to do is copy the supplied mobile.browser file to App_Browsers/Device/* in your MVC application.

That’s all the wiring you need to get everything up and running.

If you’ve done it right, a default “New MVC Template” project with these additions might look something like this:

image

For the sake of this demo, I put the view engine in a “DemoCode” sub-namespace. You don’t want to do that, put it somewhere sensible!

The more astute reader might now notice that my Views/Home directory in the above screenshot has an extra file, not supplied by the template, called “Index.mobile.aspx”.  Likewise, my /Views/Shared directory has Site.mobile.Master.  These are files I want the view engine to resolve if a user hits the http://localhost/Home default route from a mobile device.

Index.mobile.aspx looks like this:

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.mobile.Master" Inherits="System.Web.Mvc.ViewPage" %>

<asp:Content ID="indexTitle" ContentPlaceHolderID="TitleContent" runat="server">
    Mobile Home Page
</asp:Content>

<asp:Content ID="indexContent" ContentPlaceHolderID="MainContent" runat="server">
    <h2><%= Html.Encode(ViewData["Message"]) %></h2>
    This is my mobile index page.
</asp:Content>

and Site.mobile.Master looks like this:

<%@ Master Language="C#" Inherits="System.Web.Mvc.ViewMasterPage" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title><asp:ContentPlaceHolder ID="TitleContent" runat="server" /></title>
</head>

<body>
    <div class="page">

        <div id="header">
            <div id="title">
                <h1>My MVC Application – mobile master page</h1>
            </div>
            <div id="logindisplay">
                <% Html.RenderPartial("LogOnUserControl"); %>
            </div>
            <div id="menucontainer">
                <ul id="menu">             
                    <li><%= Html.ActionLink("Home", "Index", "Home")%></li>
                    <li><%= Html.ActionLink("About", "About", "Home")%></li>
                </ul>
            </div>
        </div>

        <div id="main">
            <asp:ContentPlaceHolder ID="MainContent" runat="server" />

            <div id="footer">
            </div>
        </div>
   
</div>
</body>
</html>

Nothing especially revolutionary.

 

Testing

First, install that FireFox plugin I mentioned at the top of the article, create a new MVC project (just use the template), and wire up the view engine and browser file.  Add some extra views with the .mobile.aspx prefix.  Or download the sample attached to the bottom of this post!

Start the site up in Cassini (Visual Studios default web server) and hit /Home.  You should see this:

image

Now, lets use the new FireFox plugin…

image

Select iPhone 3.0 from that menu and refresh the page…

image

Bang!  The more eagle eye reader might have notice that all I did in my “mobile” master page, was delete the CSS reference from the default MVC template, thus the above screenshot.

And that’s it.

 

Further Thoughts

This solution goes quite a way, but here are a few other ideas:

  • Don’t just do browser type detection, detect and switch on subdomain, so any visitors hitting http://m.mysite.com get a different view.
  • Allow the user to opt-out of the reduced view with a session cookie.
  • Switch views to a low-fi version to victimise IE6 users!
  • Target tablet PCs based on resolution to build a “touch UI”

It’s all pretty simple.  The really nice thing about this solution is that your designers can just add these mobile views as and when they see fit, as the same action methods that execute for the “full fat” website are run, and the same strongly typed view models (which you’re using, right? get out of here with your ViewData..) are delivered.  Designers can implement portable websites, piece by piece, just with a little view engine change.

As with all internet code, your mileage may vary, but this technique works for me.

You can download a working VS2008 solution (so long as you have ASP.NET MVC1 installed on your system) containing all the code used above from: http://github.com/davidwhitney/MultipleViewMvcExample