Fear Of Deployment

When you start working on a new project there are four really important pieces of information you should be aware of regarding the deployment of your software

1. Are you unafraid of deploying your code to production?
2. Do you know how your code will react in your production environment with production configuration?
3. Does your code react in the same way in your production environment as it does in development?
4. Are you confident in the repeatability of your installation procedure?

Fear and uncertainty in the deployment of software projects is a common theme I’ve seen across all the companies where I’ve worked and it’s never good for your project.  The fear of deployment is always a good indicator of the confidence your staff have in the project they are working on.

A project with unknown qualities and suspicious (or known but unreported) bugs often causes an serious fear of deployment.  There’s nothing like the fear of deploying an application and reducing a business critical system to it’s knees.

I’ve worked on a handful of projects where I’ve taken it upon myself to reign in this deployment fear and chaos, and more often than not, the fear has it’s roots in an uncontrolled development environment.

Configuring A Development Environment

I have very few suggestions regarding which tools you should use to develop your software.  I think that you gain the maximum benefit by allowing developers to pick and choose the absolute best tools for the job, be that technically or by preference.  Allowing developers to use their tools of choice makes happy developers.

By contrast however, the one thing I’m really certain about is that every development environment should start with an absolute base configuration that should not be changed.  This is the first way to solve the eternal “works on my machine” developer problem.  In essence, if all the development environments start off the same, there’s no excuse for a “works on my machine” error.  Not only will this reduce friction between team members, but it’ll speed up the development process because you’re all working to one known configuration.

Furthermore, the development environments alone shouldn’t share this configuration, they should share the base configuration with your production servers.  A huge percentage of software release issues are relating to the differences between production and test and development and the only way to solve what amounts to human error is to make this discrepancy go away.

In the current IT landscape where virtualisation is painfully simple (and often free) there are no excuses not to have a development environment (or at the very least development staging virtual machine) that accurately and completely mirrors your production server.  If you’re struggling with licenses of third party software, produce mocks / proxies / approximations.  I can assure you that the time you spend building these glorified testing rigs will work healthily towards eradicating development-to-production errors.

If It’s Worth Doing Once, It’s Worth Doing Three Times

Another nasty pattern I see frequently is the development-to-production pattern.  If you’re compiling software on your development machines and releasing it to production, “you’re doing it wrong”.  What’s worse than a “works-on-my-machine” conversation with a colleague?  You got it, a “works-on-my-machine” conversation with a system administrator.  It’s not an excuse.  Stop it.

You need to start dog-fooding your deployments, and this means environment cloning and real test servers.  I always follow a simple pattern; developer machine -> developer test box -> system-test box -> user acceptance test box -> production.

In order to do this you’ll need a series of environments (physical or virtual)

• Developer machine
• Developer environment
• Test environment
• UAT environment
• Production environment

That might sound like a lot of hardware but it really isn’t.  Use virtualisation and use it well.  Clone your virtual environments if need be.  Use snapshots and rollbacks to test deployments.  It shouldn’t take any time to administer (and I’ll explain why in the next section) and by the time your code is running on production you’ll have moved it through three different environments.  This is absolutely key to having faith in your deployment procedures.  When you’ve deployed your code so many times that you’re sure it works, in an environment that’s a mirror image of production, production is a far less scary place to be deploying to.

The key to this technique is that the environments all have very clearly defined purposes.  The developer produces his code changes to a system on his development box.  The code seems to work so it’s deployed to the developer environment and checked into your source control repository.  The development environment should be treated as a place where things could potentially get messed up, but it’s also a good place to host joint resources (a database, some middleware) that all the developers make use of.  If the code works in the developer environment it can be promoted to the test environment for the system tester (or a developer wearing a system testers hat) to test.  Once passed on the test box, the code changes can be promoted to the UAT environment for users to preview coming code releases.  Once these changes have been accepted, the code is released into production.

Beware of release insanity.  It makes perfect sense to frequently deploy to the developer environment but only occasionally promote code up to test and UAT.  This should fit around your way of working, just be sure to keep the conceptual integrity of the environments.  If you find a bug in test, don’t fix it in the test environment, fix it in development. It’s up to you to ensure you know what code is running across your environments to avoid any nasty surprises.

One Install To Rule Them All

At this point you’re probably thinking that this all sounds like a huge amount of effort but it isn’t.  The reason it isn’t is that you should, for the most part, be able to deploy your entire system in about 5 clicks / commands.  Don’t let your deployment mechanism be the week link in your application.  There are tonnes of install technologies on the market, both free and costly, to match whichever platform and tool chain you’re working in.

I wish I could remember where I read it, but one of the pieces of setup advice I read long ago that always stuck with me is that you should aim for a one-click install.  You might not achieve it, but you should aim to get very very close.

You need to invest time in configuration management for your application.  Build tools to auto-generate configurations for your deployment environments, build tools to automate setup and deployment.  These tools are worthwhile, because from the moment they’re built, you never need to worry about deploying your software.  They remove risk, and anything that removes risk from software development should be embraced.

Once you’ve build these tools and installers you’ll find your able to do things that you only thought were possible in large companies with seemingly limitless resources.  You’ll be able to configure nightly builds using a continuous integration package of your choice and have the nightly builds install themselves in your developer environment,  You’ll be confident that at any point you can provide a working copy of your software on request.

No Fear Of The Unexpected

The wonderful benefit you get at the end of this process is confidence in your software.

So you’ve released your system to production, do you then feel the need to slavishly test the system in
the live environment? 

Imagine that urge going away, knowing that by the time the code hits production that any environmental glitches that could manifest have had the ability to fail three times already.  Any amount of cursory testing you could perform at this point should pale in comparison to the testing you did in the test environment and the user testing environment.  If your “at a glance” testing somehow discovers a bug, you’ve done something very very wrong in the lead up to release.  You should be confident in the fact that your code has been run exhaustively and often by the time it’s reached production.

The world isn’t perfect however, and it’s obviously advisable to a quick audit of the software once installed, just don’t force yourself to repeat your system tests in production (an obvious anti-pattern) and you’ll gain no benefits from doing so.

If you can devise an automated tool to verify your deployments after install, you’ll be able to round off your sense of deployment-security.

A Zen-like State Of Satisfaction

I opened this piece with a few simple questions.

• Are you unafraid of deploying your code to production?
• Do you know how your code will react in your production environment with production configuration?
• Does your code react in the same way in your production environment as it does in development?
• Are you confident in the repeatability of your installation procedure?

I can answer yes to all three questions because I understand my environments and I hope you can too.

[NOTE: This post relates to an old version of MobileTFL - Please see http://www.davidwhitney.co.uk/software for the latest version]

I’ve just compiled what I hope is the final version of this application barring London growing extra underground lines.

I’ve written a small Windows Mobile application for phones running WinMo 5.0+ with Compact Framework 2.0+.  It syndicates the Transport for London live data in similar way to the pre-existing iPhone and Android applications allowing you to view the current tube network status, planned engineering outages and local station notifications at the push of a finger on your Windows Mobile.

It’s currently living at a holding page at http://www.davidwhitney.co.uk/software (soon to become a larger site) and it’s a free download for anyone that uses Windows Mobile.

The 1.0.0.4 update is a reaction to some feedback I’ve received over at http://www.xda-developers.com and comes complete with coloured glyphs representing each tube line and enhanced resolution support for devices with “unusual” resolutions such as the Sony X1.  I’ve also had a stab at some font and UI element scaling so hopefully the interface will look quite natural on all your devices.

I’ve only managed to test it in the wild on a Touch Diamond, Touch Cruise and the Windows Mobile 5.0 Virtual Machine (along with in Windows).  It seems to run admirably in those conditions.

Obviously it requires a data plan so beware of any provider costs.

You can download the .cab directly at this link: http://www.davidwhitney.co.uk/software/repository/MobileTFL/1.0.0.4/MobileTFL.Setup.CAB

Recently I’ve been trying to pay a debt of sorts.  I use the software provided on the fantastic XDA-Developers frequently on my Windows Mobile devices and have been doing for almost three years now.

So currently, if I have a little bit of free time, I’ll nip past their development section and try and fulfil a random request.

Today’s request might come in useful to a little bit of a wider audience so I’ll post it here.  The initial problem is outlined in this thread and the upshot was that someone needed a very simple Http server that could run on Windows Mobile, written in C#’, and was programmable.

What the guy really seemed to be looking for was something that used the Http protocol to return random computed data, so after a tiny bit of googling and a simple MSDN example, I’ve built a really simple web server for Windows Mobile.

Amusingly, my sample doesn’t behave much like a web server at all.  It just tells you what you requested, but it should be enough to get you going in the right direction.  It’s derived from the MSDN example with some additional sugar, and I certainly wouldn’t suggest that it’s got a threading model that’d stand up in a production environment (at a glance it looks like it’d process requests in sequence…) but hopefully it’s useful to somebody looking to produce a simple server, or who is just interested in how Http works.

I’m not really sure if I can accurately call it a webserver, seeing as it doesn’t even support a full set of Http Verbs, but you get the idea.

I’ve not actually bothered compiling this on a mobile device yet (lack of inclination) however seeing as it was explicitly based on a socket programming for Windows Mobile MSDN example, I suspect it’ll work just fine.  No warranty, do what you will with it.

Download Simple C# Http Server for Windows Mobile (Source Code Only) (7kb)

Ever wanted to generate a change log from all your (dubious) Subversion repository comments?  For some reason want to do it in C# or just by calling an exe?  Now you can!  You can even tie it in to continuous integration!

About 6 months ago I spent an evening porting the Subversion Change Log generator to C# in order to integrate Change Log generation it into our Cruise Control.Net nightly builds at work.

It’s not a direct port, more a C# reimplementation, but it produces the same output as svn2cl (which in itself was inspired by cvs2cl).  The gist is, that it calls svn.exe on the command line and retrieves the full history of comments for whichever svn path you request.

It’s not the most elegant thing I’ve ever written (it pretty much just spawns the svn.exe and pipes the output through an XSL transform) and requires you having a command line svn.exe on your system (which if you use subversion on windows is pretty likely), but if you do you should be able to generate change logs from your repositories by calling the compiled executable like so:

SubversionReportProducer.exe –style=ChangeLog.xsl –outputLocation=out.txt –repositoryPath=svn://repo/trunk

If you don’t want to pass the subversion exe path and repository path through on each execution you can specify them by modifying the values in SubversionReportProducer.exe.config.  Those values will always attempt to load from configuration, and then attempt to load from a command line parameter (overwriting any app.config settings) allowing for maximum flexibility in batch scripts and automation.

I’ve bundled it with a couple of XSL transform files to produce some output, mostly taken from svn2cl and one subtle adaption that we use in our build process (ingeniously named changelog.xsl).

Your mileage by vary, but this could well be of use to someone.

Source code provided, feel free to contact me about any bugs but I’m pretty much just kicking this out into the wild..

Download Source
Download Compiled Executable
Requires .Net 2.0+