Fear Of Deployment

When you start working on a new project there are four really important pieces of information you should be aware of regarding the deployment of your software

1. Are you unafraid of deploying your code to production?
2. Do you know how your code will react in your production environment with production configuration?
3. Does your code react in the same way in your production environment as it does in development?
4. Are you confident in the repeatability of your installation procedure?

Fear and uncertainty in the deployment of software projects is a common theme I’ve seen across all the companies where I’ve worked and it’s never good for your project.  The fear of deployment is always a good indicator of the confidence your staff have in the project they are working on.

A project with unknown qualities and suspicious (or known but unreported) bugs often causes an serious fear of deployment.  There’s nothing like the fear of deploying an application and reducing a business critical system to it’s knees.

I’ve worked on a handful of projects where I’ve taken it upon myself to reign in this deployment fear and chaos, and more often than not, the fear has it’s roots in an uncontrolled development environment.

Configuring A Development Environment

I have very few suggestions regarding which tools you should use to develop your software.  I think that you gain the maximum benefit by allowing developers to pick and choose the absolute best tools for the job, be that technically or by preference.  Allowing developers to use their tools of choice makes happy developers.

By contrast however, the one thing I’m really certain about is that every development environment should start with an absolute base configuration that should not be changed.  This is the first way to solve the eternal “works on my machine” developer problem.  In essence, if all the development environments start off the same, there’s no excuse for a “works on my machine” error.  Not only will this reduce friction between team members, but it’ll speed up the development process because you’re all working to one known configuration.

Furthermore, the development environments alone shouldn’t share this configuration, they should share the base configuration with your production servers.  A huge percentage of software release issues are relating to the differences between production and test and development and the only way to solve what amounts to human error is to make this discrepancy go away.

In the current IT landscape where virtualisation is painfully simple (and often free) there are no excuses not to have a development environment (or at the very least development staging virtual machine) that accurately and completely mirrors your production server.  If you’re struggling with licenses of third party software, produce mocks / proxies / approximations.  I can assure you that the time you spend building these glorified testing rigs will work healthily towards eradicating development-to-production errors.

If It’s Worth Doing Once, It’s Worth Doing Three Times

Another nasty pattern I see frequently is the development-to-production pattern.  If you’re compiling software on your development machines and releasing it to production, “you’re doing it wrong”.  What’s worse than a “works-on-my-machine” conversation with a colleague?  You got it, a “works-on-my-machine” conversation with a system administrator.  It’s not an excuse.  Stop it.

You need to start dog-fooding your deployments, and this means environment cloning and real test servers.  I always follow a simple pattern; developer machine -> developer test box -> system-test box -> user acceptance test box -> production.

In order to do this you’ll need a series of environments (physical or virtual)

• Developer machine
• Developer environment
• Test environment
• UAT environment
• Production environment

That might sound like a lot of hardware but it really isn’t.  Use virtualisation and use it well.  Clone your virtual environments if need be.  Use snapshots and rollbacks to test deployments.  It shouldn’t take any time to administer (and I’ll explain why in the next section) and by the time your code is running on production you’ll have moved it through three different environments.  This is absolutely key to having faith in your deployment procedures.  When you’ve deployed your code so many times that you’re sure it works, in an environment that’s a mirror image of production, production is a far less scary place to be deploying to.

The key to this technique is that the environments all have very clearly defined purposes.  The developer produces his code changes to a system on his development box.  The code seems to work so it’s deployed to the developer environment and checked into your source control repository.  The development environment should be treated as a place where things could potentially get messed up, but it’s also a good place to host joint resources (a database, some middleware) that all the developers make use of.  If the code works in the developer environment it can be promoted to the test environment for the system tester (or a developer wearing a system testers hat) to test.  Once passed on the test box, the code changes can be promoted to the UAT environment for users to preview coming code releases.  Once these changes have been accepted, the code is released into production.

Beware of release insanity.  It makes perfect sense to frequently deploy to the developer environment but only occasionally promote code up to test and UAT.  This should fit around your way of working, just be sure to keep the conceptual integrity of the environments.  If you find a bug in test, don’t fix it in the test environment, fix it in development. It’s up to you to ensure you know what code is running across your environments to avoid any nasty surprises.

One Install To Rule Them All

At this point you’re probably thinking that this all sounds like a huge amount of effort but it isn’t.  The reason it isn’t is that you should, for the most part, be able to deploy your entire system in about 5 clicks / commands.  Don’t let your deployment mechanism be the week link in your application.  There are tonnes of install technologies on the market, both free and costly, to match whichever platform and tool chain you’re working in.

I wish I could remember where I read it, but one of the pieces of setup advice I read long ago that always stuck with me is that you should aim for a one-click install.  You might not achieve it, but you should aim to get very very close.

You need to invest time in configuration management for your application.  Build tools to auto-generate configurations for your deployment environments, build tools to automate setup and deployment.  These tools are worthwhile, because from the moment they’re built, you never need to worry about deploying your software.  They remove risk, and anything that removes risk from software development should be embraced.

Once you’ve build these tools and installers you’ll find your able to do things that you only thought were possible in large companies with seemingly limitless resources.  You’ll be able to configure nightly builds using a continuous integration package of your choice and have the nightly builds install themselves in your developer environment,  You’ll be confident that at any point you can provide a working copy of your software on request.

No Fear Of The Unexpected

The wonderful benefit you get at the end of this process is confidence in your software.

So you’ve released your system to production, do you then feel the need to slavishly test the system in
the live environment? 

Imagine that urge going away, knowing that by the time the code hits production that any environmental glitches that could manifest have had the ability to fail three times already.  Any amount of cursory testing you could perform at this point should pale in comparison to the testing you did in the test environment and the user testing environment.  If your “at a glance” testing somehow discovers a bug, you’ve done something very very wrong in the lead up to release.  You should be confident in the fact that your code has been run exhaustively and often by the time it’s reached production.

The world isn’t perfect however, and it’s obviously advisable to a quick audit of the software once installed, just don’t force yourself to repeat your system tests in production (an obvious anti-pattern) and you’ll gain no benefits from doing so.

If you can devise an automated tool to verify your deployments after install, you’ll be able to round off your sense of deployment-security.

A Zen-like State Of Satisfaction

I opened this piece with a few simple questions.

• Are you unafraid of deploying your code to production?
• Do you know how your code will react in your production environment with production configuration?
• Does your code react in the same way in your production environment as it does in development?
• Are you confident in the repeatability of your installation procedure?

I can answer yes to all three questions because I understand my environments and I hope you can too.

I hate meetings but I love stand up meetings- contradiction?  No.

They’re a brilliant, informative, low impact way to keep communication fluid amongst your (small to medium sized) team, regardless of profession.

I’m going to talk a little about the reasons I’m writing about this, before getting on to the anatomy of my perfect stand up meeting, then tell you what stand up meetings could do for you and your team.

The Media Circus

My partner works in magazine journalism as the chief sub-editor for a title with a reasonably large international circulation.  Consequently, every month without fail, press week is a nightmare.

As is apparently the norm in the publishing industry (or what’s left of it) the eternal cycle of content creation is a very stressful process.  Freelancers are always late; things never hit the editors desk on time, people have to be continously chased up, facts need to be checked, and all before a press date.  If this isn’t complete by the press date, the company starts haemorrhaging money by the hour as the presses are literally “stopped”.

All of the above is exaggerated by the fact that writing is one of the industries that allows single workers to effectively “go dark”, drop off the radar and appear x-days later with finished content.  Not exactly communicative, ironically.

You Were At Work How Long?

She’s quite durable in regards to getting work done and quite effective at not letting the battle of press week become emotional or stressful, but I get the distinct impression it can be highly frustrating when you’re working until 11.30pm and getting the last tube home for a few days at the end of every month just to cover for the lack of structure to publishing a medium sized monthly title.

This pattern cumulated in a conversation with the good lady, with her expressing a little bit of exasperation on the topic and wondering if there was actually anything she could do to ever change this bad pattern of behaviour and improve the workflow of the entire team.  To me, there were a few simple tricks that could be employed to help facilitate change.

The Programmers Perspective

As a stark contrast I work in technology, specifically software development using what commonly comes under the banner of “Agile Methods” (there’s an on-going debate as to what counts as an agile method, but that’s beyond the scope of this piece, and frankly, my patience).

I’ve been lucky enough to only suffer one professional role that insisted on following an old trusty waterfall model of software development- that is to say, a software project where you first sit down and collate a monolithic requirements document, then produce a complete system design, then implement it, then deliver it.  Because of this, I’ve been working “agile” since my second professional role, and the first thing I was introduced to on my first day was the concept of the “Stand Up” (meeting).

I’d never come across one before, but they turned into one of the most useful parts of my day, and for something that lasts the best part of 10 minutes, that’s saying something.

What’s A Stand Up?

To quote the Wikipedia entry in it’s entirety:

“A stand-up meeting (or simply stand-up) is a daily team meeting held to provide a status update to the team members. The ‘semi-real-time’ status allows participants to know about potential challenges as well as coordinate efforts to resolve difficult and/or time-consuming issues. It has particular value in agile software development processes, such as Scrum, but can be utilized in any development methodology.

The meetings are usually time boxed to 5-15 minutes and are held standing up to remind people to keep the meeting short and to the point. Most people usually refer to this meeting as just the stand-up, although it is sometimes also referred to as the morning roll call or the daily scrum.

The meeting is usually held at the same time and place every working day. All team members are expected to attend, but the meetings are not postponed if some of the team members are not present. One of the crucial features is that the meeting is intended to be a status update to other team members and not a status update to the management or other stakeholders. Team members take turns speaking, sometimes passing along a token to indicate the current person allowed to speak. Each member talks about his progress since the last stand-up, the anticipated work until the next stand-up and any impediments they foresee.

Team members may sometimes ask for short clarifications but the stand-up does not usually consist of full fledged discussions.”

You’ll notice that there’s a hell of a lot of references to software development both in the prose and on the Wikipedia page and it surprised me to discover that outside of technology, stand ups aren’t a common thing, they essentially don’t exist.

I could hardly believe it at first, but a cursory Google search for “stand up meeting” concluded that outside of software, stand ups either don’t exist or are so rare as to not appear on general business websites.

For the sake of completeness, it’s worth noting that the internet is always skewed to technological topics in these kind of searches, but when all the results are about programming I can at least see some anecdotal evidence to confirm my suspicions.

My Favourite Stand Up Meeting Pattern, And What It Gets You

First things first, I really don’t like calling “stand ups” “stand up meetings”.  I always feel like the word meeting is loaded with negative and tedious connotations- meetings are boring and shitty and people don’t like going to them, period.

It’s important to note that because a stand up isn’t really a meeting, it doesn’t have the emotional baggage associated with one.  A stand up works best for teams of about 2-15 people (for the sake of brevity).

The Ideal Stand Up

A perfect stand up lasts between about 10 and 15 minutes (you’re “standing up” to try and enforce this) and takes place EVERY day.

The jist is, somebody starts it off (it doesn’t matter who) and everybody states what they did yesterday, what they’re planning on doing today, and anything that they think is going to get in the way of their current task.

Other team members are allowed and encouraged to comment and ask brief questions after the current speaker has finished.

The meeting should take place roughly 20 minutes after the start of the working day, to allow people time to sit down, get a coffee, have a chat, read their email and work out what they’re doing.

What You Get

The idea is, through the use of this simple 1 to 2 minute exchange from each person, everyone has a much clearer understanding of what each team member does.  Just ask yourself, do you REALLY know what all the members of your team do?  This is especially pertinent if your job involves managing your colleagues in some way.

The other participants should be encouraged to ask questions and comment after the speaker has finished but in-line with the comments they just made.  The idea behind this is that another team member may well have solved a problem that the speaker was suffering, but due to isolation, the pace of the workday or a lack of communication the fact that the problem has been solved may have been lost.

It’s a great way to share knowledge and has a very low impact on time.  The one caveat is that any lengthy discussions should be followed up privately after the stand up, because if you stray into detail or go off on a conversational tangent you’re wasting collective group time.

Psychological Effects

Stand ups make your team work harder and more efficiently.

That sounds absurd but stay with me.

Because, of the permanent rolling accountability of stating what you’ve been up to, stand ups are a good way of reducing the likelihood of your staff time wasting.  They’re nice and self regulating, nobody wants to stand in front of their team and say exactly the same thing, every day for a week.  It attracts the kind of attention that most people, lazy or not, prefer to avoid.

Past that, people enjoy having something new to say every day.  It’s a little like a grown up, caring sharing version of show and tell for the business world.  You don’t want to be the kid that doesn’t participate.  Peer pressure?  Totally.  But it’ll make your team more effective.

This side effect has two key uses.  Firstly, it positively encourages teams to be proactive, but secondly, it gives the managing member a very solid handle on team activities (or if relevant, the lack of).

Will This Work For Me?

Honestly?  I don’t know.  My experience is very slanted towards technology businesses so please remember your mileage may vary.

I think stand ups are a vital and useful part of an agile team however you shape them, and I’d say it’ll certainly work for you in software development.

Outside of technology, I genuinely don’t see how these clearly transferable practices wouldn’t apply and give you at the very least, team communication benefits.  Even if you think your team communicates well, ask yourself how much you know about both your colleagues’ roles, and what they’re doing today.

I found a good article on the anti-patterns of stand up meetings; the stuff to avoid.  I’d recommend having a quick read of it, in order to have all the information.

If you think the answer could be “not enough” than I’d really recommend you give it a go.  I have no idea if the good lady is going to suggest her team experiment with stand ups to help smooth out the pressures of their press week, but I think it would be an awesome idea for everyone.

I love stand up meetings and you should too!