[NOTE: This post relates to an old version of MobileTFL - Please see http://www.davidwhitney.co.uk/software for the latest version]

I’ve just compiled what I hope is the final version of this application barring London growing extra underground lines.

I’ve written a small Windows Mobile application for phones running WinMo 5.0+ with Compact Framework 2.0+.  It syndicates the Transport for London live data in similar way to the pre-existing iPhone and Android applications allowing you to view the current tube network status, planned engineering outages and local station notifications at the push of a finger on your Windows Mobile.

It’s currently living at a holding page at http://www.davidwhitney.co.uk/software (soon to become a larger site) and it’s a free download for anyone that uses Windows Mobile.

The 1.0.0.4 update is a reaction to some feedback I’ve received over at http://www.xda-developers.com and comes complete with coloured glyphs representing each tube line and enhanced resolution support for devices with “unusual” resolutions such as the Sony X1.  I’ve also had a stab at some font and UI element scaling so hopefully the interface will look quite natural on all your devices.

I’ve only managed to test it in the wild on a Touch Diamond, Touch Cruise and the Windows Mobile 5.0 Virtual Machine (along with in Windows).  It seems to run admirably in those conditions.

Obviously it requires a data plan so beware of any provider costs.

You can download the .cab directly at this link: http://www.davidwhitney.co.uk/software/repository/MobileTFL/1.0.0.4/MobileTFL.Setup.CAB

A very quick late night post.

I’m a big fan of HTC’s Touch Diamond once you remove all the cruft that the major phone networks like to cram onto their devices.  It’s a pocket sized, powerful smart phone.  I picked one up only a few months ago as a replacement for my HTC Touch, which was starting to feel very very slow compared to some of the handsets on the market, and the addition of HDSPA on the Diamond was the clincher.

Anyway, as a result the good lady has been quite impressed with the functionality of the Diamond.  She works in the media and the ability to access the internet with an almost desktop like experience (and without an iPhone) was very appealing.  While we were in town this weekend sorting out a new contract for her, I noticed that Google’s G1 ships with a London Tube service status application.  Seeing as Eleanor lives in London it seemed like an instantly cool thing to have on hand, but to my surprise, there isn’t anything comparable available for Windows Mobile (or there is and it’s too difficult to track down).

So I wrote one.

I’ve put up a really REALLY retro holding page at http://www.davidwhitney.co.uk/software for the purpose of releasing this application, hopefully over the coming months I’ll use it as a gather place for all the applications I’ve written, both free and pay-for, but for the moment, feel free to go and pick up a copy of the ingeniously named MobileTFL (after the transport-for-London website, where the application sources its data).

If you’re too lazy to click through one link:

Download Mobile TFL for Windows Mobile Here (Cab file)

You’ll have to forgive the exceptionally low-fi website and hilarious low res Visual Studio 2008 virtual machine screenshot.  I’ve only really tested the app on the Diamond I have sitting on my desk right now and it works a treat.

Obviously it requires you have a data plan that’s from a company that believes in Mobile internet rather than customer robbery but it only uses a tiny amount of data (it’s just a Http request).

It also requires the Compact Framework.  I’ve built it under CF3.5 but I suspect it’ll run just fine under CF2.0 (they’re binary compatible after all).

Feedback is more than welcome, just send me an email about it.  I’m especially interested on how the rendering looks on your devices, as the rendering code was written quite quickly and as soon as it looked ok in the VM and on my device I pretty much packaged it up.

I doubt it’ll brick your device, but if it does, you know the drill, it’s your problem.

Time for sleep now.

[Update: Microupdate to version 1.0.0.3 to fix a few bugs and aestetics, the above link has changed, revisit it for the new version]

I’ve jumped right in to the deep end with my game project (which is going under the working title of “Encounters” – I needed to call the solution something!) and one of the core design goals of the project is the multiplayer focus of the game.

I’m a huge fan of Id Software.  John Carmack is probably my favourite “celebrity computer programmer”.  He’s incredible smart and consistent and has made some of both the most influential and my own favourite games.

Standing on the shoulders of giants – The “Quake” model

I’ve always thought the “Quake model” of single player and multiplayer game development to be a good one.  For those that don’t know, one of the simple design tenants of the Quake engine is that everything is a multiplayer game.  When you play the campaign in the original Quake what the game actually does is start a local game server which you then connect to.  Due to the proliferation of the Quake engine and other engines that have been inspired by it’s design this became quite a common way to build first person shooters with multiplayer support from the offset.  It reduced the implementation of multiplayer to simply having a second player connect to the active session.

Standing on the shoulders of giants, I’ve decided that this is the model I wish to follow for Encounters, especially seeing as the primary work-in-progress game design desires 4-player coop as the main campaign (with NPC assistants if you play with fewer humans).  Because of the way supporting multiplayer from the offset effectively means that all the games “thinking” is done in the server component (collision detection, cheat prevention, state management..) I decide that it’d be the best place to start the implementation.

Prototyping

So far so good, I spent an hour or two last night designing a simple state managing game server and retrofitting it to my previous prototype (an arena with a player-controlled unit moving around it) and moving all the logic and validation to the server side.

At this point the “game server” was a singleton class that the game accessed though a fake “proxy” class (left empty for eventual network implementation) and it worked pretty well.  The collision detection worked on the “server side” when the server was running in the same application domain and as a proof of concept everything was quite sound. I’m still left facing a few issues surrounding the frequency of syncing with the server and client side prediction, but they’re all relatively well solved problems in gaming (and there’s certainly some prior art to take inspiration from in XNA tutorials around the web).

Building a network stack

Once the proof of concept was working I started looking into the networking support provided by XNA and hit the mother of all roadblocks.  Because I’m targeting both the 360 and Windows, networking becomes significantly more complicated.  See, Microsoft offer no direct network access on the 360, via XNA, or even to their licensed partners (internet hearsay claims).  They provide Xbox Live APIs as part of the XNA framework however, which seemed like a decent solution until I realised that these networking APIs function only on a 360 as part of Xbox live.

Not so multiplatform really

So I’ve started thinking around the problem.  I do a lot of work with WCF in my day-job writing distributed systems so my obvious inclination was to provide a WCF client-server architecture for the Windows targeted version of the game.  This adds an additional set of issues.  For a start it looks as though the WCF assemblies (System.ServiceModel) are probably not available on the 360.  You’d think that’s not much of a problem if you’re going to use the Live APIs, but then you realise that if you’re using a version of .Net 3.5 prior to the first service pack, you need to annotate any classes you want to transfer via WCF with the DataContract attribute.  Which is in the System.ServiceModel namespace.

I’m not sure (as I currently don’t have a subscription to deploy my test code onto my 360) what version of the framework the 360 is running, but I’d hazard a guess that it’s pre SP1.  I’m going to have to further research this problem, because if you can use SP1 on the 360 (which supports serialization without any attributes on the classes) then we’re home free.

Multiplatform networking code using Inversion of Control

In the interim I’ve devised a cunning solution (or perhaps workaround, I’m not sure).  See, all of these issues lead me to need two entirely separate network stacks for my game.  One that supported the Windows, and the other that supported Xbox live, without contaminating my data model with mark-up that’s useable in only one or the other of the stacks.

Castle Windsor to the rescue.

This is purely speculative as I haven’t attempted to implement the 360 network stack yet, but I intend to use the Castle Windsor IoC (inversion of control) container to load a separate network stack at runtime, depending on the platform.

My idea is that the game knows about two interfaces IGameServer and IGameServerConnection.  I then create two entirely separate assemblies, one with the networking code for the PC, the other with the Live networking code.  The PC version contains a class called WcfGameServer which implements IGameServer (I’ve already written this code), and a class called WcfGameServerConnection which implementes IGameServerConnection (and acts as a hand crafted WCF proxy).  The 360 implementation will feature XBL counterparts to these classes.

As far as the game is concerned, it’ll use the Windsor container to load an instance of IGameServer, call the StartListening(); method, then use Windsor to load an instance of IGameServerConnection() and call JoinGame() on that connection.  This way, the specific networking implementation is entirely removed from the game and hidden behind these two simple interfaces.

I’ve currently got a good way through implementing the WCF version of this model though I’ve hit a few snags on the way.  Because I’m deliberately developing in a pre-SP1 environment for the sake of this exercise, WCF doesn’t like serializing the game model to send over the wire.  As a result I’ve had to produce an inelegant hack to work around my desire to keep the System.ServiceModel assembly reference clear of my game model.

Trying to keep it light

I’ve marked up my model with Serializable attributes (.NET 1.1, that’ll be fine) and inside my WCF implementation of IGameServerConnection I’m marshalling all my data into byte[]’s before sending it over the wire.  This isn’t ideal as it requires the WCF implementation to manually deserialize the byte[]’s into their correct data types in the service implementation and the client library, but it does work.  Unfortunately at the moment these byte arrays are being stored as XML before being sent over the wire (as is the default WCF way) so I’m going to need to force WCF to binary serialize all it’s data rather than bloat my packets (and as a result, the game latency).  I’m using Net.Tcp so it’s pretty lightweight as far as protocols go but I suspect I’ll need to do some additional fine tuning to make the WCF implementation viable.

Either way, I’ve got a good feeling about the model, subtle hacks aside, and I think this is quite a good way to target both platforms with minimal impact to your game code.

So this is day 3 of development (I’m sure I’m going to loose count pretty quickly).

[Footnote]

It occurs to me that I got a few fundamental things wrong when this post was originally written regarding the requirements for .Net 3.5 SP1.  These mistakes almost complicated the design of the networking stack.  Read more in my follow up here.

Recently I’ve been trying to pay a debt of sorts.  I use the software provided on the fantastic XDA-Developers frequently on my Windows Mobile devices and have been doing for almost three years now.

So currently, if I have a little bit of free time, I’ll nip past their development section and try and fulfil a random request.

Today’s request might come in useful to a little bit of a wider audience so I’ll post it here.  The initial problem is outlined in this thread and the upshot was that someone needed a very simple Http server that could run on Windows Mobile, written in C#’, and was programmable.

What the guy really seemed to be looking for was something that used the Http protocol to return random computed data, so after a tiny bit of googling and a simple MSDN example, I’ve built a really simple web server for Windows Mobile.

Amusingly, my sample doesn’t behave much like a web server at all.  It just tells you what you requested, but it should be enough to get you going in the right direction.  It’s derived from the MSDN example with some additional sugar, and I certainly wouldn’t suggest that it’s got a threading model that’d stand up in a production environment (at a glance it looks like it’d process requests in sequence…) but hopefully it’s useful to somebody looking to produce a simple server, or who is just interested in how Http works.

I’m not really sure if I can accurately call it a webserver, seeing as it doesn’t even support a full set of Http Verbs, but you get the idea.

I’ve not actually bothered compiling this on a mobile device yet (lack of inclination) however seeing as it was explicitly based on a socket programming for Windows Mobile MSDN example, I suspect it’ll work just fine.  No warranty, do what you will with it.

Download Simple C# Http Server for Windows Mobile (Source Code Only) (7kb)

I’ve been working with WiX ( wix.sourceforge.net ) for generating application installers over the past few weeks.

The project is rapidly evolving (if I recall, it was one of Microsofts first forays into open source development) but as a side effect finding up to date documentation can be a little taxing. The documentation is good and quite comprehensive, but often subtly incorrect or outdated.

Anyway, we have a few services at work that require certificates to be installed at install time into the Windows certificate store. Previously we had a couple of custiom actions designed to configure the user and store, but after a little investigation it appears like this functionality comes for free in the Wix toolkit.

It’s confusingly in the IIS extensions, which is a bit of a misnomer- it’s only in there because it was originally designed to install certificates for web servers, however it works perfectly for any certificate.

So how do you do it? In Wix3, ensure you first have a reference to WixIIsExtension.dll (in the default install, it’s in c:\Program Files\Windows Installer XML v3\bin) in your project if you’re using voltive, or manually linked if you’re building on the command line. The following example is of a fragment which installs two certificates, one as a Root certificate authority and another as a certificate in local machine.

<?xml version=”1.0″ encoding=”utf-8″?>
<Wix xmlns=”http://schemas.microsoft.com/wix/2006/wi”
xmlns:iis=”http://schemas.microsoft.com/wix/IIsExtension”>

<Fragment>
<Directory Id=”Directory_Certificates” Name=”Certificates”>
<Component Id=”MyRootCert.cer” Guid=”*”>
<File Id=”MyRootCert.cer” Name=”MyRootCert.cer” Source=”..\..\Path\To\MyRootCert.cer” />

<iis:Certificate Id=”Certificate.RootCA”
Name=”MyRootCert.cer”
Request=”no”
StoreLocation=”localMachine”
StoreName=”root”
Overwrite=”yes”
BinaryKey=”Certificate.RootCA.Binary”
/>

</Component>
<Component Id=”RandomCert.p12″ Guid=”*”>
<File Id=”RandomCert.p12″ Name=”RandomCert.p12″ Source=”..\..\Path\To\RandomCert.p12″ />

<iis:Certificate Id=”Certificate.MnpTestCertificate”
Name=”RandomCert.p12″
Request=”no”
StoreLocation=”localMachine”
StoreName=”personal”
Overwrite=”yes”
BinaryKey=”Certificate.RandomCert.Binary”
PFXPassword=”myCertPassword_Optional”
/>

</Component>
</Directory>

<Binary Id=”Certificate.RootCA.Binary” SourceFile=”..\..\Path\To\MyRootCert.cer” />
<Binary Id=”Certificate.RandomCert.Binary” SourceFile=”..\..\Path\To\RandomCert.p12″ />

</Fragment>

<Fragment>
<ComponentGroup Id=”Component.InstalledCertificates”>
<ComponentRef Id=”MyRootCert.cer” />
<ComponentRef Id=”RandomCert.p12″ />
</ComponentGroup>
</Fragment>

</Wix>

Ever wanted to generate a change log from all your (dubious) Subversion repository comments?  For some reason want to do it in C# or just by calling an exe?  Now you can!  You can even tie it in to continuous integration!

About 6 months ago I spent an evening porting the Subversion Change Log generator to C# in order to integrate Change Log generation it into our Cruise Control.Net nightly builds at work.

It’s not a direct port, more a C# reimplementation, but it produces the same output as svn2cl (which in itself was inspired by cvs2cl).  The gist is, that it calls svn.exe on the command line and retrieves the full history of comments for whichever svn path you request.

It’s not the most elegant thing I’ve ever written (it pretty much just spawns the svn.exe and pipes the output through an XSL transform) and requires you having a command line svn.exe on your system (which if you use subversion on windows is pretty likely), but if you do you should be able to generate change logs from your repositories by calling the compiled executable like so:

SubversionReportProducer.exe –style=ChangeLog.xsl –outputLocation=out.txt –repositoryPath=svn://repo/trunk

If you don’t want to pass the subversion exe path and repository path through on each execution you can specify them by modifying the values in SubversionReportProducer.exe.config.  Those values will always attempt to load from configuration, and then attempt to load from a command line parameter (overwriting any app.config settings) allowing for maximum flexibility in batch scripts and automation.

I’ve bundled it with a couple of XSL transform files to produce some output, mostly taken from svn2cl and one subtle adaption that we use in our build process (ingeniously named changelog.xsl).

Your mileage by vary, but this could well be of use to someone.

Source code provided, feel free to contact me about any bugs but I’m pretty much just kicking this out into the wild..

Download Source
Download Compiled Executable
Requires .Net 2.0+

So, it looks like Free Radical, a Nottingham based games studio, closed its doors today. 

The surrounding scenario isn’t a pretty read (or at least the picture the gaming press paints isn’t) and it’s the latest in a trend of SME sized games developers folding despite the games industry growing month on month with disregard to the current economic climate.  Month after month websites like Gamasutra seem to report record sales of both console hardware and games (and that’s ignoring the Wii shaped monolithic elephant in the corner).

To directly quote a Kotaku commenter (Pyrefly)

“All that will emerge will be EA, Ubisoft and Activision with each consoles respective first party devs. It would be awful for all us gamers to keep losing more and more studios.”

This seems to be the pervading feeling amongst gamers but I can’t help feel that it might be slightly off target.  Nobody likes to see the little guys fail.  In any community people love rooting for the people they can identify with, with the human faces to development.  However, I just can’t help feeling that the studios currently suffering aren’t just suffering as a result of the economic climate, but more because of the evolving nature of games development. 

Gamers seem to be pointing fingers blindly at the mega-publishers as the cause of the collapse of these small studios but I can’t help feel that this isn’t really the case.  It appears as if they’re really just the victims of technological progression in a time of restricted finances.

Nobody likes seeing the little guys fail, but just to single out today’s example.  From memory (and a little wikipedia verification), Free Radical were formed as an offshoot of Rare, started by some of the Goldeneye and Perfect Dark team.  They had pedigree and went on to produce (what wikipedia claims) is the “highest-ranked first person shooter on the Playstation 2″, Timesplitters.  Since then, they’ve released a few Timesplitters sequels, Second Sight on the xbox, Haze on the Playstation 3 and they were apparently working on an unknown Lucasarts game (speculatively, Star Wars: Battlefront 3).

I’m going to go out on a limb here, I never played second sight (though I hear it was decent), I disliked all of the Timesplitters games (they had no soul, and I found them utterly trite) and Haze (which I also haven’t played) was critically panned and a commercial failure.  In my own personal opinion, their games sucked (straw-man here, they’re not terrible, but I’d not choose to play them).

Don’t kick a man while he’s down, I know.  I’m sure the developers that worked at Free Radical are exceptionally talented.  Technologically, their products all seemed very sound and polished, but from the rundown, they’re not exactly iD software or Epic.  They probably fall into the “averagely reviewed, decent sales” bracket of games companies.

I don’t think what happened to Free Radical today was necessarily their fault.  Whilst people point the finger at the big publishers for not supporting enough SME innovation, I think actually this is indicative of the diverging nature of game development at the moment.

This year has seen the resurgence of the tiny independent game studios using the new content delivery channels, operating outside of the bounds of the current mega-publishers and there seems to be an increasing little and large divide to games development.

The middle tier seems to be being either gobbled up by majors (perhaps saved from bankruptcy) or collapsing and if I were to suggest a reason, I’d say that it’s probably indicative of the increased pressure of producing “AAA quality” games for the current generation of consoles.

You can’t produce a Halo 3 or Mass Effect on a shoestring budget anymore.  You either have to think really big and get help from a large publisher (EA have done some great work this year with games like Mirrors Edge and Dead Space, Ubisoft I’d argue have been doing great work for a number of years), or you need to actually scale down to smaller productions to be competitive with a smaller team.

Huge titles are plagued by content creation, gamers want more re-playability and value for money.  Your games have to look fantastic.  You have to be able to afford expensive middleware engines and components. 

Middle tier, under resourced attempts at AAA grade titles end up like Haze and Too Human it seems, whilst gems like Braid and World of Goo thrive in their microcosm and appear to be sustainable and competitive in their corner of the market (I have no idea what the financial rewards of small productions look like, to be clear and open).

I guess there’s always the option of pushing out a bunch of Wii shovelware if you’re an SME that’s running out of ideas, I hear people buy anything with added waggle.

At the end of the day while we’re losing more studios I’m curious if we’re actually loosing talent.  I’m not sure what the job markets like for a game developer, but I’d have thought that actual talent tends to get re-employed.  Your average coder producing average games might suffer, but anyone with a stellar portfolio (or even a half decent one) in an industry so starved of skills must be re-employable?  I guess the bigger risk is losing game development talent to other areas of computer science where jobs are seemingly more secure and the work is less based around a sustained 8 month crunch.  You really have to be in it for the love of games or be working for a good company (as seemingly the scenario this morning at free radical proved, the staff appear to have been quite badly treated in the last few hours).

I just suspect that the industry is going through a period of restructuring at the moment.  Maybe it’s time for the major labels to stand up for more than just profit margins and defend the culture that helps them thrive.  But at the end of the day, the SME with world class talent will probably be just fine.  Look at iD, I’m sure John Carmack isn’t too worried that somehow his skill set will become irrelevant and he won’t be able to compete.

It genuinely is always sad to see the little guys go, you often need the space in a small company to come up with some of the most brilliant ideas, and as a fellow developer, I know how intellectually nourishing small team environments are, I just hope that the majors that hold the financial security (or what’s left of it) choose to nourish and re-house these individuals so everyone can thrive (and
more importantly, keep making great games for me to play).

I don’t work on games, and I’m just a casual observer of the games industry, albeit an enthusiastic one, so take everything above with a pinch of salt.  This has been one of the best years for games I can remember, I just hope this year end doesn’t wipe out lots of talented developers.

Cornered By Technology

It’s not uncommon in enterprise software development to be tasked with integrating with a third party platform that just won’t play nicely however hard you try. These scenarios often creep up on you at the most unexpected time, be it as a requirement, or a partnership with an uncooperative or unskilled vendor or technical partner. Even worse, it can often become clear that the relationship between your platform and a third party platform is effectively untenable after you’ve invested significant development effort to get “most of the way” there.

The Need for Application and Data Separation

The really bad news is that this isn’t an uncommon scenario, and there are countless businesses operating whilst being held to ransom by systems that they paid for. Unfortunately, these systems cannot be abandoned, normally because the system holds some crucial operational data that the company would cease to function without. Antiquated CRM solutions, Billing platforms and databases are a few of the most common places these issues arise. The fact that you’ve been effectively backed into a corner by a technology partner isn’t the root of the problem however.

These scenarios normally spawn from a historic or bad choice in application design either on your own part or on the part of the offending partner as a side effect of intertwining corporate data and applications into a tightly coupled relationship. You shouldn’t offer the keys to your business to any single piece of software, especially a piece of software operated by a third party, and it’s of the highest importance that you have a clear exit strategy if you choose to enter such an abusive relationship.

Anyone that works with real software on a day to day basis well knows that a full separation of software and the data on which it operates isn’t entirely practical. In reality, you are going to end up with applications controlling certain portions of your data in proprietary formats, but with due diligence and a mind for separation of concerns when choosing available solutions, business crippling crisis should be easy to avoid.

When choosing platforms and service partners, don’t skim over the details of data storage, try and ensure any application that you’re trusting with your business data is storing it in a way that you can migrate that data elsewhere if you so desire.

All People Perform To the Best of Their Abilities When Available Time and Constraints Are Accounted For

It’s very easy to become adversarial when faced with a business crippling technology partner or application and it’s of the upmost importance that you keep your ego and temper in check when dealing with an especially uncooperative partner. This sounds like an obvious piece of advice when dealing with people but it’s just as appropriate a guideline when dealing with software.

Don’t do or say anything you’ll regret, don’t antagonise your opposition (and if you’re unlucky, an uncooperative service partner will certainly become an opponent), and certainly don’t make any knee-jerk decisions regarding your own codebase. It’s important to remember that whilst the system you’re attempting to integrate with may well appear intentionally antagonising or impossible. It’s likely that the functionality of the system was not achievable in any other way, and you genuinely are working with the best product that is available. Concentrate on productive measures, not the conflict and stress of a business threatening problem.

It’s Not You, It’s Me

It should be quite evident that there’s only one solution to this particular problem; technical divorce.
The goals of a project to end an abusive software relationship are reasonably clear cut and simple:

1. Don’t negatively impact business operation without good reason.
2. Achieve data and application logic separation.
3. Escape with minimal impact on other integrated systems.
4. Take preventative measures; don’t get hurt again.

Your number one goal is to not disrupt day to day business during this period of technical therapy. You need to continue working with your technology partner on a daily basis in order to operate and it’s important that you don’t anger your partners or hinder the operation of the soon to be replaced system with disruptive development. Following good software development practices in general should ensure you avoid technical issues of these kinds.

Your second goal is to separate your mission critical data from the dying system. Data ownership is a huge task and not something that can be summed up in a few paragraphs. You may need business analysts to investigate who owns the data in your business and where it should reside. As a good guideline, it’s worth considering that as a business you should own all of your business data with specific applications fed on the data they require to operate. Results should be imported back into a master database or data-store. It’s not a silver bullet or a solution for everyone, but if you approach the problem of data ownership from this perspective then you’re at least thinking along the right track where your applications are secondary to your data needs. An order tracking system should only be concerned with order items and names, a billing system should only be concerned with financial data and billing, try and ensure you don’t get lost in a spaghetti mess of duplicated data.

Your third and most difficult goal is to make this process transparent to the end user by reducing the impact of the change on your other systems. I’ll spoil the fun now by telling you that this is probably impossible and that the user will see a difference. The key is ensuring that difference is only ever iterative improvement to existing business processes and never prohibitive to productive work.

The fourth goal is preventative and hard to define. Seeing as you’re going though this painful technical divorce, ensure that the amount of thought and work that goes in to this project is sufficient to prevent this scenario reoccurring. Employ good business analysts to help you design the project, investigate data ownership and data warehousing, invest in technologies that adhere to open standards and allow for data portability. Build your replacement system in such a way that when you want to replace it with the next wave of technology, that your own developers won’t face the challenges you’re facing now.

Designing a Solution

In order to escape from this heavily relied upon piece of software we’d do well to learn a few tricks from the people who have participated in Software Modernisation over the years.

The Wikipedia entry for software modernisation (http://en.wikipedia.org/wiki/Software_modernization) reads;

“Software Modernization is the process of understanding and evolving existing software assets.

There is a vast amount of highly functional, operational software, representing enormous commercial value deployed in organizations around the globe. To be precise, existing systems are defined as any production-enabled software, regardless of the platform it runs on, language it’s written in, or length of time it has been in production.

These entrenched software systems often resist evolution because their strategic value and ability to adapt has diminished through factors not exclusively related to its functionality. Common examples of such factors are a system’s inability to be understood or maintained cost-effectively, inability to interoperate or dependence on undesired technologies or architectures”

The goals of software modernisation are functionally similar to the goals of replacing an uncooperative system, albeit with a different motivation for the system replacement. This “modernisation” is happening due to the need to maintain defective but business critical operations in the short term rather than retaining currently good functionality in the long term.

Software modernisation is a whole topic in and of itself, but at its core there are two common methods of modernising software; black box modernisation and white box modernisation. These terms retain their standard computer science meanings.

Both modernisation practices require the encapsulation of the legacy system in modern code, creating a fresh set of APIs in a modern language, which in turn calls an API or function of the legacy system.

Black box modernisation involves calling only the publicly facing APIs of the legacy system where as white box modernisation involves a degree of knowledge of the underlying platform and its operations. It’s exceptionally common for extensive reverse engineering of a platform to take place in white box modernisation whilst black box modernisation is more or less a thin wrapping layer.

We’re going to take the idea of system encapsulation from software modernisation and treat our uncooperative system as though it were a legacy system and develop an API to encapsulate the systems functionality.

Function Not Form

During the “aggressive replacement” of the uncooperative system, modern development methodologies will get you a long way. You should keep a keen focus on loose coupling of the interface you create to describe the business needs and the underlying wrapping code of the system beneath it.

When designing your replacement API, you should endeavour to describe the business needs rather than mapping new API calls to the uncooperative system in a one to one fashion. You should take great care to describe function rather than form, decoupling the core business needs from the implementation underneath.

Good API design isn’t something you wake up with the innate ability to do, and thoroughly describing the business requirements of the uncooperative system in your new API will likely take several iterations. This isn’t a problem however, as it’s recommend that you iterate through a process of this scope rather than attempt some big bang replacement of the uncooperative system to avoid great risk and inconvenience to the business.

Your API should creep in to existence, method by method. Existing applications should be updated to use the new API methods as their developed to allow for an iterative migration to the new wrapping code.

Embrace, extend, extinguish

Once you’ve designed the API you wish to use to replace your uncooperative system, you actually have to produce code that maps between the two.

This is no small task and will probably be the majority of the work required by the project. This code is very implementation specific. You may be able to just codify your new APIs in to a few API calls of the uncooperative system itself, conversely you may have to deconstruct the behaviour of the system and replicate it in entirely fresh code. Either way, it’s exceptionally important that at this stage you keep your implementation separate from your API and service layer.

Thus far we’ve embraced and accepted our uncooperative system, we’ve extended a hand towards it in the form of a neatly designed API, and the final step of your project will revolve around replacing the system entirely.

Once your API is fully featured, and all the dependant systems in your business are referencing it for any communication they need to perform you can begin seeking a replacement system.

Choose wisely, at this point you’ll likely be more than aware of the pains involved in another costly migration! Take your time, there’s no hurry, developing your own solution is even an option.

Once you’ve chosen the replacement, you need to write some code again, this time to translate calls from your new API across to your replacement system. Because you did remember to keep your API decoupled from the implementation, right?

At this point you’ve succeeded in using an iterative approach to software design to fully replace a system that may have previously seemed inseparable from the business. The ROI of a project of this nature is hard to quantify, but I thoroughly believe that if you feel that you need to embark on a project of this nature, its completion will surely strengthen one of the most brittle parts of your operational business.

Congratulations! You made it out alive.

Footnote: Avoid the same thing happening to you

This is honestly a plea to anyone that’s ever going to write a system they expect other developers to communicate with. Practice designing APIs. Write them well. Document them well. Keep them up to date. Support them. Define concise operations. Be good. If everyone is “good” then hopefully fewer people will have to suffer difficult integration exercises.

It appears as though the Microsoft M.V.P’s that were called upon to advise on the technicalities of the forthcoming Entity Framework hit a little bit of a roadblock.

When I say “a little bit”, it seems as though Microsoft just point blank disregarded their warnings and recommendations in regard to creating OR mappers. It’s probably the first time I’ve seen this kind of scenario end with the technical advisors posting a warning and a general vote of no confidence.

I’d be lying if I didn’t say I felt a little disappointed that after making steps in the right direction and asking domain specialists for advice, that Microsoft entirely disregarded the advice of the specialists they consulted resulting into what seems like an unusable shipping product.

I’ve not attempted to use the entity framework in and form, but the technical criticisms in the vote of no confidence are quite explicit, and as a developer who makes extensive use of OR/M, if those criticisms are accurate (which is very likely) I’d certainly treat the entity framework with the same unfortunate disregard.

At least nHibernate isn’t broken!

There’s a very good argument for saying that the thing that makes or breaks a new programming language is actually the tools the work with and around it, rather than language features themselves.  If you ask the average programmer to guess the percentage of language features of their language of preference that they used on a daily basis, or even ever, I suspect many of them would respond somewhere in the 25-30% range. 

Programming languages these days are big.  By that I mean, programming frameworks these days are big.  Most people tend to not just use the language natively, but the selection of a preferred platform tends to be based on the features of the dominant framework available for that language.  As a simple statement this is neither here nor there, but it illustrates the point that the development environment, for a large proportion of its users, makes a language.  I’d argue that Visual Studio made Visual Basic popular, and that without it, it would probably haven’t stood a chance against Turbo Pascal and Delphi, even with all the clout of Microsoft.  So we rely on tools, and in honour of tools, here are a list of tools I find invaluable whilst developing in .NET.

JetBrains ReSharper

This is the king of all visual studio plug-ins.  Imagine it as a kind of, interactive FX cop while you work.  It offers background compilation, refactorings, stream lines existing visual studio functionality and basically makes you code better.  It teaches you things you didn’t know about the language and helps you improve your code quality proactively rather than retrospectively.  The biggest programming crutch known to man, and a descendant of the IntelliJ IDE for Java.  JetBrains have trials available and if you drop my name in you can get an improved trial period and a bit of a discount on purchases.  If I could take just one tool home, it’d be this.

Lutz Roeder’s Reflector

Ever thought that there was a bug in someone else’s compiled assembly?  Fancy a crawl around the internals of the .NET framework?  This is the tool for you.  Reflector translates the IL of a compiled assembly back in to C# and lets you poke around inside.  It’s very much read only, but if you ever think there’s a bug in something you’re not supposed to touch, you can prove it and code around it with the help of this excellent tool.

System Internals / Microsoft Debug View

I’ve written about the usefulness of this little application in the past.  Debug View lets you view the Windows Debug output stream in real time.  That’d be anything written using System.Diagnostics.Debug.WriteLine or the more traditional C / C++ equivalents.  Allows you to monitor an application built in a debug state in real time.  Perfect for debugging Windows services or other troublesome applications that you need to monitor whilst running during development.

Visual Studio Remote Debugger

If you have Visual Studio you actually already have this (the link above is to setup information).  It’s hidden away in C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\Remote Debugger (replace the visual studio version as applicable) and it allows you to attach an instance of Visual Studio to an application running on another machine.  Again invaluable if you’re writing distributed services and deploying to test servers, this application allows you to run your service or application on another machine, then connect to it using the visual studio debugger and break and step through your code as though it were on your local machine.  Great for when you deploy something to an integration test server, it fails, and you just can’t work out why without digging through the code in real time.

MbUnit and Rhino Mocks

Two for one, an xUnit derived unit testing framework, and Ayende’s mocking framework.  Unit test with confidence and unit test often.

ANTS Profiler

ANTS profiles your .NET applications at a level I’ve not seen in any competing products, giving you a line by line, method my method breakdown of execution time and code bottlenecks along with all the stats you could ever want to generate on a piece of code.  You really can use this to find horrible hidden performance issues in your code and fix them.

Honourable Mentions

Notepad++

Everyone has their own favourite notepad replacement utility.  This is mine.  I used Context for years but this is just better in every way..

Pro C# and the .NET 3.5 Platform

The only “giant programming bible” I’ve ever been able to read like a book.  Grab an E-book so it’s searchable and you’ll probably get more information than you could Google for.

These are just my personal preferences, and applications that I use on a daily basis.  Give a few of them a try and you’ll see your productivity increase and hopefully, your code improve.