Cornered By Technology

It’s not uncommon in enterprise software development to be tasked with integrating with a third party platform that just won’t play nicely however hard you try. These scenarios often creep up on you at the most unexpected time, be it as a requirement, or a partnership with an uncooperative or unskilled vendor or technical partner. Even worse, it can often become clear that the relationship between your platform and a third party platform is effectively untenable after you’ve invested significant development effort to get “most of the way” there.

The Need for Application and Data Separation

The really bad news is that this isn’t an uncommon scenario, and there are countless businesses operating whilst being held to ransom by systems that they paid for. Unfortunately, these systems cannot be abandoned, normally because the system holds some crucial operational data that the company would cease to function without. Antiquated CRM solutions, Billing platforms and databases are a few of the most common places these issues arise. The fact that you’ve been effectively backed into a corner by a technology partner isn’t the root of the problem however.

These scenarios normally spawn from a historic or bad choice in application design either on your own part or on the part of the offending partner as a side effect of intertwining corporate data and applications into a tightly coupled relationship. You shouldn’t offer the keys to your business to any single piece of software, especially a piece of software operated by a third party, and it’s of the highest importance that you have a clear exit strategy if you choose to enter such an abusive relationship.

Anyone that works with real software on a day to day basis well knows that a full separation of software and the data on which it operates isn’t entirely practical. In reality, you are going to end up with applications controlling certain portions of your data in proprietary formats, but with due diligence and a mind for separation of concerns when choosing available solutions, business crippling crisis should be easy to avoid.

When choosing platforms and service partners, don’t skim over the details of data storage, try and ensure any application that you’re trusting with your business data is storing it in a way that you can migrate that data elsewhere if you so desire.

All People Perform To the Best of Their Abilities When Available Time and Constraints Are Accounted For

It’s very easy to become adversarial when faced with a business crippling technology partner or application and it’s of the upmost importance that you keep your ego and temper in check when dealing with an especially uncooperative partner. This sounds like an obvious piece of advice when dealing with people but it’s just as appropriate a guideline when dealing with software.

Don’t do or say anything you’ll regret, don’t antagonise your opposition (and if you’re unlucky, an uncooperative service partner will certainly become an opponent), and certainly don’t make any knee-jerk decisions regarding your own codebase. It’s important to remember that whilst the system you’re attempting to integrate with may well appear intentionally antagonising or impossible. It’s likely that the functionality of the system was not achievable in any other way, and you genuinely are working with the best product that is available. Concentrate on productive measures, not the conflict and stress of a business threatening problem.

It’s Not You, It’s Me

It should be quite evident that there’s only one solution to this particular problem; technical divorce.
The goals of a project to end an abusive software relationship are reasonably clear cut and simple:

1. Don’t negatively impact business operation without good reason.
2. Achieve data and application logic separation.
3. Escape with minimal impact on other integrated systems.
4. Take preventative measures; don’t get hurt again.

Your number one goal is to not disrupt day to day business during this period of technical therapy. You need to continue working with your technology partner on a daily basis in order to operate and it’s important that you don’t anger your partners or hinder the operation of the soon to be replaced system with disruptive development. Following good software development practices in general should ensure you avoid technical issues of these kinds.

Your second goal is to separate your mission critical data from the dying system. Data ownership is a huge task and not something that can be summed up in a few paragraphs. You may need business analysts to investigate who owns the data in your business and where it should reside. As a good guideline, it’s worth considering that as a business you should own all of your business data with specific applications fed on the data they require to operate. Results should be imported back into a master database or data-store. It’s not a silver bullet or a solution for everyone, but if you approach the problem of data ownership from this perspective then you’re at least thinking along the right track where your applications are secondary to your data needs. An order tracking system should only be concerned with order items and names, a billing system should only be concerned with financial data and billing, try and ensure you don’t get lost in a spaghetti mess of duplicated data.

Your third and most difficult goal is to make this process transparent to the end user by reducing the impact of the change on your other systems. I’ll spoil the fun now by telling you that this is probably impossible and that the user will see a difference. The key is ensuring that difference is only ever iterative improvement to existing business processes and never prohibitive to productive work.

The fourth goal is preventative and hard to define. Seeing as you’re going though this painful technical divorce, ensure that the amount of thought and work that goes in to this project is sufficient to prevent this scenario reoccurring. Employ good business analysts to help you design the project, investigate data ownership and data warehousing, invest in technologies that adhere to open standards and allow for data portability. Build your replacement system in such a way that when you want to replace it with the next wave of technology, that your own developers won’t face the challenges you’re facing now.

Designing a Solution

In order to escape from this heavily relied upon piece of software we’d do well to learn a few tricks from the people who have participated in Software Modernisation over the years.

The Wikipedia entry for software modernisation (http://en.wikipedia.org/wiki/Software_modernization) reads;

“Software Modernization is the process of understanding and evolving existing software assets.

There is a vast amount of highly functional, operational software, representing enormous commercial value deployed in organizations around the globe. To be precise, existing systems are defined as any production-enabled software, regardless of the platform it runs on, language it’s written in, or length of time it has been in production.

These entrenched software systems often resist evolution because their strategic value and ability to adapt has diminished through factors not exclusively related to its functionality. Common examples of such factors are a system’s inability to be understood or maintained cost-effectively, inability to interoperate or dependence on undesired technologies or architectures”

The goals of software modernisation are functionally similar to the goals of replacing an uncooperative system, albeit with a different motivation for the system replacement. This “modernisation” is happening due to the need to maintain defective but business critical operations in the short term rather than retaining currently good functionality in the long term.

Software modernisation is a whole topic in and of itself, but at its core there are two common methods of modernising software; black box modernisation and white box modernisation. These terms retain their standard computer science meanings.

Both modernisation practices require the encapsulation of the legacy system in modern code, creating a fresh set of APIs in a modern language, which in turn calls an API or function of the legacy system.

Black box modernisation involves calling only the publicly facing APIs of the legacy system where as white box modernisation involves a degree of knowledge of the underlying platform and its operations. It’s exceptionally common for extensive reverse engineering of a platform to take place in white box modernisation whilst black box modernisation is more or less a thin wrapping layer.

We’re going to take the idea of system encapsulation from software modernisation and treat our uncooperative system as though it were a legacy system and develop an API to encapsulate the systems functionality.

Function Not Form

During the “aggressive replacement” of the uncooperative system, modern development methodologies will get you a long way. You should keep a keen focus on loose coupling of the interface you create to describe the business needs and the underlying wrapping code of the system beneath it.

When designing your replacement API, you should endeavour to describe the business needs rather than mapping new API calls to the uncooperative system in a one to one fashion. You should take great care to describe function rather than form, decoupling the core business needs from the implementation underneath.

Good API design isn’t something you wake up with the innate ability to do, and thoroughly describing the business requirements of the uncooperative system in your new API will likely take several iterations. This isn’t a problem however, as it’s recommend that you iterate through a process of this scope rather than attempt some big bang replacement of the uncooperative system to avoid great risk and inconvenience to the business.

Your API should creep in to existence, method by method. Existing applications should be updated to use the new API methods as their developed to allow for an iterative migration to the new wrapping code.

Embrace, extend, extinguish

Once you’ve designed the API you wish to use to replace your uncooperative system, you actually have to produce code that maps between the two.

This is no small task and will probably be the majority of the work required by the project. This code is very implementation specific. You may be able to just codify your new APIs in to a few API calls of the uncooperative system itself, conversely you may have to deconstruct the behaviour of the system and replicate it in entirely fresh code. Either way, it’s exceptionally important that at this stage you keep your implementation separate from your API and service layer.

Thus far we’ve embraced and accepted our uncooperative system, we’ve extended a hand towards it in the form of a neatly designed API, and the final step of your project will revolve around replacing the system entirely.

Once your API is fully featured, and all the dependant systems in your business are referencing it for any communication they need to perform you can begin seeking a replacement system.

Choose wisely, at this point you’ll likely be more than aware of the pains involved in another costly migration! Take your time, there’s no hurry, developing your own solution is even an option.

Once you’ve chosen the replacement, you need to write some code again, this time to translate calls from your new API across to your replacement system. Because you did remember to keep your API decoupled from the implementation, right?

At this point you’ve succeeded in using an iterative approach to software design to fully replace a system that may have previously seemed inseparable from the business. The ROI of a project of this nature is hard to quantify, but I thoroughly believe that if you feel that you need to embark on a project of this nature, its completion will surely strengthen one of the most brittle parts of your operational business.

Congratulations! You made it out alive.

Footnote: Avoid the same thing happening to you

This is honestly a plea to anyone that’s ever going to write a system they expect other developers to communicate with. Practice designing APIs. Write them well. Document them well. Keep them up to date. Support them. Define concise operations. Be good. If everyone is “good” then hopefully fewer people will have to suffer difficult integration exercises.

This entry was posted on Saturday, July 12th, 2008 at 12:03 pm and is filed under Programming, Software patterns, geek. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.