Archive for September, 2009

Designing Client Facing APIs – Best Practices

Wednesday, September 30th, 2009

With the popularity of service oriented architectures and other buzz phrases related to software as service, good API design has become a significant selling point for any software platform in the past 5-10 years.  People make purchasing decisions based on how easy it is to interoperate with your applications and code and as such the number of client / public facing APIs attached to software has skyrocketed.  Iíd like to believe the days of dropping strategic text files in directories to trigger some action or another in an application are behind us.

In this article Iím going to talk about the following things

  • Why you should choose your method names carefully, and what to call
  • Why pretending to be a data access layer is a terrible thing for an API to do
  • Talk about the dangers of leaky abstractions in an API
  • Explain the benefits of creating a data contract between you and the calling code
  • Explain why itís vital to support standards
  • Make sure that your users can retrieve values theyíre going to want to modify
  • Suggest supplying compiled libraries alongside your API documentation
  • Explain why itís important to keep your API implementation clean
  • Talk about the benefits of dogfooding your API
  • Consider supporting atomic operations including rollbacks on failure
  • Discuss bulk operations
  • Try and convince you that both logging and security should be first class citizens
  • Beg you to maintain integration tests and most of all to keep it simple!

Your API Sucks

Youíve probably used an API and thereís a good chance youíve had to write one.  This probably wonít surprise you; most APIs suck.  Theyíre horrible to use and built around illogical leaky abstractions that leave you flicking through huge wads of documentation just to make the most rudimentary feature work.

About 18 months ago, after a year of struggling with a broken third party API that almost brought a business to itís knees by placing significant roadblocks in front of in house development, I was part of a team tasked with designing our own client facing API.  With no desire to expose other developers to the cruel and unusual punishments of software design weíd had to endure, we came to the conclusion that it was really important that we god this piece of the system right.  People say first impressions are everything, and your API design can make or break the faith other developers have in your ability to produce software.  Show somebody a shitty API and theyíll perhaps correctly assume the rest of your code sucks too.

The Best Man For The Job

Thereís a bit of a trend that Iíve noticed with some of the worst APIs Iíve worked with: they seem to be designed by the wrong people.  The wrong people to design an API are 1) the guy that wrote the internal code to do the job the API is providing access to and 2) the consumer of the API. 

The guy that wrote the code that the API is calling under the hood will be inherently slanted to implement an API which exposes this functionality and will have a predisposition to creating a leaky abstraction.  This is especially bad for the consumer APIs designed by the internal implementer tend to assume the consumer knows far more than he really does, or has access to internal data that in reality, he doesnít.

Conversely, an API designed by the consumer of the API will have a tendency towards solving problems that are not the concern of the API itself.  The consumer will, either accidently or by intention, attempt to offload some of the work that should be the responsibility of the calling code into the API.

Ideally, the person thatís writing the API will have knowledge of the system internals, but not be the guy that wrote them. A fellow team member with passive experience to the code would be a good person, or ideally, a pair design exercise between the person that originally wrote the code and an API designer, with the consumer as a consultant.

Speaking The Same Language

Like a lot of software development, you make good progress when you get your terminology right and understand exactly what youíre trying to produce.  Iíve consistently found that the best way to think of a client facing API is as a orchestrating thin wrapper that summarises, in code, a set of business processes that you wish to expose to the public.

In order to get your API design right, you need to clearly define and agree on the boundaries of the system with both your internal team, and your consumers.  Itís important that you have a clear understanding of the following:

  • The responsibility of the calling code
  • The responsibility of the API layer
  • The responsibility of the internal code the API makes calls to

This might sound like a really simple suggestion but Iíve taken part in countless discussions where people on both sides of the API just ďpresumedĒ that either the calling code or the API would perform specific functions (data cleansing, logging etc) when in fact, this confusion had lead to none of the implementers bothering to write the required functionality.  Make sure you know for certain what your API is responsible for doing.

Defining Your API Ė Tips and Tricks

Defining your API methods (or the ďcontractĒ of the API) is the most important thing to get right and there are several vital things to remember.

  • Choose Your Names Wisely Using the language of the business

    Itís vital that your API methods speak in terms that the caller is going to understand.  Your API should be readable.  If your users go hunting for the documentation every time they want to use a method, then youíre probably doing it wrong.

    Clarity in naming is exceptionally important.  The names of your API methods should succinctly state what action that method call is going to perform.  Donít fear using long method names, embrace them for clarity.  As a general rule, your pmethods should probably always be in the form DoSomething(object withThis);

    Ensure that when naming methods you reflect business operations in the method names, not the underlying implementation.

    Bad example:     void InsertToTblCustomer(string[] custDataValues);
    Good example:   void AddCustomer(Customer customer);
    Good example:   void DisableAccount(string accountId); 

  • Donít pretend to be a data access layer

    APIs should summarise business operations in a logical and meaningful fashion.  You are not a public facing data access layer and you should never pretend to be.  If your users want raw database access give them read only permissions on some tables and a copy of SQL Management Studio.  So donít write methods for CRUD operations in your API (unless youíre writing some kind of online file management utility).

    Bad example:     void InsertToTblCustomer(string[] custDataValues);
    Bad example:     void UpdateTblCustomer(string[] custDataValues);
    There are no good examples!

  • Avoid leaky abstractions

    This is a fundamental and simple rule Ė donít expose your callers to anything that theyíre not interested in or wonít understand.  If itís not important, donít show it.  Donít code for things nobody will ever need and donít require your callers to have intimate knowledge of data types or internal categories in your system.

  • Creat
    a data contract between you and the calling code

    Iím going to borrow some of the terminology from WCF here because Iíve found it an appropriate label.  Create a Data Contract library for use in your API.  This library should summarise the business process and the outward facing view of your software.  It might contain terminology that doesnít actually exist in the software itself, but in the business processes that the software models.  Either way, this, and only this, should be the language that the API talks to your callers.

    Where possible, create this data contract in a separate assembly thatís entirely decoupled from your core system and distribute it to people that want to use your API.  This is especially beneficial when using WCF as your clients can generate a service proxy and deal in the same data types that you are in your API code.

    It should be the responsibility of the API layer to marshal the data from your data contract into the domain model of your internal components.

    You data contract should contain every type used to communicate with your API and the object model should be named in a way which is meaningful for the consumers.

    Because your data contract is NOT the object model of your internal components, youíre able to add properties and objects that donít logically exist in your internal components.  This means that you can perform an operation using some internal component, gather the output in your API layer and then compose the output data in a meaningful way using classes written specifically for the data contract.  This way, by the time the user has access to the output data, itís in a format and language which they understand.

  • Support standards!  Donít reinvent the wheel!

    Hereís a true story; while working with an API, my team was faced with the following API method:
    object Run(string request);

    It was the only method on the API, and ďcovered upĒ for around 30 methods all made available through one giant black hole in the side of the system.  Underneath that there was an XML format that the request had to be in in order to call the appropriate method.

    If youíre writing an API, stick to some kind of standards.  Ideally, expose a web service endpoint with an accurate WSDL that people can call or use a simple and obvious REST endpoint.  

    Please, please, please, do not make other developer suffer by rolling your own delivery mechanism.  We have enough of them, donít confuse people by adding some more.  If youíre going to use a raw sockets connection, supply a calling library and stick to some standard middleware like WCF rather than rolling your own. 

    Thousands of people have spent thousands of man years writing code based on existing techniques, youíre not better than all of them combined.

    Reinventing the wheel is never good for anyone.

  • Ensure that the user can retrieve values theyíre going to want to modify

    The number of times Iíve used APIs that let me set or update objects without returning the current state of the object is mind blowing.  So always remember: If youíre going to let them set it, let them get it.  Providing an update or ďupsertĒ method without allowing your consumers to query the current state of data in the system is a complete waste of time.

  • Supply compiled libraries to work with your API documentation

    This may not always be possible, but itís a really good idea to supply a sample implementation and compiled binaries with your API that covers the most common scenarios of usage.  Not only does this prevent the consumer from struggling to get to grips with you API, but it allows you to outline and illustrate a set of best practices for usage.  In an ideal world, the user could just use your sample code in their application, so ensure you license the code appropriately.

    This is an exceptionally good way to deal with any authentication your API may require as you have the ability to provide additional helper classes to perform some common tasks (authenticate Ė> perform action Ė> logout, for example).

  • Keep your implementation clean

    Delegate the API logic to your middleware components / reusable libraries.  Do your best to ensure the API layer doesnít actually contain the logic required to perform operations, just the logic required to marshal the data from the API format into your internal data structures.  The API should simply orchestrate calls to one or more internal methods because your API should simply be exposing existing functionality.

    If the API is exposing some new, API specific functionality, consider splitting this behaviour into a separate assembly or binary to aid testability.

  • Consider dogfooding

    Dogfooding is the act of using the software youíre creating.  I worked on a project where we were developing an order placement system in MVC, and as part of the design process we decided that we wanted to have an API that was a first class citizen.  It then dawned on us that in order to produce the API in a way that accurately mirrored the functionality of the website, we should have the website consume the API like any other client would.  The website had itís own concept of user authentication, and when a user logged in, the web application logged in to the API as the current user.

    Doing this not only ensured that our security model was watertight, but that any additional web functionality would immediately be available to API users because they were actually the same thing.  On top of that, you gain confidence in your own API because you know that itís called often by your own code, reducing the likelihood of users discovering bugs in your API because itís not a product you actually use.

  • Support atomic operations including rollbacks on failure

    When implementing your API methods, ensure that if an exception occurs or an operation doesnít complete, the all the changes made by your API call are reversed.  Consider explicitly supporting transaction scopes in your API to let your consumers compose their own ďsetĒ of operations.

  • Support bulk operations where appropriate

    Building support for bulk operations into your API can often prevent performance issues occurring later when a user tries to, for example, insert 10,000 customers sequentially.  Consider pluralising your methods, so instead of providing an AddCustomer(Customer customer) method, provide only a AddCustomers(List<Customer> customers); method.  Doing this prevents callers from overloading your system by bulking data through your API in unintended ways, allowing you to properly cache required data and cater for these bulk operations.

    This isnít always appropriate, however Iíd always strongly suggest offering pluralised versions of methods that you suspect may be used in bulk, in order to help optimise your API calls and reduce the amount of data being transferred over the wire.

  • Logging as a first class citizen

    Donít wait until somebody asks about API usage to decide to log it.  Build logging into your API wrapper, from the start, at the point of every method call.  It doesnít need to be fancy,
    and you can use a number of freely available components to handle these logs and log rotation (consider using log4net or log4j for simple log rotation).

    Log each method call and some summary or identifiable element of the data passed to it.  Thisíll help you profile API usage, and identify how data changed in your previously closed system.

  • Security as a first class citizen

    Consider the security of your API from the start of the project.  Understand who will have access to your API, which organisations and which individuals.  Do you require roll based security?  Do you need a way to disable API support for specific customers?  Are you transferring data that needs to be encrypted over the wire?

    Beware of over baking your security.  WS-* offers some very robust packet level security features, but if your API doesnít need them, or is restricted to an internal network, then donít bog down your implementation in unneeded security.  Beware of making security choices that tie you down to a specific protocol or technology stack Ė you want to keep your API usable for the consumers.  Do the simplest thing that works.

  • Have integration tests!

    Make sure you have integration tests with mocking at the business logic layer. These tests are for your API wrappers, NOT your logic. The logic should be tested independently, youíre just ensuring your API layer, marshalling and method calls operate correctly at this level.

    With any luck, your business logic should already be tested as part of your existing test suite (which you have right?) but if not, ensure the business logic is tested separate from the API code.

    Consider using a TDD or BDD approach to designing your API calls, designing the specification first in the form of some calling code, then write the code required to make your usage examples compile.  This will help you understand exactly what calls the client will have to make for to your API to achieve specific functionality.  These tests can happily double up as regression tests when you make changes to the API.

  • Keep it simple!  If all else fails, do what the big boys do.

    Always strive to keep your API simple.  Pretend your the consumer at all times.  If youíre unsure of how to proceed, Iíve always found inspiration, both for what to do and what not to do, from reading the API documentation of some large companies that have widely used APIs.  Itís safe to say that the likes of Amazon, Google and Microsoft have had to put some thought into their API designs.  Beware of trusting their decisions blindly, but liberally borrow anything you, as a consumer, would find pleasing in your API.

Iím not going to try and convince you that by following my advice that your API design will be flawless.  Iím really hoping for a little discussion on this topic as it seems like something that is rarely covered and often ďfelt outĒ by the people left to implement APIs for the public.  These are just some lessons Iíve learnt on the way while implementing several public facing APIs.

Want to talk about APIs?  Send me an email!

In defence of current-gen game design

Saturday, September 5th, 2009

I feel frustrated at the moment by the endless cyclical debate on the internet (in this case, a comment on Kotaku) claiming games are creatively plateauxing due to their middle age.  People seem incredibly pent up on the games-as-art debate to the extent that they seem to deride anything that the "it’s just a game" argument could possibly be justified by. 

I want to open this up with a really simple sentiment: I love progressive, narrative driven games, but I sure as hell enjoy playing games that are just a game.  Sometimes I just want to be entertained.

The precise comment on Kotaku (by HarlequinRiot) "The industry needs to rethink what a game can be and use all this amazing technology to make new experiences that may not be so stratified as "this is you, go here, do x, listen to y". Gaming needs it’s Moby Dick or its Brothers Karamazov, as its Citizen Kane’s come every few years." actually struck me as opposing the notion of a "game as a game".  Games have rules and boundaries, a game without a good rule set is just "fucking about", and that’s why these kinds of games are either experimental or don’t exist – they most certainly don’t make money. 

I don’t always need a new experience.  I actually believe that amidst all these calls for innovation and change, that gaming is in the midst of a renaissance of innovation.  The industry is old enough now to learn from the successes of the past, while incrementing in small steps every year.  It’s really a shame to suggest that just because the core gameplay mechanic of two games are the same that they haven’t innovated in subtle ways that will be gathered into the collective unconsciousness of game development.  The simplest examples of this in action are the incremental games – look at any yearly sports title or Halo title or Unreal title, and you’ll notice little innovation year on year, but over 3 or 5 years there’s no way you can argue that you’re playing the same game.  I think people drastically undersell the innovation in game design that happens every day.

On Michael Abbots Brainy Gamer confab (part 2) posted yesterday (which I’d strongly recommend), one of the participants was strongly arguing the point of authorial intent and influence over a game property, implying that regardless of agenda, that a controlling authorial voice on a game imparts part of their world view on a product.  While I agree this is unavoidable in narrative works (especially in the likes of books and cinema), I honestly believe it’s a stretch to state that the guys that made Trials HD or Rocket Riot somehow accidently imprinted their world view on their products.  Even if the product is an inadvertent result of personal ideals, the strict confines of a game system oppose accidental messages in game design.

When the games that gain critical acclaim are generally none of the things that the critics seem to desire, it certainly shines a light on the critics themselves.  You can’t celebrate Super Smash Brothers Brawl or Trials HD, while complaining that a games like GTA4 or Gears of War is thematically simple, and then in the same breath complain that games aren’t progressive or sophisticated enough.  Doing so is pretty much the height of hypocrisy.  If you send these kinds of mixed messages to the people who write games without realising that there’s a natural path of development that must be travelled to get the medium to where you’d love it to be.  Games like GTA4 and Gears, even Bioshock, might not be "all you want them to be" but they are progressive thinking games that try to push both technology and storytelling forwards, even if their respective stories suck.

Likewise, it’s unfair to say that games aren’t sophisticated because they ape cinema, when actually, the most successful and celebrated "intelligent games" do so by emulating cinema and books.  Nobody seems to have a clear vision of what they think "sophistication" in games should really be.  Sophistication is often seen as production values by the mainstream gaming-media when really, the sophistication of the games rules and systems are a "purer" sign of an progressive game. 

I love narrative in games, but you can’t criticise a game for aping the conventions of cinema to achieve a strong narrative without stopping to realise that they only reason they ape that ape these conventions is because they’re effective, you can’t have your narrative without it.  This is especially noticeable when games attempt to forgo book / cinema narrative conventions in exchange for something more game oriented, when this happens you end up with Braid.  I loved Braid and I thought that it was fairly unique from a story perspective but flawed in the telling. The way it told its story played right into game mechanics, and as a result, the story came across as fragmented and confusing to the majority of players because the game mechanics used to present the story donít naturally lend themselves to story telling.  When backed up by the authors refusal give anyone the answer, much of the message was lost. Regardless of authorial intention, if lots of your audience don’t understand the story or narrative you’re trying to convey, you’ve failed as a storyteller and end up being accused of pretention for doing something "different".

A more recent example of "emergent storytelling" would be the well reviewed "The Path".  I enjoyed playing the path, but honestly, it utterly fails as a game.  It’s a terrible game.  Interesting and thought provoking as an experience, but an horrible game.  The controls are awful (practically digital, PS1 era 3d game controls), the interface is counter intuitive and the tasks are deliberately obtuse.  I still enjoyed it and rate it highly, but it failed at being a quality game while succeeding as being a quality experience. 

I think that really sums up a lot of the progressive discussion on videogames – people want something more from their entertainment, but it isn’t games.  Maybe the engineer in me is being pedantic about naming, but I’d much rather "interactive entertainment" for software like "The Path" than "videogame".  Game is a loaded term, and along with it come certain expectations; a set of rules, some gameplay mechanics and a way to progress.  I believe games can be more, I believe games can tell stories, make you feel and make you think, but I don’t think those are required aspects of a good game, games can be "good" outside of those constraints, as an exercise of entertainment through gameplay mechanics.  You can’t criticise a game for being too much like a game, it’s like criticising a book for not being a film.  I really believe that a game can involve art, but if the game "becomes" art?  Well, then you’ve got art that’s art first, game second.  That’s valid expression, but it’s intention is to be art.

On the same podcast there was more talk of the recent debate around Shadow Complex, and the reaction of people towards Orson Scott Cards involvement in regard to his personal politics.  Just a quick note really; the mainstream doesn’t care about your protests.  I really mean that.  You’re telling me that you REALLY think people will boycott Activision because of their (sexist) Sin To Win advertising campaign?  Really?  It’s "just" marketing.  That doesn’t mean I don’t think it was excluding, but what it does mean is that I see sex and inequality used to sell products every single day of the week.  If you stopped buying everything that a person you disagreed with had worked on or had sex used in it’s sales material you would run out of things to buy and games to play pretty quickly.  And that’d be just you, because of the people that kn
about some perceived protest-able injustice so insignificant, not only will half of them not care, but most of the people that do care will do whatever is most convenient to them when a product comes along that they’re interested in.  This goes doubly for marketing, an industry that frequently abuses both it’s position and people to sell product.

While this may seem like a negative response to the confab podcast, it really isn’t.  I enjoyed listening to it, as I have in the past, and have plenty of respect for all of the people involved, some really interesting thought provoking stuff on the direction of gaming comes out of it and I’m really looking forward to the following 3 episodes.  The Brainy Gamer is still one of my favourite gaming resources on the web and certainly occupies the most prominent position in an interesting discussion.

There are plenty of quality, innovation filled games on the market, many of them produced recently.  As for most of the critics?  Part of the problem and not part of the solution.  It’s all well and good to criticise game design inadequacies, but until you’ve really considered the design of compelling, fun, game systems as part of your argument, I’ll write off your "innovation is dead!" arguments as hot air.