Remember.. APIs are an abstraction

Have you ever noticed how the best APIs on the market seem to provide a simple elegant and clean interface to the data? Developers and API designers take note – this does not happen by accident.

This simple design principle was something that I didn’t really pay much attention to when i was first building out our API, however I’ve become much more aware of it recently, and it is now almost impossible to change;

APIs are an abstraction of the data; not the representation of it

The easiest approach to developing an API is to simply take either the Database representation of a resource (e.g. the row), or it’s associated application file (e.g. Java class) and run some sort of JSON converter over it.

This will give you a lovely output in JSON of the data. Expose this via a URL and job done, you’ve got an API.. right? …Wrong.

This pattern has two very large flaws;

  1. Exposure of unnecessary fields
  2. Tightly coupling your API to your Back End

Exposure of Unnecessary Fields

By simply exposing your internal resource representation as an API, you are potentially asking the API user fields they both don’t need, and don’t care about.

Take the following example, suppose this is a row in a database table:

id name email billingreference
12345 Johnny Smith c_28f898joc90

Now this seems like a completely appropriate database table, so let’s expose this as an API:

Accept: application/json

   "id" : "12345",
   "name" : "Johnny Smith",
   "email" : "",
   "billingreference" : "c_28f898joc90"

Looks good. Easy to consume, and matches the database perfectly. The only issue now is that you’ve exposed your billingreference data likely used for an internal system via your publicly accessible API.

This means that the user now has to inspect and make note of this data, (which they shouldn’t really be seeing in the first place) and worse they potentially may store it ‘just incase’ they may need it in the future.

The second issue with this approach is that you’ve also tightly coupled your representation in the Database with your API representation.

Tightly Coupled API with your Back End

Tightly coupling is a major flaw in API development, and I have seen it happen numerous times.  By tightly coupling this information, you remove the ability to change your Back End system without having to also change your API.

APIs should be completely abstracted from the back end system, allowing users to consume the representation that you want them to receive, while being completely independent from where the data is stored.

If one day you wanted to remove the billingreference column above, and put it in another table, you now have to modify your API to continue providing this field, or version the API to deprecate it.

If it was never provided in the first place, you’ve saved yourself a rather large headache.

Provide only the fields that are useful, and nothing else

When designing your API resources, make sure you follow this mantra.  It will ensure that you have simple, clean and useful API representations that your users will love, and it will allow your developers to change their back end systems independent of the API representation.

This method also allows you to use tools like and apiblueprint to design your API well before any actual coding begins, which is a major plus when trying to understand what your users actually want.

Feel free to tweet me @jordwalsh with any comments or feedback.

How do users want to see you securing your API?

This has been a really interesting point of discussion with my API users (especially with the potential users).

Users want to make sure the API is secure, however most are actively put off by the steps required to implement the proposed security measures.

There are a number of common security practices that your API can follow, some are simple, some extremely complex, but all are there to achieve one purpose; allow access to only those users that are allowed to see the data.

The main security practices (that I’ve noticed) being;

Security Practice Description Types of APIs
No authorization required Users can access the API without any security at all Open data APIs, general information provided to all e.g. Library Book API
Shared Key Access Users must know some key prior to being able to access the API. This key can be shared, but is generally permitted on a per application basis Secure open data APIs, general information provided to all, but some security required e.g. Train Line API
Per-User Access Users must be provided an API key or Username and Password to access the API. Secure private data APIs, information is normally provided on a per user basis. Service based APIs.

Within the secured models, there are a number of different methods that you can also use to secure the API, from Base64 encoded username and passwords through to HMAC (as described in my post here).

The important thing to remember is; the more secure your API is, the more effort is required from the developer to gain access.

Security Practice Time to get going
None Users will be up and running as soon as they can make their first request (< 1 hour)
Key Required / Basic Authorization Users will be up and running once they are provided access. With a self-provisioning model this can be the same as above, but most service providers will only provide keys after approval, so this generally takes a bit longer. (1 – 2 hrs)
Complex Signature Generation / Certificate Authorization Users will need to understand your signature generation methodology and custom build this into their application. Unless you’ve provided an SDK in their language, this is generally difficult and annoying for the user. (2 – 4 hrs)

In my experience, getting users into your API as fast as possible and showing them how easy it is to use is key to retaining these users.

If there’s another API on the market that offers the same capability and is easier to use, users will naturally gravitate towards that service instead of yours.

That said, most API product managers offering a service based API will find themselves locked in a discussion with the Security Manager from a potential customer about how secure their API is.

This is often a discussion for the Security Manager to re-enforce how important they are, more than the actual security vulnerability on your API, but nevertheless it is a discussion that is needed in most sales cycles.

What you’ll need to get through this discussion is reasoning as to why you chose the security model on your API, and an understanding as to what the potential threats are.  As long as you’ve mitigated the risks, the security manager will give your API a pass.

So what are the possible threats?

There are thousands of possible threats, but the main ones that I’ve worried about in my API design are:

Threat Description Possible Mitigation
User Input When someone is trying to input information into your database that doesn’t belong there (e.g. SQL injection) Input sanitisation and validation
Unauthorised Access When someone who doesn’t have access attempts to gain access to your API. Using one of the security measures described above e.g. Basic Auth, Keys, Signatures or OAuth
Denial of Service (DoS) When someone intentionally floods your API with requests to try and reduce your service capacity, or bring it down altogether. Rate Limiting with an API Provider, or building this into Apache with mod_qos
Man in the Middle attack (MitM) When someone is monitoring the requests made on a connection (e.g. wifi) and can easily re-run those requests, or modify them slightly, to perform a similar outcome. Signatures, timestamps or other request level uniqueness will prevent this type of attack.

And the list goes on..

The take away from this post should be that it doesn’t really matter how you choose to secure your API.  As long as you think it is secure enough for the data that you are providing, and you can stand up to scrutiny (e.g. you know what the threats are, and you’re confident you’ve mitigated them, or understood the risks) then it is likely it’s secure enough.

API Versioning – So many opinions!

Firstly, I don’t want this post to be another rant about what you should do or shouldn’t do when it comes to versioning your API.

What I’m discussing here is what I did with our API, why I did it, and (3 years down the track) what I am planning on changing due to current usage patterns.

What was available on the market?

When I was doing my research on how we could version our API, there were 3 main options, each seeming to have their own pros and cons:

1. Version using the URL

HTTP 1.1 GET /v1/messages

2. Version using a URL query parameter

HTTP 1.1 GET /messages?v=1

3. Version using headers (with vendor specific mime types)

HTTP 1.1 GET /messages
Accept: application/

 and the winner was..

After much debate (mostly internal) I decided to choose door number 3, versioning with VSMT (vendor specific mime types).

For those interested, I go on about it in detail here:

The main reason I chose VSMT and versioning was so that we could change the versions of each resource representation independent of the entire API, and so that versions were not tied to the URL and therefore not polluting the code.

e.g. let’s say we have a contact object with details of a person:

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"

In my mind, if I wanted to update the representation of this contact, I could simply do so and change the version.

e.g. Let’s add an email and website to our contact.

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/
   "id" : "1",
   "name" : "Neil Armstrong",
   "email" : "",
   "website" : "",
   "url" : "/contacts/1"

What’s the problem with this approach?

Well, what I’ve found is two things:

1. Users don’t like adding extra headers to their requests

In any language, it’s relatively easy to add HTTP headers. The problem is that users will often either not get how to do this, or not see the benefit in it.

I’ve had more complaints from users about adding custom headers than any other part of our API.

Also, if your argument that using versioning in the URL pollutes your code with versions so you should put it in the headers, you’re just moving your problem!

=== REQUEST ===>
HTTP 1.1 GET /v1/contacts?firstName=Neil

<== RESPONSE ===
HTTP/1.1 200 OK
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"

Is the same as…

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"

You’ve just shifted your coding of the versioning from line 1 to line 2, it’s still in the code though.

2. Developers don’t like maintaining lots of versions of the schema

It may be different in your development team, by the devs that I work with are quite happy to continue on version 1 of the schema ensuring that backwards compatibility is always there.

In the example above, as long as we didn’t remove the “Name” field, v1 can be used forever more.

Most users will parse your API responses using json_decode style functions anyway.

$json = json_decode($response);

$name = $json->{'name'}; //Neil Armstrong
$url = $json->{'url'}; // /contacts/1

//do something useful

If you now add the email and website elements to this version, the existing code will have no issues at all, so adding a new schema version is simply a headache for your developers and your users.

What I’m planning on changing…

Based on the users feedback, and my annoyances every time I’ve needed to build something with the our API, I’ll be changing the following:

1. Support default media types as well as VSMT

As a lot of users don’t understand how to use VSMT, I’m going to implement support for default mime types as well as VSMT.

I got this idea off GitHub ( so thanks guys!

e.g. Instead of only supporting:

HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/

I’m going to also allow:

HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/json

This will simply return the latest version of the resource requested. Users will need to be aware that this can change in the future at any time, but we’ll support backwards compatibility as much as possible and send notifications when we are making major changes.

2. Allow content type requests on the URL

Again, as a lot of users are complaining about all these extra headers, I want to make things simple and allow the extension of the URL to support the mime type.

e.g. Instead of only supporting:

HTTP 1.1 GET /contacts
Accept: application/

I’m going to also allow:

HTTP 1.1 GET /contacts+json

This change is based on the premise above where it will simply return the default version of the resource in the format requested.

Keep the users happy, and they’ll keep using your API

The whole premise of this comes back to asking your users what they want. They are the ones who are using the service, and if they don’t like it, they’ll go somewhere else.

Keep them happy, and they’ll keep using your API instead of someone else’s.

Authorising your API – Keep it simple

Where I work, where we supply a software platform that is providing a service to users. Like thousands of others of these services, people can log into our web interface using their username and password and can perform some actions within the tool (wordpress is a good example of such a system).

When this type of service is transitioned into an API, we generally are no longer dealing with username and passwords, we are now dealing with API keys, signatures, HMAC, and a whole raft of other jargon to do essentially the same thing.

When I was thinking about how I was going to set up Authorisation to our API, I thought about the types of API users who would be using the service.  I could identify two kinds of people building applications that use my API:

1. The Service Provider
This is the guy that is building a new application, and wants to utilise the functionality provided by the API.  He is going to have is own users database, and he’ll authenticate to the API using his own credentials.

2. The Service Extender
This is the guy that is building an application that extends the functionality of the existing service using the API.  He may or may not have his own users database, but the likelihood is that he’ll allow his users to authenticate to the API using their own credentials within the service (e.g. OAuth).

As I follow the YAGNI principle as much as possible, the only users that I had in the first instance fit into the first category.  Based on this, there wasn’t much point in supporting an OAuth style framework until we had some users that were jumping up and down for it.

Keep the authorisation simple and your users will thank you.

HTTP already has support built-in for authorising users, so I deemed the easiest way was to use the out of the box Authorization header.

Take the existing username and password to access the service that users already know, turn it into Base64 and use Basic HTTP to access the API.

So a username and password of john.smith / Password123 becomes:

HTTPS 1.1 GET /messages
Authorization: Basic am9obi5zbWl0aDpQYXNzd29yZDEyMw==

Every programming language comes with a Base64 encoder built-in, so it’s simple for the users to set up.

There are no timestamps, signatures, HMAC or any of the other cumbersome approaches.  However, there is one caveat; you must use HTTPS.

As long as you force HTTPS, this is method secure.

Without HTTPS however, this mechanism is clearly open to packet sniffing and man-in-the-middle attacks.

I have no doubt the day will come where we need to build this out to support services like OAuth or perhaps generation of signatures using timestamps and shared secrets, but in 3 years of running this service, we haven’t had any complaints from users about this methodology.

Perhaps when you are thinking of authorising your API, check with your users about this methodology before jumping straight to OAuth or similar.