swagger-gist.io – Saving and Versioning Swagger definitions

TL;DR – Check out swagger-gist.io to start saving Swagger definitions as Gists.

During my time as an API Product Manager, I found myself constantly looking for ways to make my life easier and more efficient.

One of the problems that I encountered was that it was difficult to share and accurately version my swagger definitions.

I started using SwaggerHub, but I found it to be overkill for the simple Swagger definitions that I was creating.

Eventually, the easiest way for me was to use Github’s gist service frequently to push YAML or JSON files and was asking people to copy these into editor.swagger.io to view and manage.

Given that this relationship was quite nice, I thought I could modify the open source Swagger editor to auto save into Gist for me.  This both allows me to continue to share my Gists, whilst removing the need to manually save them there.

After a bit of tweaking, and a bunch of AWS services, I’ve today pushed live swagger-gist.io.

Get Started:

  1. You’ll need a github account, so sign up at github.com.
  2. Follow the instructions here to generate an access token. Please make sure you grant Gist access when creating the token.
  3. Browse to swagger-gist.io and click ‘File -> Connect to Github’
  4. Enter your github username and the access token that you generated.
  5. Click ‘File -> Open API Definition’ and this will search your Gists for any that have a ‘Description’ with the word ‘Swagger’ in it.
  6. Click ‘Open’ and the app will retrieve the JSON gist and render it as YAML.

Note: i suggest saving this access token in your favourite password safe as you may need it across multiple devices.  At this stage I only save it into the browser’s localstorage object.

Creating a new Swagger Definition

If you don’t have any Swagger as Gist yet, then you’ll need to create a one using the instructions below.

For the moment you have to create new definitions manually, but once it’s created, the auto-saving will take place. I’ll improve this eventually.

  1. Browse to gist.github.com and sign in.
  2. In the gist description, make sure you include the word ‘Swagger’ (with the capital S) and make a note of it.
  3. The filename can be whatever you desire, just make a note of it.
  4. Enter some sample value in the gist (e.g. ABC123) it doesn’t matter what it is.
  5. Click Create Public Gist.
  6. In the URL you will now see something like: https://gist.github.com/johnsmith123/40eea519f479fe27a831d15770255a7f
  7. The string of characters after your username is the Gist-ID.  You’ll need this next.

Creating the new file in Swagger-Gist:

  1. Click ‘File -> New’
  2. You’ll see in the sample file ‘info’ section, there are 4 elements that should be filled out:- title
    – x-gist-id
    – x-gist-filename
    – x-gist-description
  3. The title can be whatever you desire, but the id, filename, and description should match exactly what you created in your sample Gist.
  4. Once you’ve copied the properties, the auto-save will automatically occur after 10s.

That’s it! You’ve now got a swagger definition that is hooked up to Gist.

Next Steps:

  • Hook up proper Github OAuth to generate the token and store them in a more permanent store.
  • Allow generation of new Swagger docs directly from the editor
  • Publish to Heroku (I use Heroku heavily for hosting swagger definitions).
  • Migrate code to the latest fork of swagger editor.  It’s changed significantly since I started this project.

Tech Tools Used:

Front End: Angular JS
Hosting: AWS S3 / CloudFront

Back End: AWS API Gateway -> Lambda -> Gist API.

Shout out to @gojkoadzic who created claudiajs which helped me significantly with writing my AWS API / Lambda code.

If you’re interested in finding out more, ping me or comment and I’ll write another post.

Remember.. APIs are an abstraction

Have you ever noticed how the best APIs on the market seem to provide a simple elegant and clean interface to the data? Developers and API designers take note – this does not happen by accident.

This simple design principle was something that I didn’t really pay much attention to when i was first building out our API, however I’ve become much more aware of it recently, and it is now almost impossible to change;

APIs are an abstraction of the data; not the representation of it

The easiest approach to developing an API is to simply take either the Database representation of a resource (e.g. the row), or it’s associated application file (e.g. Java class) and run some sort of JSON converter over it.

This will give you a lovely output in JSON of the data. Expose this via a URL and job done, you’ve got an API.. right? …Wrong.

This pattern has two very large flaws;

  1. Exposure of unnecessary fields
  2. Tightly coupling your API to your Back End

Exposure of Unnecessary Fields

By simply exposing your internal resource representation as an API, you are potentially asking the API user fields they both don’t need, and don’t care about.

Take the following example, suppose this is a row in a database table:

id name email billingreference
12345 Johnny Smith johnny.smith@myapi.com c_28f898joc90

Now this seems like a completely appropriate database table, so let’s expose this as an API:

GET http://myapi.com/customers/12345
Accept: application/json

{
   "id" : "12345",
   "name" : "Johnny Smith",
   "email" : "johnny.smith@myapi.com",
   "billingreference" : "c_28f898joc90"
}

Looks good. Easy to consume, and matches the database perfectly. The only issue now is that you’ve exposed your billingreference data likely used for an internal system via your publicly accessible API.

This means that the user now has to inspect and make note of this data, (which they shouldn’t really be seeing in the first place) and worse they potentially may store it ‘just incase’ they may need it in the future.

The second issue with this approach is that you’ve also tightly coupled your representation in the Database with your API representation.

Tightly Coupled API with your Back End

Tightly coupling is a major flaw in API development, and I have seen it happen numerous times.  By tightly coupling this information, you remove the ability to change your Back End system without having to also change your API.

APIs should be completely abstracted from the back end system, allowing users to consume the representation that you want them to receive, while being completely independent from where the data is stored.

If one day you wanted to remove the billingreference column above, and put it in another table, you now have to modify your API to continue providing this field, or version the API to deprecate it.

If it was never provided in the first place, you’ve saved yourself a rather large headache.

Provide only the fields that are useful, and nothing else

When designing your API resources, make sure you follow this mantra.  It will ensure that you have simple, clean and useful API representations that your users will love, and it will allow your developers to change their back end systems independent of the API representation.

This method also allows you to use tools like apiary.io and apiblueprint to design your API well before any actual coding begins, which is a major plus when trying to understand what your users actually want.

Feel free to tweet me @jordwalsh with any comments or feedback.

API Management Platforms – do I need one?

One thing every enterprise API creator needs to think about at some point prior to the API going to production is whether an API Management Platform is required.

The main ones that I looked at when we were in the design phase of our API were:

  1. Mashery (http://www.mashery.com/)
  2. Apigee (https://apigee.com/)
  3. 3 Scale (http://www.3scale.net/)
  4. Layer 7 (http://www.layer7tech.com/)

There are lots of others out there, but these four seem be the most active contributors to the API Management community.

What exactly is an API Management Platform?

All of these platforms all fundamentally do the same thing, but each have their nuances and provide different features and functions.

Generally they act as a proxy between your API and the customer, consuming application, or partner.

 

These platforms protect your back end services by limiting the customer to a certain number of allowed queries per second (QPS) or queries per day (QPD).

This prevents a user from inadvertently performing a DoS through poor coding or some form of attack.  If they can only get through a small number of QPS, they’ll bring down the API Management Platform before they’ll bring down your service.

Along with this, they also generally provide:

  • API Key & Authorisation Management
  • Usage Reporting and Analytics
  • Live documentation (Swagger/io-docs)
  • Developer Community Management
  • Billing and Payment Management

Why not just build it myself using mod_qos?

Our network administrator was adamant (and still is) that he could build all of this for me with without the need to pay an external vendor.

Projects like Apache’s mod_qos provide an excellent solution for throughput management, this bundled with an analytics engine (awstats) and you’ve got half the service done right?

Wrong. Do you really feel like designing all of the features and functions specifying exactly what you need.  How long is that going to take with all of the other stuff your network guy (or girl) has to work on? What about ongoing support, maintenance, new features… don’t you have an API programme to be worrying about?

The beauty of these SaaS/PaaS providers is that they’ve done the thinking for you, and they are generally experts in the area, so they know what you want before you do!

They also generally have great Partner and API Growth programmes, so if you can get your API on their marketing ticket, then you’re kickstarting your growth before you’ve even lifted a finger.

We had web demo’s from both Mashery and Apigee, and in the end we ran with Mashery.

Implementing an API Management Platform

The main reasons we chose Mashery was that it seemed very simple and easy to use, their price point was right for us, they had all of the features out of the box that we were looking to implement, and they could stand up an environment for us quickly.

They’ve got some really smart people and they will give you some valuable insights into usage patterns, similar use cases and implementation best practices.

When implementing an API management platform, you’ll also need to think about:

  1. The UI of your developer portal; they can help you with implementation, but the design is up to you
  2. Sandboxing/Trial access for your developers and the associated registration flow
  3. Documentation of your API; are you just going to use swagger/io-docs or are you going to manually write the documentation.
  4. Network level access; whitelisting/blacklisting and firewalls.

Downsides/Pitfalls of using an API Management Platform

  • Change Requests, customisations can be difficult to get approved, and can be costly
  • VSMT isn’t really supported, so stick to generic content types
  • It can be harder to debug issues (timeouts, network access, authentication)
  • Their downtime (if any) may reflect poorly on your API

In the end, it really does come down to what will work in your organisation.  In my opinion, I believe there is value in using an API management platform.

Their whole purpose is to assist your company in building out your APIs, so it’s in their best interests to help you grow.

It will assist you with;

  • Speed to market
  • Accordance with best practices
  • Out of the box reporting, key management and developer portal/CMS
  • Network opportunities with other organisations

However, this may come at an up-front cost, and a bit more effort to get things going.

Happy to answer any questions, or even introduce you to my good friend Jason (from Mashery). Just leave a comment.

Jord

How do users want to see you securing your API?

This has been a really interesting point of discussion with my API users (especially with the potential users).

Users want to make sure the API is secure, however most are actively put off by the steps required to implement the proposed security measures.

There are a number of common security practices that your API can follow, some are simple, some extremely complex, but all are there to achieve one purpose; allow access to only those users that are allowed to see the data.

The main security practices (that I’ve noticed) being;

Security Practice Description Types of APIs
No authorization required Users can access the API without any security at all Open data APIs, general information provided to all e.g. Library Book API
Shared Key Access Users must know some key prior to being able to access the API. This key can be shared, but is generally permitted on a per application basis Secure open data APIs, general information provided to all, but some security required e.g. Train Line API
Per-User Access Users must be provided an API key or Username and Password to access the API. Secure private data APIs, information is normally provided on a per user basis. Service based APIs.

Within the secured models, there are a number of different methods that you can also use to secure the API, from Base64 encoded username and passwords through to HMAC (as described in my post here).

The important thing to remember is; the more secure your API is, the more effort is required from the developer to gain access.

Security Practice Time to get going
None Users will be up and running as soon as they can make their first request (< 1 hour)
Key Required / Basic Authorization Users will be up and running once they are provided access. With a self-provisioning model this can be the same as above, but most service providers will only provide keys after approval, so this generally takes a bit longer. (1 – 2 hrs)
Complex Signature Generation / Certificate Authorization Users will need to understand your signature generation methodology and custom build this into their application. Unless you’ve provided an SDK in their language, this is generally difficult and annoying for the user. (2 – 4 hrs)

In my experience, getting users into your API as fast as possible and showing them how easy it is to use is key to retaining these users.

If there’s another API on the market that offers the same capability and is easier to use, users will naturally gravitate towards that service instead of yours.

That said, most API product managers offering a service based API will find themselves locked in a discussion with the Security Manager from a potential customer about how secure their API is.

This is often a discussion for the Security Manager to re-enforce how important they are, more than the actual security vulnerability on your API, but nevertheless it is a discussion that is needed in most sales cycles.

What you’ll need to get through this discussion is reasoning as to why you chose the security model on your API, and an understanding as to what the potential threats are.  As long as you’ve mitigated the risks, the security manager will give your API a pass.

So what are the possible threats?

There are thousands of possible threats, but the main ones that I’ve worried about in my API design are:

Threat Description Possible Mitigation
User Input When someone is trying to input information into your database that doesn’t belong there (e.g. SQL injection) Input sanitisation and validation
Unauthorised Access When someone who doesn’t have access attempts to gain access to your API. Using one of the security measures described above e.g. Basic Auth, Keys, Signatures or OAuth
Denial of Service (DoS) When someone intentionally floods your API with requests to try and reduce your service capacity, or bring it down altogether. Rate Limiting with an API Provider, or building this into Apache with mod_qos
Man in the Middle attack (MitM) When someone is monitoring the requests made on a connection (e.g. wifi) and can easily re-run those requests, or modify them slightly, to perform a similar outcome. Signatures, timestamps or other request level uniqueness will prevent this type of attack.

And the list goes on..

The take away from this post should be that it doesn’t really matter how you choose to secure your API.  As long as you think it is secure enough for the data that you are providing, and you can stand up to scrutiny (e.g. you know what the threats are, and you’re confident you’ve mitigated them, or understood the risks) then it is likely it’s secure enough.

API Versioning – So many opinions!

Firstly, I don’t want this post to be another rant about what you should do or shouldn’t do when it comes to versioning your API.

What I’m discussing here is what I did with our API, why I did it, and (3 years down the track) what I am planning on changing due to current usage patterns.

What was available on the market?

When I was doing my research on how we could version our API, there were 3 main options, each seeming to have their own pros and cons:

1. Version using the URL

HTTP 1.1 GET /v1/messages

2. Version using a URL query parameter

HTTP 1.1 GET /messages?v=1

3. Version using headers (with vendor specific mime types)

HTTP 1.1 GET /messages
Accept: application/vnd.company.resource-v1+json

 and the winner was..

After much debate (mostly internal) I decided to choose door number 3, versioning with VSMT (vendor specific mime types).

For those interested, I go on about it in detail here: http://bit.ly/TDNzLN

The main reason I chose VSMT and versioning was so that we could change the versions of each resource representation independent of the entire API, and so that versions were not tied to the URL and therefore not polluting the code.

e.g. let’s say we have a contact object with details of a person:

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/vnd.company.contact-v1+json

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.contact-v1+json
{
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"
}

In my mind, if I wanted to update the representation of this contact, I could simply do so and change the version.

e.g. Let’s add an email and website to our contact.

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/vnd.company.contact-v2+json

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.contact-v2+json
{
   "id" : "1",
   "name" : "Neil Armstrong",
   "email" : "neil@nasa.com",
   "website" : "http://www.nasa.com",
   "url" : "/contacts/1"
}

What’s the problem with this approach?

Well, what I’ve found is two things:

1. Users don’t like adding extra headers to their requests

In any language, it’s relatively easy to add HTTP headers. The problem is that users will often either not get how to do this, or not see the benefit in it.

I’ve had more complaints from users about adding custom headers than any other part of our API.

Also, if your argument that using versioning in the URL pollutes your code with versions so you should put it in the headers, you’re just moving your problem!

=== REQUEST ===>
HTTP 1.1 GET /v1/contacts?firstName=Neil

<== RESPONSE ===
HTTP/1.1 200 OK
{
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"
}

Is the same as…

=== REQUEST ===>
HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/vnd.company.contact-v1+json

<== RESPONSE ===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.contact-v1+json
{
   "id" : "1",
   "name" : "Neil Armstrong",
   "url" : "/contacts/1"
}

You’ve just shifted your coding of the versioning from line 1 to line 2, it’s still in the code though.

2. Developers don’t like maintaining lots of versions of the schema

It may be different in your development team, by the devs that I work with are quite happy to continue on version 1 of the schema ensuring that backwards compatibility is always there.

In the example above, as long as we didn’t remove the “Name” field, v1 can be used forever more.

Most users will parse your API responses using json_decode style functions anyway.

$json = json_decode($response);

$name = $json->{'name'}; //Neil Armstrong
$url = $json->{'url'}; // /contacts/1

//do something useful

If you now add the email and website elements to this version, the existing code will have no issues at all, so adding a new schema version is simply a headache for your developers and your users.

What I’m planning on changing…

Based on the users feedback, and my annoyances every time I’ve needed to build something with the our API, I’ll be changing the following:

1. Support default media types as well as VSMT

As a lot of users don’t understand how to use VSMT, I’m going to implement support for default mime types as well as VSMT.

I got this idea off GitHub (https://developer.github.com/v3/media/) so thanks guys!

e.g. Instead of only supporting:

HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/vnd.company.contact-v1+json

I’m going to also allow:

HTTP 1.1 GET /contacts?firstName=Neil
Accept: application/json

This will simply return the latest version of the resource requested. Users will need to be aware that this can change in the future at any time, but we’ll support backwards compatibility as much as possible and send notifications when we are making major changes.

2. Allow content type requests on the URL

Again, as a lot of users are complaining about all these extra headers, I want to make things simple and allow the extension of the URL to support the mime type.

e.g. Instead of only supporting:

HTTP 1.1 GET /contacts
Accept: application/vnd.company.contact-v1+json

I’m going to also allow:

HTTP 1.1 GET /contacts+json

This change is based on the premise above where it will simply return the default version of the resource in the format requested.

Keep the users happy, and they’ll keep using your API

The whole premise of this comes back to asking your users what they want. They are the ones who are using the service, and if they don’t like it, they’ll go somewhere else.

Keep them happy, and they’ll keep using your API instead of someone else’s.

Authorising your API – Keep it simple

Where I work, where we supply a software platform that is providing a service to users. Like thousands of others of these services, people can log into our web interface using their username and password and can perform some actions within the tool (wordpress is a good example of such a system).

When this type of service is transitioned into an API, we generally are no longer dealing with username and passwords, we are now dealing with API keys, signatures, HMAC, and a whole raft of other jargon to do essentially the same thing.

When I was thinking about how I was going to set up Authorisation to our API, I thought about the types of API users who would be using the service.  I could identify two kinds of people building applications that use my API:

1. The Service Provider
This is the guy that is building a new application, and wants to utilise the functionality provided by the API.  He is going to have is own users database, and he’ll authenticate to the API using his own credentials.

2. The Service Extender
This is the guy that is building an application that extends the functionality of the existing service using the API.  He may or may not have his own users database, but the likelihood is that he’ll allow his users to authenticate to the API using their own credentials within the service (e.g. OAuth).


As I follow the YAGNI principle as much as possible, the only users that I had in the first instance fit into the first category.  Based on this, there wasn’t much point in supporting an OAuth style framework until we had some users that were jumping up and down for it.

Keep the authorisation simple and your users will thank you.

HTTP already has support built-in for authorising users, so I deemed the easiest way was to use the out of the box Authorization header.

Take the existing username and password to access the service that users already know, turn it into Base64 and use Basic HTTP to access the API.

So a username and password of john.smith / Password123 becomes:

HTTPS 1.1 GET /messages
Authorization: Basic am9obi5zbWl0aDpQYXNzd29yZDEyMw==

Every programming language comes with a Base64 encoder built-in, so it’s simple for the users to set up.

There are no timestamps, signatures, HMAC or any of the other cumbersome approaches.  However, there is one caveat; you must use HTTPS.

As long as you force HTTPS, this is method secure.

Without HTTPS however, this mechanism is clearly open to packet sniffing and man-in-the-middle attacks.

I have no doubt the day will come where we need to build this out to support services like OAuth or perhaps generation of signatures using timestamps and shared secrets, but in 3 years of running this service, we haven’t had any complaints from users about this methodology.

Perhaps when you are thinking of authorising your API, check with your users about this methodology before jumping straight to OAuth or similar.

Designing the API – Identifying the resource types

Before I jump directly into my version of “Donny Do” and “Donny Don’t” for APIs, I’ll give a bit of background as to the steps I took to develop our API.

(Note: If you get REST and how to structure a REST API, skip this post)…

When I was asked to design our API my first step was to watch Teach a Dog to REST. Then watch it again, and again. I made other members of the team watch it, and took as much as I could from it.  This video is probably the most elegant demonstration I’ve seen that articulates “proper” REST API design.

From here I took my instructions from the video and broke down the application into its different types of resources.

As the application I was dealing with was to do with SMS messaging, the resources became things like messages, message templates, recipients, and recipient groups.

I didn’t start with the use cases, e.g. “how do I send a message” because identifying the resources and the resource types within the API is far more important at this stage than worrying about what you can do with the API.

The reason is that if you focus on the use cases (the verbs) you’ll end up with a list of actions. This may sound OK, but ultimately this path will lead to a bunch of action based URLs e.g. /sendMessage, /createRecipient or (worse) /addRecipientToDistributionList.

Focus on the resources (the nouns) instead.

What you actually want at this stage is simply the list of resource group URLs:

  • /messages
  • /templates
  • /recipients
  • /recipientgroups

From here I simply thought of each of the main 4 HTTP verbs acting on my URLs; POST, GET, PUT and DELETE (Create, Read, Update and Delete respectively).

Resource GET POST PUT DELETE
messages Retrieve a listing of messages. Create and send a new message Not Implemented Not Implemented (could refer to deleting all messages)
message templates Retrieve a listing of message templates Create a new message template Not Implemented Not Implemented (could refer to deleting all templates)
recipients Retrieve a listing of recipients. Create a new recipient Not Implemented Not Implemented (could refer to deleting all recipients)
recipient groups Retrieve a listing of recipient groups. Create a new recipient group Not Implemented Not Implemented (could refer to deleting all groups)

Note: we still haven’t gotten into any schemas or technical implementation details.  During this phase I am trying to define a simple API structure; that’s it.

After completing the table for the resource groupings, I then moved on to each individual resource.

Resource GET POST PUT DELETE
messages/[ID] Retrieve an individual message Not Implemented Update an individual message (possibly relevant for drafting) Delete an individual message
messagetemplates/[ID] Retrieve a message template Not Implemented Update an individual message template Delete an individual message template
recipients/[ID] Retrieve a recipient record Not Implemented Update an individual recipient record Delete an individual recipient record
recipientgroups/[ID] Retrieve a recipient group Not Implemented Update a recipient group’s details, or add a recipient to the group. Delete a recipient group

Structuring my table like this forced me to think about what each action is going to perform as I only had 4 actions to work with (there are a few others but we’ll get to them later).

For example, I made the call that when you performed a POST to /messages this would not only Create the Message, it would also Send it.

Some would argue that the resource should have been created in one step, and sent in another e.g. create it with a POST to /messages, then update it with a PUT to move it to a ‘sent’ state.

This was an option I considered, but I decided that more of my users would be sending messages in one single step than there were those that would require a draft/send process, so I went with the two in one approach.

When you are designing your API, I would encourage you to start here also.  It will take some time to map out what each resource is, and what the 4 verbs mean for each, but it will be worthwhile in the long run.