REST Web services demystified

These days, REST APIs are available to integrate your application or website with thousands of platforms -- Web 2.0 properties such as Twitter, Facebook, and Foursquare, as well as old-school companies like USA Today, Best Buy, and Edmunds.com. Unfortunately, tapping this dizzying buffet of options isn't always easy.

Most API documentation assumes an understanding of the foundational technology, and while REST API architecture gives developers enormous freedom to innovate, that freedom comes at a price -- namely, fewer guidelines for application behavior. But because these APIs are based on HTTP, a mature protocol, there are numerous strategies you can use to develop and debug your application. First, though, you need a comprehension of the basics of HTTP, a grasp of REST architecture, and some tools for inspecting traffic and strategies for debugging problems.

[ Experts explain all key HTML5 specs, from canvas to local storage, plus the latest browser benchmarks and report cards on HTML5 support in InfoWorld's megaguide to HTML5. Download the PDF today! | Keep up with the latest developer news with InfoWorld's Developer World newsletter. ]

HTTPHyperText Transfer Protocol (HTTP) -- the protocol used throughout the Internet for transactions between Web servers and clients -- is simple, tested, and supported by libraries in every modern programming language. HTTP includes methods supporting all of the necessary actions for a server/client transaction: PUT, GET, POST, and DELETE. It has a rich set of status codes to cover success and error situations, and headers and query parameters make it possible to send additional information about the transaction or request to the server.

To better illustrate how HTTP works, let's take a look at a simple transaction. This is a basic request from a Chrome browser on a Macintosh to Google's home page.

Looking over the request, you can see that the method sent (1) was GET -- a read operation on an HTTP resource. The URL (2) indicates the full path being retrieved -- in this case, the document at the root of http://www.google.com. The request gets a little more interesting with the headers (3). An HTTP transaction can have any number of headers, and these allow the client to send information about the transaction to the server.

In addition to the types of headers seen here, headers can also be used to send user-specific information such as cookies. Servers aren't under any obligation to use information passed along via headers, but they frequently use the information to craft the response appropriately -- HTML that will display nicely on a specific browser, or content in the language the user prefers.

The response from Google includes information that is useful to the client -- and that can be useful to you too. The first and most important piece is (4) the HTTP Status code. This code is returned with every response from an HTTP server. Understanding these codes makes it a lot easier to figure out what's happening between the client and the server.

• 50x: Something went wrong with the server.
• 40x: The developer did something unexpected.
• 30x: These codes are used for redirects.
• 20x: Cool, your request succeeded.

Watching HTTP trafficWith a basic understanding of how these transactions work, it's time to take it to the next level and watch some actual traffic. Fortunately, because HTTP is so widely used, it's easy to find traffic to inspect even before you start accessing the APIs. Many modern Web browsers provide a Web inspector function that allows you to see the browser's traffic. However, since the focus of this article is on software development using REST APIs, you'll need to use an application that lets you watch all of the HTTP traffic flowing to and from your system -- not just your browsing activity.

HTTP sniffersHTTP sniffers are applications that provide a console to view the Web traffic on your system. All of them report on the transaction information detailed above: the URL, headers, method, and body of the request; and the status, headers, and body of the response. On the Macintosh, HTTP Scoop works quite well. On Windows, I suggest Fiddler, and for Linux/Unix systems there's Wireshark.

As an example, here are a few screenshots from HTTP Scoop. The first one shows the summary view with basic information about the request.

(Click to enlarge.)

Double-clicking on one of the log lines gives a more detailed view of that particular transaction.

(Click to enlarge.)

In the case of HTTP Scoop, you're provided with different tabs to explore specific aspects of the transaction. The Headers tab is frequently useful in understanding the request and response.

(Click to enlarge.)

Being able to inspect the headers for the transaction gives you a lot more information about how the client and server are communicating. Another particularly useful tab in this tool is Request/Response, which shows the GET parameters and POST body as well as the body of the server response.

You can almost always figure out what's going wrong with a request using a combination of the Headers and Request/Response tabs (or the equivalent in one of the other sniffers). If you can create a successful request, and compare the values to your failing request, it's generally fairly easy to figure out what you need to change in order to fix the issue.

Install one of these sniffers on your system and experiment with it a bit. Visit a Web page and make sure you can find all of the important components of the requests and responses listed above.

RESTThis section provides a very high level (and somewhat abstract) overview of REST API architecture. If you want to dig a little deeper into REST, there's a fantastic tutorial at rest.elkstein.org. REST, for "Representational State Transfer," is a lightweight architecture for network applications that use HTTP requests to read and write data. Because REST services are built on this well-supported standard, these APIs are platform independent, and standard libraries for facilitating these simple Web requests are available in all modern programming languages. Because REST exposes data rather than methods, it gives developers the freedom to create new and innovative uses of the architecture. However, because it is so open-ended, some amount of caution is needed in the design. Most important, a REST API should be designed so that a GET request can never change the state of any resource in the system. Good REST API design protects against accidental deletion or change by third-party applications or overzealous Web crawlers.

REST APIs generally format the responses in JSON or XML. Both formats are well supported, with libraries available in most any modern programming language. Using one of these libraries, you'll find it relatively straightforward to convert the response into an object that you can manipulate in your application.

A REST exampleTwitter has a great API to use for testing. Because some of Twitter's resources don't require authentication, you can see them right in the browser. As an example, click on the following link: http://search.twitter.com/search.json?q=synedra&count=1

The response you see in your browser is the JSON representation of the search for "synedra" in the Twitter Search API. Using the JSON library for your programming language of choice, you can convert this response into a usable object. For instance, in Python you could do the following:

As you can see, the information contained in the Twitter response has been loaded up as an object (jsonobj).

TroubleshootingOne of the best troubleshooting strategies is avoiding problems in the first place. When developing against REST APIs, you usually want to go with supported, tested libraries and follow sample code wherever possible. As in any other situation where your application relies on a service you don't control, you need to code defensively. The structure of the response could change, or the call could simply fail due to network or server issues. Your code needs to handle these unexpected issues gracefully. You can save yourself a lot of trouble by logging the responses you get locally. Most servers don't log the responses they send, so you may need that local log to figure out what happened when something goes wrong. And asking for support will be much easier if you can send along an example of the failing response.

Successful requests. Sometimes when you encounter failures, it can feel like the server has a vendetta against you. But take heart, the server simply isn't smart enough to recognize your request and behave differently. Most likely your request has some minor issue that prevents the server from handling it correctly. Your very best strategy for figuring out why a request isn't working is to use the tools provided by the API team itself -- consoles, sample code, test tools -- to see what a successful request looks like. For instance, Mashery provides IO/Docs, an interactive explorer, for USA Today and the other APIs it hosts. Similarly, Apigee has a console for Twitter and many other APIs. Both tools give you the ability to see successful calls without writing any code. Check out the headers, URL, and body of those requests; if you can make your request match it's going to succeed.

4XX errors. If the response you receive is a 4xx error, your request isn't what the server is expecting to see. Here's a quick rundown of what these specific codes mean.

• 400: Formatting error. Your headers, parameters, or POST body are not formatted correctly. You may be missing a required parameter or you have added an unexpected one. The response body should have information that will help you spot the misstep.
• 401: Authentication error. The server doesn't recognize something about the authentication you're using. Your signature may not match, or the token may be invalid. Again, check the response body for helpful clues. If you're lucky, the API team has made tools available that will allow you to check your authentication and work through the issue.
• 403: Authorization error. The server knows who you (and your user) are, but you aren't allowed to see the resource. Sometimes this is a permission issue (restricted to certain users). But if you start seeing 403 errors after successful requests of the same type, then most likely you're being throttled (so you're no longer allowed to access that resource). Review the throttle limits for the API, and remember that throttles are frequently different for an application as a whole as opposed to specific users of an application.
• 404: Not found. The resource you requested isn't a valid path. We all know what this means.

Asking good questions. Most APIs have documentation, forums, and other resources available to help you achieve success. However, if you're not able to solve the problem on your own, you'll want to ask for help on a forum -- either one devoted to the specific API or a good general-purpose developer's venue like StackOverflow. Technical questions asked without enough context can kick off a frustrating cycle of exploratory questions, guesses, and clarifications, delaying the resolution of your issue.

In general, you want to follow this kind of template:

• I did X.
• I expected Y to happen.
• To my dismay, Z happened instead.

All three pieces are critically important in a good question or bug report.

I did X. What exactly were you doing when the problem occurred? Can you consistently reproduce the problem? What else did you try? Be sure to give full context, including the language or libraries (or browser) you're using -- all the details a person starting from scratch might need to reproduce your problem. In the case of a REST API call, you'll need to include the URL, the headers, and the body of the request.

I expected Y to happen. Remember, it often happens that things are working just as intended, but your expectation doesn't match what the system is doing. This might mean the documentation needs improving or (just maybe) you haven't read the documentation closely enough. If you describe what you were expecting to happen, the API team or support community will better understand how to get you what you need. If you can point to the documentation you're following, so much the better.

To my dismay, Z happened instead. Did you get an error? Do you have the complete response? Provide all the headers, status code, and response body.

A good question provides support staff or the forum reader with enough context to answer the inquiry correctly the first time. Sure, it takes a little longer to craft questions with all the relevant detail, but you'll get a much quicker response and a faster resolution. Your goal is to give the reader everything they need to answer the question correctly the first time. It will save you -- as well as the support person -- both time and frustration.

RESTful resourcesThis overview should arm you with enough information to get started using REST APIs. Again, I strongly encourage you to check out the in-depth tutorial at rest.elkstein.org. Or if you just want to dive right in, there's excellent sample code available for many REST APIs. An easy way to get started is to work through the tutorials or get the example code functioning in your environment, then move on from there.

Once you're comfortable with REST APIs, they are a great way to add integration between your application and existing Web services. REST APIs not only share a consistent interface, but they invariably rely on HTTP; after you've used one, you're in great shape to try out others. REST's consistency and openness make it one of the easiest API types to work with. With the ability to inspect the traffic between the client and the server, developing and debugging REST integrations is made that much easier.

This article, "How-to: REST Web services demystified," originally appeared at InfoWorld.com. Follow the latest news in programming at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

Read more about application development in InfoWorld's Application Development Channel.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email Computerworld
• Follow Computerworld on twitter

• Share
  Share this article1

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark