Testing your REST client in Python

When you start to write a client for a REST API in Python at beginning it's easy to test it using a Python interactive session, but at some point you'll have to write tests, at that moment you'll see that it's not easy to test your code against live data from the RESTful web API. You may encounter various problems, for example, you can have network problems when tests run, the web server may be temporarily down or tests become slow due to network latency.

A solution to this problem is to use Mock objects, they simulate the behavior of your real objects in a controlled way, so in this case a mock object may simulate the behavior of the urlopen function (from the urllib2 module) and return something like an HTTP response (a file-like object) without hit the real REST API. The file-like object returned may map a RESTful resource to a file which contains a pre-saved response from the real web server.

To show the idea I wrote a simple REST client for the Github API. Here's what the directory structure looks like

$ tree project
project
├── client.py
└── tests/
    ├── test_client.py
    └── resources/
        └── users/
            └── test_user

3 directories, 3 files

I use nose as test runner and Foord's Mock library to create mock objects. You can install them into a virtualenv by typing

$ pip install nose mock

Here's the content of the client.py file

import json
from urllib2 import urlopen


class ClientAPI(object):
    def request(self, user):
        url = "https://api.github.com/users/%s" % user
        response = urlopen(url)

        raw_data = response.read().decode('utf-8')
        return json.loads(raw_data)

As you can see, it calls urlopen, parse the JSON data from the HTTP response and return a Python dictionary.

And here's the content of the tests/test_client.py file

import os.path
import unittest
from urlparse import urlparse

from client import ClientAPI
from mock import patch


def fake_urlopen(url):
    """
    A stub urlopen() implementation that load json responses from
    the filesystem.
    """
    # Map path from url to a file
    parsed_url = urlparse(url)
    resource_file = os.path.normpath('tests/resources%s' % parsed_url.path)
    # Must return a file-like object
    return open(resource_file, mode='rb')


class ClientTestCase(unittest.TestCase):
    """Test case for the client methods."""

    def setUp(self):
        self.patcher = patch('client.urlopen', fake_urlopen)
        self.patcher.start()
        self.client = ClientAPI()

    def tearDown(self):
        self.patcher.stop()

    def test_request(self):
        """Test a simple request."""
        user = 'test_user'
        response = self.client.request(user)
        self.assertIn('name', response)
        self.assertEqual(response['name'], 'Test User')

There are two things to note about these lines of code

The fake_urlopen function maps https://api.github.com/users/test_user to tests/resources/users/test_user and returns the file-like object.
Creating the patch in the setUp method can be very useful to reuse it between tests of the same test case. But, you can also use @patch('client.urlopen', fake_urlopen) as a decorator for specific tests.

The content of the tests/resources/users/test_user file must be generated from a real HTTP response. Here's an example of its content

{"login":"test_user","id":161046,"html_url":"https://github.com/test_user","type":"User","name":"Test User"}

And finally, you can verify that everything works by running nosetests

$ nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.011s

OK

This was just a little example of what can be done with mock objects to test REST APIs. This example is based on the tests I wrote for pabluk/twitter-application-only-auth.

If you have ideas, suggestions, corrections or improvements, let me know!

D'oh!

Testing your REST client in Python