Using UNICEF's RapidSMS to Build an SMS-based Application

15 December, 2009

I've mentioned Clay Shirky's Design For UNICEF class once before: here. The goal of the class is to come up with ways of using the rapidly growing number of cell phones in sub-saharan Africa as a platform for projects that would improve the health, education, or security of young people.

My final project for the class was Infone, a tool for rapid surveying of local conditions by text message with incentives for participation. The idea is to send a regular stream of text messages to people in an area in which it's hard to gather data about living conditions, paying them with free cell phone credits. In these messages you ask simple factual questions designed to elicit information that might be useful for deriving facts such as road outages and the existence of conditions that might lead to disease outbreak.

For example, if you regularly ask about the existence of standing water in a village and suddenly you see an increase, that's a good reason to expect a rise in malaria in that area due to improved mosquito breeding conditions.

The project is designed to ask simple factual questions that are easiest to answer truthfully. That combined with the fact that we control who gets asked and how frequently should limit the danger of bad actors exploiting the system for more than their share of the free cell phone credits we plan to distribute to encourage participation.

We've gotten great feedback throughout the semester from UNICEF staff, Clay, and our fellow students. For more info, you can view the slideshow we presented at UNICEF and checkout our in-progress website.

RapidSMS

We're building our Infone prototype on top of RapidSMS, a framework developed by UNICEF for handling incoming text messages inside of Django.
RapidSMS consists of three main pieces: the router, which processes incoming text messages and sends them to each application for handling; a library of existing applications that implement common functionality such as tracking users, sending credits, even bridging between SMS and IRC; and, finally, Django itself for building a web interface for non-SMS tasks.

RapidSMS is a great project that's surprisingly hip for something developed inside of such a large organization. For example, they use GitHub for distributed version control with a ton of forks. However, the project's biggest downside is its lack of clear and well-organized documentation. If we hadn't had constant access to the project's developers themselves inside of UNICEF through this class, I don't think we would have gotten nearly as far with it as we did in implementing Infone.

To attempt to partially remedy this last problem, I'm going to document here everything I've learned about using RapidSMS to develop a prototype SMS-handling application.

Installing Django

It goes without saying that in order to run RapidSMS you'll need Python and Django. For this, Django's install page is probably the best starting point. To my surprise, I struggled quite significantly with this stage and was taken aback by the lack of a consensus packaging system in Python. Somethings come as "eggs" some as tarballs with make scripts, some in other forms.

I'm not going to try to walk you through all of the steps of getting Django running as Google and persistence will provide better and more up-to-date instructions than I could hope to. I will, however, mention that one thing that particularly plagued me was incompatibilities between Python 2.5 and 2.6. I ended up simply pointing the Python in my Bash path to point at 2.5 to get everything to work smoothly.

Once you've got Django installed follow UNICEF's instructions for getting RapidSMS installed and configured. RapidSMS is simply a Django app with some additional infrastructure to allow you to connect to a GSM modem and to build convenient abstractions for handling incoming text messages.

Responding to SMS

Now, we're ready to start using RapidSMS. I'm not going to focus too much on the technical details of configuring and administering a RapidSMS install. Those are available in the project's documentation. I'm going to focus on explaining the RapidSMS architecture so that you can use it to build your own application.

As mentioned above, the first part of that architecture is the router. The router lets us handle SMS by implementing a single class with a series of methods that get called at each stage in the message life cycle: "start", "parse", "handle", "cleanup", "outgoing", and "stop". Each of these methods gets passed an object representing the current text message, including its content, the phone number from which it came, etc.

Before we can experiment with receiving messages, however, we need to create an app. Now that RapidSMS is installed, we can use its command line utility to generate our app:

rapidsms startproject test-project

Running that command will create a new directory inside of your RapidSMS checkout's app directory named "test-project". If you navigate in there you should see two files: test-project/__init__.py and test-project/app.py, which contains a stub of our text message-handling class.

Let's look at the app.py file that got generated for us:

Just like I mentioned above, here are six methods that each get passed an object representing the text message (they also get passed "self" as the first argument, which is a Python convention). They each also have a comment explaining when they get run in the router life cycle. From that, we can determine that the only method we really care about is "handle".

Let's take a look at an example of a handle implementation that actually does something. For this example, I'll pull code from the "echo" app. one of the built-in apps that ships with RapidSMS. Echo simply repeats back anything sent to it with "You said:" in front of it. It's useful for debugging and is probably the simplest demonstration of how the message object works.

This should be relatively self-explanatory. There are only two method calls with any substance. The first thing to note is that we retrieve the full text of the incoming SMS by calling "message.txt". Then, once we've done what we wanted with that (in this case, simply appending a string to it), we write back our response by calling "message.respond()" with our response as the argument. Simple.

(The one other tiny detail to note here is that this method returns True. As we'll see in more detail momentarily, each RapidSMS install can play host to any number of apps. When multiple apps are installed, you declare an order of precedence for them in a configuration file. If any app wants to prevent other subsequent apps from processing a message at any stage, it simply has to return False from the appropriate method in its App class.)

For many reasons, developing an SMS-based app while testing with actual text messages would be extremely annoying: having to acquire and setup a GSM modem before being able to get started, dealing with the vicissitudes of the GSM modem like message cache limits, the lag while waiting for each new message to arrive, not to mention using up your carrier text message allotments on both the outgoing and incoming sides.

Thankfully, RapidSMS provides an HttpTester app that lets you send test messages and receive test responses directly from the web interface. Assuming you didn't modify the default rapidsms.ini file, you should have the HttpTester app running. Visit http://localhost:8000/http and you should see a simple form with fields for "Phone Number" and "Message" (make sure you use HttpTester from Firefox; it uses some javascript that doesn't seem to work correctly in Safari (I haven't tested it in Internet Explorer, but I'd be highly surprised if it worked there either)). Enter a phone number (7 or 8 digits, no spaces would be most realistic) and a test message and hit "send". You should see the "Message Tester" log at the top of the page display your outgoing message and then the responses from any apps that are running (at this point, likely just echo).

Once you're seeing responses from echo, try adding your app to the list of installed apps in the rapidsms.ini file (under the "apps" variable around line 14 of that file; the order that apps are listed here will be the response priority as I described earlier), restarting your router with "./rapidsms route", and seeing if you get the response you're expecting from your app. Watch the log from your route process for errors if you don't get what you're expecting.

Using Apps as Libraries

Now that you've got basic SMS handling under control, the next step is to start building your actual application logic. The key tools for this work are Django models. Models are where you define your application logic: the structure of the data you want to store and the behavior you want to perform. Django models are abstractions on top of relational database tables, which they use for storage, so most of the code in these classes will be to declare the fields on each model (which correspond to columns in the sql table). You can read more about in the Django model documentation.

As a simple example, here's the Response model from Infone which we use to store the text messages that we receive from our participants:

The fields declared as foreign keys point to other models in the system (questions and respondents, respectively), the Char field holds the content of the text message, and the DateTime field holds the time at which the response was received.

Now, we could set about using Django models to build our entire application from scratch. But, if we did that, we'd end up solving a few common problems to SMS-based apps over and over again: recording and tracking respondents, aggregating poll questions, sending out credits, etc.

To prevent this problem, RapidSMS ships with a whole suite of existing applications many of which incorporate these common bits of functionality already. We can use this functionality in our own apps by simply importing these models into our own app and then constructing relations with them.

For an example, let's take a look at the real models.py file for Infone:

Notice, first of all the third line of this file: "from reporters.models import PersistantConnection, PersistantBackend, Reporter". With that one line we've brought over three classes from the RapidSMS "reporters" app. The reporter's app is designed to take in regular messages from a series of people out in the field. It has a lot of functionality that we don't need (such as the ability to classify reporters into different roles), but it has three classes that are a perfect fit for what we're trying to do with Infone: Reporter (which tracks personal details such as first and last name and phone number), PersistantConnection (which tracks the last time we heard from a reporter and what the best medium to contact them is) and PersistantBackend (which encapsulates a set of details for contacting reporters via different backends such as adapters for different GSM modems, or even via IRC or email potentially).

(Note: the misspelling of "persistent" as "persistant" is ubiquitous throughout the RapidSMS codebase; if you're looking for an easy patch to submit, going through and systematically fixing that would be a nice place to start.)

Once we've imported these models, the tables corresponding to them will automatically get created when we run "./rapidsms syncdb" right alongside the ones we need for our own models. And we can create relations with these models just like with our own. Look through the "register_from_message()" class method in our Respondent class for an example of how we use all of these classes together to accomplish something we need for our own application: automatically registering a new respondent from an incoming text message.

A really useful project for someone to undertake (either from within UNICEF or just a good-hearted volunteer) would be to systematically document all of the classes in all of the existing apps and write-up what they're useful for and any quirks or tricks associated with them. Currently, 35 apps ship with the main RapidSMS fork and if even half of them have useful models in them, that's a large and rich library that could be extremely useful to developers if its affordances were well understood.

Django Web Interface

For any non-trivial application, you'll probably want some kind of web interface in addition to the ability to respond to text messages. For example, Infone has an interface for administrators to create questions, assign respondents to them based on their past responses, and view results as they come in, including exporting them to Excel.

Again, the best place to look to learn how to build web interfaces in Django is the Django Documentation, but I'll walk you through the basic outline here of what goes where to get you started and provide some sense of an overview that can sometimes be hard to get from the detailed API documentation.

The two main files for defining a Django app are urls.py and views.py. In RapidSMS, these files live inside of your app directory alongside your app.py file, for example at apps/infone/urls.py and apps/infone/views.py.

The first of these, urls.py defines what URLs your application will respond to and what happens at each one. The second, views.py, defines a method to handle each url; this is where the requisite data is gathered from the models, form data gets parsed and used to create, update, or delete records, and the correct template gets selected for rendering. In the parlance of another popular web framework urls.py defines "routes" and views.py is the "controller".

Let's take a quick look at Infone's version of each of these to get a more realistic sense of what this means. First, urls.py:

Each line here represents a URL pattern and declares which view method should be called when that pattern matches. There are two basic types of patterns here: ones without wild cards and ones with wildcards. For example, this line: "url(r'^infone/questions/new/?$', views.new_question)" simply matches the url "http://localhost:8000/infone/questons/new" (with or without a trailing slash). When that url is requested, the view method, "new_question" gets called.

On the other hand, this line: "url(r'^infone/questions/(?P<id>d+).csv/?$', views.question_csv)" includes a wild card for the "id" parameter. In other words, if someone requests "http://localhost:8000/infone/questions/1.csv", they'll get the CSV file for responses to question 1 and if they request "http://localhost:8000/infone/questions/42.csv", they'll get the CSV for question 42, but either way the save view method, "question_csv" will get called. The parameter will get passed to that view method as an argument, which we can then use to render the correct CSV (or do whatever is appropriate depending on the URL).

So, what do those view methods actually look like? The full Infone views.py is rather long, but here's an excerpted version that shows the implementation of the two view methods mentioned above: new_question and question_csv:

Note how the question_csv method takes two arguments (the request itself and then the "id" parameter) whereas the new_question method only takes one (the request).

I won't go through the details here, but you can see how the new_question method simply uses Django's built in form-rendering helpers to build a form for creating questions and then uses that to render the template (look for more about templates in the Django documentation). On the other hand, question_csv uses the Python CSV library to create a CSV file represent some data about the requested question and then renders that as an attachment so that the user's browser will automatically download it.

Django views can do a lot of additional things not demonstrated here and I highly recommend you spend some time with the Django view documentation.

Initiating SMS sending

One final note before wrapping this up. RapidSMS was designed to respond to incoming text messages, not to initiate sending outgoing messages. All of its infrastructure for sending messages is built to run in a loop that checks the GSM modem for incoming messages and then processes them when they arrive. However for Infone, we needed to initiate sending outgoing messages from the web form in order to send out our questions to registered respondents. This was exceedingly challenging under RapidSMS's current architecture.

Our eventual solution was to create a Target model as a queue of outgoing messages that need to be sent (see the models.py file I embedded above for the details). Then, we implemented our own custom backend that adds a method to the run loop to poll this table looking for new Targets that need sending. Take a look at the "send_next_message()" and "run()" methods in our infone_backend.py to see the code. Other than those two methods, it is identical to the generic RapidSMS GSM backend.

While this is not a perfect solution, it will do the trick for most applications and it is much simpler than running two copies of the GSM connection code (or some other workaround). We've talked some to the UNICEF team about adding this kind of message triggering functionality to the system as a whole, but since it bridges between the apps and the backend, we haven't yet hit on a way of doing it that is adequately modular.

Get Involved

Our deep gratitude goes out to the UNICEF team working on RapidSMS, especially Evan Wheeler, without whom Infone wouldn't have been possible. They've provided incredible support in class, on the #rapidsms channel on Freenode, and on the RapidSMS mailing list which are both great places to seek out support for your projects and to get involved with making RapidSMS better.