The Iron Law of Complexity

You can't wish complexity away. The challenges we undertake in tech and development are, well, challenging. They might involve a lot of people, a lot of documents, a lot of votes, a lot of citizen reports, etc, etc: in any case, a lot of information. Working in development one can - and should - simplify problems as much as possible and focus on what's truly essential, but even the bare-bones solution can still be a lot to handle. Managing that flow of data is at the heart of many NDItech projects, and we've learned that there's an iron law for using IT in development: call it conservation of complexity. It's a bit like squeezing a balloon animal - you can move the air around, but it can't go away. Take an election monitoring mission like Nigeria. (please!) 8,000 monitors sent 5 sets of data into a national center to enable the civil society organization's leadership to make authoritative statements on what had transpired during the day of voting. So, you've got to collect accurate information from your thousands of observers and you can't go in front of the cameras for the big press conference until you've got it. How to make it happen? That decision involves considering program goals, budgets, technical limitations, and the abilities of the people involved. The best fit will depend partly on where you can park the complexity. Here's a few approaches NDItech has considered for election monitoring missions in different countries:

1. Observers write on paper forms, carry them by hand to their bosses, and have them transport them to the center. From there the sheets will be data-entered in a series of parallel Excel spreadsheets which are later merged.
2. Observers text in encoded form responses in a SMS message; software then takes those incoming texts and put it in a database.
3. Observers have special java apps (or smartphones with custom apps) that mimic the layout of the form. When submitted this makes a connection to a central server and updates a database.

Any of these might be OK for a given election - but the balance of capabilities of the observers and analysts and the budget for external techies can help decide where to concentrate the complexity. 1) is simple for the observers - they just write on forms. However, the data entry is hard, and there's a lot of room for error. Typing large amounts of information from thousands of observers into spreadsheets is tedious, difficult and error prone; merging data typed by different folks can generate yet more mistakes. Since it's just in spreadsheets, Excel ninjas working with well-trained statistical analysts will need to run complicated formulas to extract meaningful stories out of the vast array of numbers. You've pulled the complexity into the center. 2) requires having an SMS management system that takes the texts and translates them into information in the database. Creation of that software is hard (NDItech's done it over and over) and probably requires expensive external consultants - but once created, the data entry problem is largely solved. Most folks have or can get at cell phones these days, but that becomes a requirement - as does being in an area with mobile service. Writing those texts aren't easy for the observers - they look like gibberish and that can itself be error prone. In Nigeria a full report could be over 80 characters of alphanumeric soup. There's extra training and difficulty for the observers, as well as the services of a sophisticated (read: expensive) software developer, but makes for relatively easy analysis. 3) means you need to have more sophisticated phones in the pockets of all your observers that are, at a minimum, java-enabled "feature" devices. With the proper (perhaps relatively very expensive) equipment in hand, monitors have to somehow install the form app. Once that's in place, though, collecting and sending data can be a piece of cake, with validation on the phone, easy-to-understand fields, and simple submission. You've still got to have your data processing system on the server side, though there are pre-built