#NigeriaDecides 2.0: The Presidential
I will be liveblogging Nigeria's presidential election from inside the headquarters of Project Swift Count 2011, NDI's local election monitoring partner. At least, whenever firefighting permits.
7:45 PM Well folks, that's it. We've hit our targets for the presidential; as a statistically based sample, it is critical that we get a certain amount of complete, accurate information. However, once you've got that, in some ways anything else is overkill. At 7 we got well beyond what we needed for our distribution, and an astonishingly high 95% or so of all our core observers got us confirmed data. The data clerks whooped and gave a big round of applause. I imagine they were pretty enthusiastic to learn that they would not have to spend the next 4 hours yelling "No, D P! Not D D!" Apparently the NATO alphabet is not in common usage in Nigeria. Now our number crunchers will dig into the information and come up with the detailed analysis.
I will go home and brood over the various tech problems we encountered and attempt to mitigate them. With some work and some luck, the Gubernatorial races in a week will go better from that perspective. The project as a whole, however, has been a tremendous success. While this was politically the highest-stakes day for Nigeria, the most difficult one for NDI and Swift Count will be the next. We're doing 6 concurrent PVTs. Unprecedented, as far as I know.
Check out the interim statement on the presidential election put together by Project Swift Count. And check out some of the news coverage. I've been so focused on the immediate problems in front of my nose that it's been hard to step back and think about the magnitude of what's going on here. People - not just Swift Count observers - are incredibly invested in this election, and see it as an opportunity to break the cycles of corruption in this country. Now, for me it's beer, food, and bed.
1:45 PM Finally a bit of calm. Things have been stable for the last couple of hours, giving our monitors a chance to catch up on their DB entry. Thanks to that, numbers are looking great. Even the last report of the day with final result counts is up over 80% in, with less than 10% entirely missing. Time to catch up on the extremely-pressing-analysis that was my tippity-top priority 24 hours ago when the Four Horsemen came through.
9:00 AM Well, they did work for a while. Maybe half an hour. Then we had a half-dozen techies and semi techies huddled in circles arguing about what to do or - worse - running off in different directions trying to twiddle different knobs. International development programs, particularly in places as challenging as Nigeria can be, always try to anticipate technical problems. Usually not quite as many as we've been plagued with. All our data clerks dutifully rolled over to paper and continued on their way, then prepared to enter it in the database later.
7:00 AM Bright-eyed and bushy-tailed, we're ready to rock. By the end of the day we got a lot of the data in, but far from all. Some, thanks to the major delays on the MTN network, was simply very late to arrive. I pity those who finally had their messages processed at - and received the auto-generated response text - at 3:00 AM. OK, maybe I don't pity them. We're back today to chase the rest of the data; the observers will be in at 8 to start pounding the phones again. I got in early to make sure things were working.
5:30 AM OK, we turned my personal Ubuntu laptop into a server. Database replication+Git repository cloning made it a challenge but entirely doable. Enough for one day.
3:10 AM You can not make this stuff up. Our local server just crashed. Some time when I'm not so generally irritated I'll outline the system we have set up; it's a reasonably slick architecture. Suffice it to say we have a remote server and a local server. Scratch that; we had a local server. The hard drive apparently decided to destroy itself. I assume it's in the pay of local despotic forces who wish to stop this monitoring mission from going forward. Tim and I are currently building out an emergency backup system. Looks like you might get some more updates after all.
1:00 AM Getting pretty quiet around the NIC. We set home the data clerks about a half hour ago, and I finally finished my data parser for the INEC ward-level information. (Yes, thanks very much Perl, I had a great time too. No, probably not this weekend. Actually, I'm going to be busy for the next couple months, but let's definitely get together again sometime. I'll give you a call, mmmkay?) The messages are continuing to pour in at the same steady rate; they should finish catching up in the next hour or so.
Regrettably the downtime means that some people texted in multiple time, since they didn't get a response - that means we're going to have to deal with a bunch of redundant messages. Meanwhile I've got some other queries to run, some other analysis and a bit of prep for tomorrow. I didn't get to the other projects I'd hoped for - we're trying to get some mapping going for the election - but maybe we can get that tomorrow.
11:45 PM <humming>Back in business and ain't it grand...</humming> We're reconnected to MTN* and messages are again coursing through at about 50/minute. Tension has dropped palpably. * Their webserver was actually down during the Time of Troubles here. Apparently they were facing some sort of major malfunction.
10:30 PM Things are getting interesting. Our message flow had dropped considerably, but we were having a really hard time diagnosing why. The most common problems - database issues, code slowdowns, DDOS - didn't seem to be the case. More investigation needed, but it looks like our SMS gateway is losing its link to one of the big network providers. Meanwhile, my friend at the next desk has decided to take her calls under the long drapes that surround the cheap fold-up tables. Probably feels like being in a fort. I might try that myself.
7:55 PM One of the things with being an international technology strategy consultant is that no one quite knows what that means. As soon as folks hear the "tech" part, they immediately need help with anything and everything that involves their computer. Since we often are in fact the most technical people around, it can degrade to helpdesk work pretty fast. Actually there are a couple extremely strong tech support folks here, so I haven't had to do too much of it, but a good bit of today has been getting Windows 7 to connect to the network, making computers print, restarting things, etc. Not exactly in the job description, but whatever it takes to get things done.
7:05 PM Well, that was invigorating. Nothing quite as refreshing as a MASSIVE NETWORK OUTAGE to wake you up and fill you with vim and vigor. We're back now. While Mr. Murphy probably took a year off my life, he did have the good grace to KO our internet access during the shift change. The long-suffering group of data clerks who arrived at 7:00 AM finally left at 6:00 PM, replaced by a bright-eyed, bushy-tailed crew. These folks will be here until 1:00 AM, unless we pull the plug earlier. Please let that not mean the Internet again.
4:50 PM In which I plead for help on a massive datamangling task working with the National Election Commission's messy information. Hey there, Perl. Yeah, yeah, I know. It's been a while. I've been meaning to stay in touch, but I just got busy I guess, ya know? I mean there was the grad school, and the politics, and before you know it it's been 6 years since I checked in with an old friend. I thought about you a lot, though. No, really, I did. Hey, that's unnecessary. It's not like you've been aging so gracefully yourself.* Yeah, I know, I did tend to chase after the sexy new things from time to time - man, PHP was sure a beauty, and I've never had a relationship that moved as fast as Rails. Mmm... What? Right. Sorry. So, I guess what I'm trying to say is that I have 38 poorly formatted spreadsheets, and I need to turn them into an aggregated database table. Can you do that for your old pal Chris? Oh c'mon, I never said that. Well, maybe I did, but you called me lazy, impatient and hubristic. Of course I'm not using you for your regex engine! Sometimes old friends are just the best friends, you know? *cough* Sure, of course I'd like to go have dinner sometime, that sounds great. Maybe in another couple weeks? Give me a call after we're done with this, we'll work something out. Exit arm in arm stage right. *Notice the dateline on that post!
4:05 PM The higher-ups at NDI just dropped by for a visit. Compared notes on what they've observed with their travels around the polling stations and remarked that it was a wee bit hotter outside than it is in the center. Personally I'd love to go make sure the sun still exists. Rather a timeless, clockless hole here. Actually, that's not true. Now it's 4:10, so time is actually racing along in an alarming manner. I sat down with a techie from the Egyptian delegation to talk them through our rather elaborate dual-server setup. He was pretty impressed. I was impressed myself; apparently he's built a pair of systems from scratch himself using GSM modems rather than gateways.
3:15 PM The data clerks are taking a breather and playing a Nigerian game called "Arrows and Rabbits." Nice little energy pickup. A lot of screaming.
3:05 PM Wouldn't be a big day without one major system failure. System rebooted, came back up. No harm done.
2:25 PM Midafternoon crunch time. The numbers are coming in fast and furious, and there's no given time to end accreditation. Some got done early, some are probably still going - if you were in line at 12:30, you were supposed to be able to get in however long it takes. Now I'm crunching some observer data to weed out anyone who might not be in the right place. There's a lot of error-checking that goes into a project like this to try and keep the data as clean as possible. Next up, Margin of Error calculation. In other critical Swift Count news, I was mislead about the Indian buffet. The spaghetti was a poor substitute.
12:25 PM Accreditation is wrapping up now, though some some places it will continue for a while yet. Without any prompting 1/4 of folks have already texted in. This will be the next big load test, though it will be more distributed than the 7:30 check in was. I'm working on grabbing the polling place a given observer is supposed to be at and comparing against the number they have submitted. If they mistyped, are in the wrong place, or are lying we can follow up via phone. The data clerks have stopped chasing the folks who did not check in from the AM - most of these will be in places where there is no cell service. We'll hit them hard starting at 9:00 PM so we can get all their information at once.
12:10 PM Huh. Looks like we don't have the web interface that lets us see results yet. Well, that's fine. There's almost five and a half hours left for the developers to code one. Meanwhile, I'm off for Indian buffet.
11:45 AM A delegation from Egypt is here to observe the election and learn more about running a big PVT monitoring operation. Lord knows they've got a big challenge coming up.
11:25 AM Been a bit of a long day already; energy levels dropping. It's a slow time in the center as apart from chasing the last missing data. For a PVT to work, you need to do margin of error calculations, so I'm working on some of those. For processing info that isn't presented via the web interface, we're pulling direct from the database and then feeding results sets into Excel for some aspects of analysis. There's always a lot of additional questions you want to ask your data that you haven't plotted out weeks in advance such that the developers can implement them. It's also nice being able to bypass the filters. The last few days have definitely been a chance to polish up the MySQL query-writing skillz. (I still can't believe that MySQL is now owned by Oracle.)
10:38-10:43 AM Pro tip: don't run database queries that will lock everyone's web interface for 5 minutes.
10:15 AM We're half way through the accreditation process. Nigeria is taking extraordinary steps to try to avoid fraud in these elections. First, people line up in the morning to check in; their names are scratched off the list but they do not vote at that point. Then, starting at about 12:30, those who have been accredited are permitted to vote. This two-stage system is designed to make it very difficult to vote in multiple places. It sure puts a huge burden on the voters, though - this will be a looooong day for them. Still, the general vibe I've heard (secondhand) is that people are really excited to vote - they believe that this time the elections are going to be legitimate. The National Assembly elections were pretty good (with plenty of room for improvement.) Fingers crossed that this will be the same or better.
9:45 AM We just hit 90% of all observers successfully checked, mostly via SMS. Actually they probably did so earlier; we appeared to hit a bottleneck above 60 messages/minute, so it took a while to chew through them all. Pretty impressive. Now our data clerks have to start calling all the laggards and chasing the data.
8:22 Coffee arrived.
7:30 AM My mood was not improved when I came in and found that our external web site wasn't functioning. Our developers, TimbaObjects, had made some significant architectural changes. This is rather heartburn-inducing, as you generally do NOT want to do such things after you've pressure tested your systems. Of course, the reason we did the simulation was to identify issues, and this was to resolve one. My mounting panic was relieved when Tim and his team came in and pointed me at a new temporary domain name. Eventually the DNS TTLs (no, not this Time to Live) will expire and we can get to it directly. All systems are go, and the first group of observers are beginning to send their "I am here" check-in messages now. Victor, our Project Manager (pictured), gave us a stirring speech to get things rolling, reminding all 40 data clerks and their managers that we are here to build a better Nigeria.
6:30 AM The Presidential is here. It's a big day for Nigeria and for the NDItech team. We've been working on this huge election monitoring project for the last 10 months. While this is only one of three elections, in many ways it's the most high-stakes of them all. I'll be updating this post throughout the day with my thoughts and observations. The 6:00 wakeup was a bit rough, with pre-gameday jitters and nightmares of major malfunctions keeping me awake. Also, of course, the Elephant Bar kept rocking into the night. For whatever reason, An Englishman In New York is a perennial favorite.