Enlarge
Election tech 2012
- Romney campaign got its IT from Best Buy, Staples, and friends
- Which consultants built Romney's "Project Orca?" None of them
- Built to win: Deep inside Obama's campaign tech
- Orca was no fail whale, says Romney's digital director
- Inside Team Romney's whale of an IT meltdown
View all…Despite running a campaign with about twice the money and twice the staff of Governor Mitt Romney's presidential bid, President Barack Obama's campaign under-spent Romney's on IT products and services by $14.5 million, putting the money instead into building an internal tech team. Based on an Ars analysis of Federal Election Commission filings, the Obama campaign, all-inclusive, spent $9.3 million on technology services and consulting and under $2 million on internal technology-related payroll.
The bottom line is that the Obama campaign's emphasis on people over capital and use of open-source tools to develop and operate its sophisticated cloud-based infrastructure ended up actually saving the campaign money. As Scott VanDenPlas, lead DevOps for Obama for America put it in an e-mail interview with Ars, "A lesson which we took to heart from 2008 [was that] operational efficiency is an enormous strategic advantage."
As we revealed in our recent analysis of the Romney team's tech strategy, the Romney campaign spent $23.6 million on outside technology services—most of it on outside "digital media" consulting and data management. It outsourced most of its basic IT operations, while the Obama campaign did the opposite—buying hardware and software licenses, and hiring its own IT department. Just how much emphasis the Obama campaign put on IT is demonstrated by the fact that the campaign's most highly paid staff member was its Chief Integration and Innovation Officer, Michael Slaby, with an annualized salary of about $130,000.
By comparison, Kevin Rekowski, the Romney campaign's Director of Technology, was barely in the top 20 salaries of the Romney campaign, with an annualized salary of $80,000. Zac Moffatt, Romney's Digital Director —a social media planner, not a technology expert—was number five, at $175,000 a year, in addition to whatever he earned from hiring his own firm, Targeted Victory, to handle much of the Romney campaign's digital strategy.
But the advantage of having a personal army of coders wasn't just financial. "Campaigns are serious tests of your creativity and foresight," VanDenPlas explained. "They are unpredictable, agile, and short—an 18 month, $1 billion, essentially disposable organization. Hackers can thrive in an environment like that, to a point where I'm not sure anyone else really can. Everything is over far too quickly to get boring."
Smart, not perfect
The strategy the Obama campaign's DevOps team used to manage the ever-growing number of applications deployed by the campaign was to "choose the lowest cost route to get us the most results—basically, be smart, not perfect," VanDenPlas said. "We did a lot of work to make things simple, and when you have a team that is unfazed by limitations, you get some really amazing and creative solutions, some of which I hope to see come out as open sourced projects here shortly."
Key in maximizing the value of the Obama campaign's IT spending was its use of open source tools and open architectures. Linux—particularly Ubuntu—was used as the server operating system of choice. "We were technology agnostic, and used the right technology for the right purpose," VanDenPlas said. "Someone counted nearly 10 distinct DBMS/NoSQL systems, and we wrote something like 200 apps in Python, Ruby, PHP, Java, and Node.js."
It also helped that the campaign, at least for internally developed applications, relied almost exclusively on Amazon Web Service for its infrastructure, eliminating a lot of the financial burden of infrastructure management. "For the applications built by the OFA [Obama for America] technology team, 99.999 percent were AWS hosted," VanDenPlas said, "purely because it was the best fit for what we were doing. As a whole, if you include privately hosted virtualized environments in the cloud architecture definition, I believe everything was 'cloud,' even down to our development environments running inside of Vagrant on our laptops."
The system configurations for the campaign's Elastic Compute Cloud (EC2) instances were created using the Puppet configuration management tool and were built as Debian packages kept in the campaign's own Advanced Packaging Tool (apt) repository—both for internally developed and third-party applications. As the number of applications and the scale of the campaign's AWS infrastructure use climbed, the DevOps team shifted to using Asgard—an open-source tool developed by Netflix to manage cloud deployments.
To help optimize applications, the OFA technology team used New Relic, a tool also used by the Romney campaign. "It is really a fantastic tool that increases your visibility into where your applications are spending time," VanDenPlas said. "They support the major languages we used (Python, Ruby, PHP) as well as the frameworks (Flask, Rails, Kohana)."
While AWS's tools were used for performance monitoring and to trigger automatic scaling-up of capacity, VanDenPlas said, much of the monitoring was handled by a suite of commercial and open source tools and home-grown code, "consisting of Cacti, Opsview, StatsD, Graphite, and Seyren, and a number of custom applications that continued to evolve right up until Election Day," VanDenPlas said.
To get better aggregated alerting and metric data, the team built a lightweight plugin for Nagios (the open-source basis of Opsview) in Python based on boto (the Python programming interface to AWS's services) and dotCloud's ZeroRPC messaging interface. "Using this," VanDenPlas explained, "we could constantly query thousands of nodes for near real-time statistics and feed them right back into the same alerting and monitoring system (Nagios) we used elsewhere."
Other performance monitoring and user experience data was collected using Chartbeat and Google Analytics. "Akamai also provided very useful statistics and logging," VanDenPlas said, "but these were mostly contextual rather than actionable." But, he added, the most heavily used monitoring system was "our community of internal and external supporters. The human factor in monitoring is huge. There are countless incidents where (OFA User Support Director) Brady Kriss notified us of pending problems derived from community help tickets."
The armor-plated cloud
The OFA engineering team also did a lot of work to ensure that they got the most out of Amazon's cloud architecture in terms of resiliency. As the election approached and the infrastructure demands surged, the engineering team took advantage of Amazon's multiple availability zones within its Virgina data center. "We built out a triply redundant, encrypted, and compressed WAN optimized tunnel between AWS regions," VanDenPlas said, "using a combination of OpenVPN, CloudOptimizer, and some DNS trickery."
The team shifted its domain name service to Amazon's Route 53 service, which uses latency-based routing to direct users to the host running in the AWS availability region with the shortest network trip time. That allowed the Obama team's application deployments to use "regionless" generic configuration settings, making deployments much simpler.
The centerpiece of the whole Obama campaign was its fundraising capabilities, without which all of the other applications may have been moot. The 2012 campaign's online donation system was a complete rebuild from the 2008 effort, VanDenPlas said, "a multi-region, geolocated, three facility processor capable of a per second transaction count sufficiently high enough that we failed to be able to reach it in load testing. It could also operate if every other dependent service had failed, including its own database and every vendor."
The Obama campaign's websites were also hosted on Amazon and hardened. The campaign's engineers built an application that created static HTML snapshots of the sites stored in Amazon's Simple Storage Service (S3); in the event of a Web server failure, requests would be instantly directed to the latest snapshot.
All of that redundancy was given an extra workout in the week before the election as Hurricane Sandy approached the East Coast. VanDenPlas said that a "complete hot replica of our entire infrastructure" was deployed to Amazon's primary West Coast data center in under 24 hours as a precaution.
Build, borrow, or buy
The tech team wasn't the only internal IT operation at Obama for America. The campaign ran its own data analysis shop and had its own army of Web designers and administrators. And with a payroll of over 1,000 people, the IT team had a lot of tech to support for an organization that had essentially a 24-month lifecycle.
That meant buying a lot of hardware and software. CDW, based outside of Chicago, was the go-to supplier for much of the campaign's computer equipment and boxed software purchases. Microsoft also sold $522,210 worth of software licenses to the campaign—which averages out to just under $500 per staffer.
| Obama Campaign Hardware and Software purchases (expenses to companies of over $1,000 total) | |
|---|
| CDW DIRECT | $1,506,548.22 |
| MICROSOFT | $522,210.04 |
| RENTRAK CORPORATION (media ratings analytics) | $359,182.43 |
| APPLE | $111,376.41 |
| SALESFORCE.COM INC | $154,441.89 |
| B & H PHOTO (the NYC-based computer, camera and video superstore) | $231,982.76 |
| NATIONAL FIELD (enterprise social networking) | $101,933.44 |
| AMAZON.COM | $74,353.10 |
| VISIBLE TECHNOLOGIES (social media monitoring) | $56,250.00 |
| ALLPOINT VOTER SERVICES, INC. (electronic voter registration) | $55,772.50 |
| GET CONNECT (Adobe Connect conferencing service) | $50,025.87 |
| KXEN (predictive analytics software) | $50,000.00 |
| NGP VAN (fundraising and compliance software) | $48,500.00 |
| MARIN SOFTWARE (advertising management software) | $45,000.00 |
| TABLEAU SOFTWARE (analytics) | $33,750.00 |
| ENTRUST, INC. (Public Key Infrastructure, certificates) | $29,124.47 |
| MBS INC. (a Mac repair shop) | $26,918.39 |
| AVF CONSULTING (Microsoft Dynamics NAV accounting software) | $24,360.00 |
| LAYERED TECHNOLOGIES (PCI compliance for online payments) | $26,324.00 |
| INTUIT | $43,919.27 |
| ZENDESK (social network monitoring and helpdesk software) | $27,395.00 |
| NETBASE SOLUTIONS | $15,000.00 |
| VIDEO EQUIP RENTALS | $14,760.87 |
| FRONT RUNNER PRODUCTIONS, INC. | $10,050.00 |
| CHARLTON VENTURES LLC | $9,995.00 |
| TIGER DIRECT | $8,553.68 |
| STATACORP | $8,242.35 |
| MARKETING LUCIDITY | $7,500.00 |
| NETBASE SO | $7,500.00 |
| TALENT TECHNOLOGY CORP | $7,200.00 |
| CLEVERBRIDGE | $6,912.00 |
| LEXJET | $6,841.99 |
| MARKERTEK VIDEO | $6,790.00 |
| ESRI | $6,756.75 |
| CONSCIOUS MOBILE | $6,700.00 |
| ZACUTO | $6,143.58 |
| DIGITAL CINEMA SERVICES | $6,000.00 |
| THE RESUMATOR | $5,997.00 |
| TIGER DIRECT INC. | $5,348.69 |
| PC CONNECTION, INC. | $5,024.49 |
| SPROUT SOCIAL | $4,247.52 |
| IUNDERTAKE | $4,054.00 |
| ISTOCK PHOTO | $3,828.50 |
| MAXON COMPUTER | $3,711.56 |
| MASHERY, INC. | $3,600.00 |
| MAXON COMP | $3,512.24 |
| PAYPAL INC | $3,465.00 |
| DOMAIN NAME REGISTRATION | $3,195.00 |
| THE MIS DEPARTMENT, INC. | $3,148.00 |
| CALUMET PH | $2,956.45 |
| CALUMET PHOTO | $2,261.05 |
| E NOM INC | $2,240.00 |
| THE HOEFLER TYPE FOUNDRY, INC. | $4,284.35 |
| CHARLTON V | $1,999.00 |
| DC CAMERA | $1,849.25 |
| THEALLSAVEDCOMPONENTS | $1,799.00 |
| SUPERGROUP | $1,739.22 |
| TERREMARK NORTH AMERICA, INC. | $1,661.30 |
| NYCE SALES | $1,618.99 |
| CHARTBEAT | $1,579.70 |
| GOHARDDRIV | $1,547.40 |
| THE HOEFLER TYPE FOUNDRY, INC. (D/B/A HOEFLER & FRERE-JONES) | $1,498.00 |
| ABESOFMAINE | $1,460.18 |
| OFFICE DEPOT | $1,430.81 |
| GITHUB | $1,400.00 |
| RED GIANT | $1,298.00 |
| TENABLE NETWORK SECURITY | $1,275.00 |
| MONOPRICE | $1,234.80 |
| OFFICE DEPOT #5910 | $1,230.42 |
| ZACUTO LLC | $1,225.80 |
| FTN GROUP INC | $1,200.00 |
| PEOPLEBROWSR | $1,200.00 |
| DEEP SURPLUS | $1,176.79 |
| BEST BUY | $1,163.61 |
| PSAKI, JENNIFER R | $1,099.00 |
| VIP PROMPTING CORP | $1,087.00 |
| PAYPAL | $1,078.50 |
| GOVGROUP | $1,045.54 |
| ENOM INC | $1,010.00 |
| Total (including all others) | $3,398,888.03 |
With its investment in the cloud, the Obama campaign's Web hosting costs were much higher than Romney's. The largest cost, however, was content hosted by Blue State Digital, the social media and interactive advertising agency; the cost for hosting the internally developed applications in the Amazon cloud was a quarter of that:
| Web Hosting and Web Services | |
|---|
| BLUE STATE DIGITAL, LLC | $1,041,488.24 |
| AMAZON WEB SERVICES | $257,287.97 |
| AKAMAI TECHNOLOGIES, INC. | $69,373.64 |
| OBAMA VICTORY FUND 2012 | $66,381.25 |
| OPTIMIZELY | $30,000 |
| PETERSON, KELLIE (a merchant services consultant) | $12,919.75 |
| GODADDY | $8,189.07 |
| LAYERED TECHNOLOGIES | $2,027 |
| LOGGLY | $1,284 |
| CLOUDABILITY, INC. | $1,194 |
| SYNETECH, LLC | $1,000 |
| MARCARIA | $948 |
| LOGGLY INC | $882.12 |
| MAILCHIMP | $550 |
| GITHUB | $475 |
| URBAN AIRSHIP INC. | $326.54 |
| GITHUB INC | $200 |
| Total (including all others) | $1,494,681.43 |
And then there was the Obama campaign's outside technology help. As mentioned in our previous coverage of the Obama campaign, advertising company Blue State Digital and campaign software provider NGP VAN provided the largest chunks of Obama for America's technology consulting, and are most directly comparable to the over $14 million paid out by the Romney campaign to its digital firm, Targeted Victory. Even taken with the software and Web hosting expenses, the Obama campaign spent a seventh of what the Romney campaign spent on digital and an even smaller fraction of what Romney spent on voter and donor contact.
| Technology consulting and services expenditures | | | |
|---|
| Name | Consulting | Services | Total |
| BLUE STATE DIGITAL, LLC | $1,384,934.46 | | $1,384,934.46 |
| NGP VAN | $743,455.80 | | $743,455.80 |
| THE MIS DEPARTMENT, INC. | $730,388.00 | | $730,388.00 |
| ANALYST INSTITUTE | $222,318.00 | | $222,318.00 |
| THIRTEEN23 CORP | $125,000.00 | | $125,000.00 |
| OBAMA VICTORY FUND 2012 | $111,500.00 | | $111,500.00 |
| COMSCORE INC | $90,000.00 | | $90,000.00 |
| BLUE CITY STUDIOS | $44,512.50 | | $44,512.50 |
| BRIGHTTAG, INC | | $29,420.08 | $29,420.08 |
| AVF CONSULTING | $22,365.00 | | $22,365.00 |
| TERAKEET CORPORATION | $19,754.00 | | $19,754.00 |
| SYNETECH, LLC | $18,200.00 | | $18,200.00 |
| AKPD MESSAGE & MEDIA | | $17,500.00 | $17,500.00 |
| BRIGHT BLUE DATA LLC | $10,000.00 | | $10,000.00 |
| UDELL, MADELEINE | $6,900.00 | | $6,900.00 |
| BIG HEAD LABS, INC. (DISQUS) | | $6,750.00 | $6,750.00 |
| DENTON, GRANT | $5,700.00 | | $5,700.00 |
| CDW DIRECT | $1,069.63 | $4,510.06 | $5,579.69 |
| ZENDESK | $3,669.00 | | $3,669.00 |
| BIG HEAD LABS, INC. (DISQUS) ATTN:ACCOUNTS RECEIVABLES | $2,812.50 | $562.50 | $3,375.00 |
| TRAMMELL, MARK | $3,000.00 | | $3,000.00 |
| THE SYNETECH GROUP | $2,750.00 | | $2,750.00 |
| TARGET SMART COMMUNICATIONS | | $1,730.76 | $1,730.76 |
| CASTLETON, DAVID | $1,450.00 | | $1,450.00 |
| ADDIS, DAVID | $1,378.10 | | $1,378.10 |
| PACIFIC EAST | | $853.56 | $853.56 |
| WAUGH, KIERAN | $600.00 | | $600.00 |
| PETERSON, KELLIE | $500.00 | | $500.00 |
| AMAZON | | $401.47 | $401.47 |
| DEMOCRACY ENGINE LLC | $400.00 | | $400.00 |
| GOOGLE WALLET | | $360.00 | $360.00 |
| APPLE | | $317.69 | $317.69 |
| PAYPAL | $53.24 | $258.00 | $311.24 |
| HUMAN RIGHTS CAMPAIGN C/O MIKE MINGS | $245.00 | | $245.00 |
| ENTRUST TECHNOLOGIES INC | | $158.77 | $158.77 |
| ILLINOIS INSITITUTE | $150.00 | | $150.00 |
| Grand Total | $3,553,105.23 | $62,822.89 | $3,615,928.12 |
Return on investment
In the end, the deciding factor wasn't what the Obama campaign spent money on, but what it did with all that money. Insourcing gave the campaign a strategic flexibility that the Romney campaign lacked, as well as other intangibles that may have contributed to leading an efficient campaign. And the reduced reliance on outside consultants allowed the Obama campaign to direct capital toward places where it had a bigger impact—such as in advertising, where the Obama campaign outspent Romney by a factor of 5 to 1.
"This is the difference," VanDenPlas said, "between a well run professional machine and a gaggle of amateurs, posing in true Rumsfeldian fashion, who 'don't know what they don't know.' I would be shocked if such a chasm exists next cycle between the parties—these aren’t mistakes to be repeated if you want to do things like win elections."
Given the response from Republican partisans to the failure of Romney's campaign and to the apparent failure of its technology investments, the stakes for the next time—the mid-term elections in 2014, and the next presidential race in 2016—will be that much higher for Republican campaigns. It's doubtful they'll ignore the lessons learned this campaign season.
Rebekah Emily Symons
If you’ve got someone on your shopping list that’s hoping to find a shiny new iPhone, iPad or MacBook in their stocking, pay attention: Apple has announced a one-day Black Friday sale on some of its hottest products.