Reverse-Engineering Twitter To Solve An Advertising Mystery

Recently I opened the Twitter app on my Mac and noticed something very strange: It was omitting promoted tweets from my timeline. In the side-by-side comparisons below, notice the "howaboutwe.com" promoted tweet from Twitter.com on the right is missing from the Twitter.app feed on the left.

An example of Twitter breaking its own rules, or just a bug?

Then I found a missing promoted "Pepsi" tweet on Twitter.com which was absent from the feed on my iPhone Twitter app. There was definitely something going on here. Why would Twitter differentiate content between their official mobile app and their web client?

Back in September 2010, Twitter said that it was using its own API. "It fetches data from the same endpoints that the mobile site, our apps for iPhone, iPad, Android, and every third-party application use," wrote Britt Selvitelle in a post on the engineering blog. That appears to no longer be the case, which raises an even more hairy question: Does Twitter have an undocumented API for promoted tweets? To find out, I did what every curious engineer would and decompiled the app. (Twitter, if you're reading this, please don't ban me from your network.)

Reverse Engineering Twitter.app

Twitter Sleuthing 101

1. Disassembled the app using class-dump.
2. Found the code routine that handles network responses using Hopper.app.
3. Detected network responses from the API using GDB.
4. Compared the API output to my Timeline feed using Rested for Mac.

My first idea was to set up a local proxy and examine if the Twitter.app was filtering out promoted tweets, or whether it was receiving them at all. Setting up a proxy proved not to be as easy as I had hoped. Twitter.app uses SSL encryption to communicate with the API server and because it's set to reject self-signed certificates, trying to spoof an SSL certificate renders the app completely non-functional.

For help, I enlisted my brother Daniel Schonfeld, a seasoned hacker for help disassembling Twitter.app. Fortunately for us, the app was written almost entirely in Objective C, which meant that we'd be able to examine some of the app's source code as it was written by Twitter's very own engineers; usually, when disassembling C/C++ binaries, source code loses its original naming symbology, but Objective-C preserves a lot of symbols which makes it easier to follow along.

Using a nifty tool called class-dump, we began examining the contents of Twitter.app and how it was designed. First, we set out to see if we could find any mentions or references to promoted tweets. It wasn't too long before we found this little gem buried within the code definition of the "Twitter Status" object. Notice the built-in flag for promoted tweets:

@interface TwitterStatus : NSObject <...>
{
NSDate *lastUpdated;
...
struct {
...
unsigned int isPromoted:1;

From this finding, we could assume that Twitter.app does in fact at times receives promoted tweets from the API server and classifies them as such using the "isPromoted" flag.

On to the Data Stream
Once we knew that Twitter.app has some built-in handling for promoted tweets, we had to dig deeper. Daniel fired up Hopper.app, a Mac disassembler, and a couple hours later, boom: He had found the code routine that handles network responses. Namely, the [ABHTTPRequest connection:DidReceiveResponse:] message (method). Now could finally answer our initial question: Does the app receive promoted tweets from the API server or not?

We used GDB to set up breakpoints in Twitter.app and forced it to show us what it was getting back from the API server. Specifically, our debugger will halt every time the app receives a network response.

To repeat our method, fire up a terminal window and launch the debugger:

> cd /Applications/Twitter.app/Contents/MacOS/
> gdb --arch=i386

Then, while working within gdb, type:

> (gdb) exec-file Twitter
> (gdb) b *0x6dec3
> (gdb) commands
> x/s $eax
> end
> (gdb) set print elements 0
> (gdb) r

This will cause the debugger to freeze Twitter.app on our given breakpoints. The first thing you'll notice is that Twitter.app is actually working with Twitter's XML format--yuck. The first couple API server responses will be your own account's information. Type **c**, then hit the **Enter** key to continue to the next breakpoint. The next block of XML you'll receive is your timeline feed. Comparing the XML to my Twitter.app feed and my Twitter.com feed, we discovered that Twitter.app is not actually not receiving the promoted tweets in the data feed from the API server at all.

Is This an Undocumented API?
Just one question remained: Were only Twitter's official apps not receiving promoted tweets from the API server? Or, does the API not return them to any client at all?

This was a rather easy test. Using Rested for Mac, I quickly polled Twitter's API server for my own timeline feed. As can be seen in the resulting output above (truncated for your convenience), it appears that the API server simply does not return any promoted tweets.

Conclusion: Twitter Has A Secret Feed For Promoted Tweets
The FAQ on Twitter's dev site says that, "As of March 12, 2012 there is no Advertising API for serving Twitter's promoted products in third party applications." Which means the promoted tweets ought to appear in any timeline that uses the API. That's obviously not the case.

The mobile ad market is an enormous and constantly growing one. Ad revenue is a major component of Twitter's revenue, and the company projects it to grow to $540 million by 2014. Twitter has also promised to "share a portion of advertising revenue" with developers who serve Twitter ads, according to its own Developer Rules of the Road.

Advertisers and developers, it's getting close to 10 p.m.: Do you know where your promoted tweets are?

Michael Schonfeld is head of developer relations at Dwolla, a service that empowers anyone with an Internet connection to send money simply. Follow Michael Schonfeld and his brother Daniel Schonfeld on Twitter.

[Magnifier Image: PzAxe via Shutterstock]

Sara Cox Kyla Cole