The A/B Test Results Are In

How IGN Makes Small Changes for Mass(ive) Effect

What follows are seven tweaks gaming site IGN made to its homepage. Employing the A/B testing methodology, IGN is constantly modifying its site looking for those small changes in wording, placement, even color and form that will yield huge increases in click-through rates.

Original Version-click image for Revised Version

1. CHANGE: Moved “Videos” in the nav bar from in between “News” and “Guides & Cheats”, to in between “My IGN” and “Xbox 360”
RESULT: Moving the “videos” nav to the left resulted in an almost 93% drop in clicks on the “videos” nav link. Clicks to the rest of the nav items were either unchanged or improved slightly.
THOUGHTS: This was a huge surprise to the IGN team, and raised further questions about how IGN users behave. Are they simply accustomed to the “videos” link in its old location? Would the click-through-rate have improved given more time for the users to get accustomed to this change? Are most of the users clicking on the “videos” link return users, or new users?

2. CHANGE: “iPhone” in the nav was swapped out for “Mobile”
RESULT: A 5% drop in clicks.
THOUGHTS: IGN believed its population of committed iPhone-haters would be put off from clicking on “iPhone” in the nav. Turns out they weren’t.

3. CHANGE: “Mass Effect 3 Review” headline rewritten as “EXCLUSIVE: Mass Effect 3 Review”
RESULT: IGN found that almost any headline with “exclusive” in it had on average 5.2% fewerclicks.
THOUGHTS: IGN isn’t giving up on using “exclusive” for other types of content. The lesson is, it really needs to be exclusive.

4. CHANGE: The “Subscribe on YouTube” button in the right rail START unit was moved to the top right of the unit, next to the START logo.
RESULT: A 65,000% increase in clicks, but by the time the test finished, the improvement was almost 264,000%.
THOUGHTS: Don’t question why, just make the change. This was such an incredible increase in clicks the change was implemented permanently.

5. CHANGE: The thumbnail for the top blog roll story went from an abstract image to an image with a human figure in it. RESULT: The art with the human figure in it performed 11% better than the abstract image.
THOUGHTS: Perhaps the human eye is more attracted to images containing human figures.

6. CHANGE: “App Store Update: April 26,” the headline for the 2nd blog roll story was rewritten as, “Jetpack Joyride Tops Mobile App List.”
RESULT: The jetpack took off with a 50%-plus higher click-through rates.
THOUGHTS: Snappier copy does make a difference. IGN has seen headline changes boost click-through rates as much as 150%.

7. CHANGE: “Sign Up For Email Updates>>” became “Get Free Exclusive Content>>”
RESULT: More than a 34% increase in click-through-rate, with a 31% increase in subscriptions through those link clicks.
THOUGHTS: People love free and exclusive. Just as with the YouTube subscription button test, this change was so positive that it was implemented permanently.

Tags: , , ,
Back to top

Test Everything: Notes on the A/B Revolution

In an A/B world something either works or it doesn't. Photo: shellygrrl/Flickr

Welcome, guinea pigs. Because if you’ve spent any time using the web today — and if you’re reading this, that’s a safe bet — you’ve most likely already been an unwitting subject in what’s called an A/B test. It’s the practice of performing real-time experiments on a site’s live traffic, showing different content and formatting to different users and observing which performs better.

Though it came into its own on the World Wide Web, the idea of A/B testing predates it, going back at least as far as catalog mailers and infomercials. In those metric-poor times, different phone numbers or discount codes could be shown onscreen or be printed on an insert as a way to track the allure of one pitch versus another. This data was a big step towards solving the age-old marketer’s bane (“half of my budget is wasted; I just don’t know which half”), but as a rule, any business insight ended at the point of sale.

If you were a blender company, you knew what made for sales conversions, but you couldn’t know how many people used the blender, at what time, how often, or whether it was for a milkshake or a margarita. On the web, and more recently in smartphone apps, companies are effectively able to monitor each press of the purée button. An app or site developer can know, for instance, exactly how many users are looking at a particular screen or clicking a certain button at a given moment—and often where in the world they’re doing so.

The rise of A/B testing online began around the turn of the millennium with internet titans like Google and Amazon, and in recent years it has been slowly seeping into ever-greater swaths of modern life, having become, now, more or less standard practice from the leanest startups to the biggest political campaigns. The touted “internet of things” concept may, in the next decade, catch the world of physical commerce up to speed with its software counterpart, finally making the purée button report back to corporate HQ.

More than this, though, A/B testing is not simply a best practice — it’s also a way of thinking, and for some, even a philosophy. Once initiated into the A/B ethos, it becomes a lens that starts to color just about everything — not just online — but in the offline world as well.

One Nation, Randomly Divisible for Statistical Significance

“It is one of the happy incidents of the federal system,” wrote Associate Supreme Court Justice Louis D. Brandeis in 1932, “that a single courageous State may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.”

In the realm of politics A/B testing makes an unexpected argument for things like block grants and state, as opposed to federal, power. As Silicon Valley’s A/B devotees can increasingly attest, not everything is best solved by discussion and debate. Differences in the way policy is implemented and issues are addressed at the state level make for a rough 50-way A/B test—yielding empirical data that can often go where partisan thought-experiments, and even debate at its most productive (but nonetheless theoretical) cannot.

Consider, for instance, the relationship between a society’s criminal justice system and its crime rates. A 2009 report from The Pew Center on the States shows that Idaho’s “correctional control” (jail, prison, probation and parole) population increased by 633% from 1982 to 2007, during which time neighbor Utah’s correctional control population increased by only 30%. In 2008, Alabama spent 2.5% of its state general fund on corrections; Michigan spent almost an order of magnitude more: 22.0%. What effect, if any, did such huge differences in policy have on the relative safety of those states? Such inter-state differences allow for a kind of side-by-side analysis that tracking federal data across different time periods doesn’t allow.

Of course, 2007 Idaho and 2007 Utah are different places, with other variables in play beside their correctional policies, and this blunts the impact of the data. A true political A/B test would look at completely co-extensive groups, truly randomly selected—say, by randomly divvying up Social Security Numbers into cohorts and providing different legal outcomes to each.

Here’s one way that could play out. Say (as has too often been the case) my car gets ticketed on street sweeping day: the ticketing officer runs my plates, which show whether I’m in the Restitutive Group or the Punitive Group. If the former, I’m fined the $10 it takes the city to hand-sweep that fifteen-foot section of curb. If the latter, I’m fined the $75 it will take to make me think twice every time I park. Lawmakers would determine the relevant metric (say, recidivism) and would quickly establish, to a scientific certainty, whether the stiffer penalty had the desired effects. Why debate when you can test?

Seemingly absurd notions like this, multiple codes of law operating simultaneously, start to make an uncanny amount of sense once one starts drinking Silicon Valley’s A/B Kool-Aid. Such a world—different permutations of the law in effect for different citizens in the same jurisdiction in the same time—starts to resemble strange speculative-fictional dystopian noirs like China Miéville’s The City & The City. It also starts to resemble the contemporary Web.

The Creative Process and the Slap of Data

A/B testing also casts an odd light on a practice close to home for me personally: writing. During my visit to the offices of all-things-gaming site IGN, I was allowed to try my hand at creating some alternative headline copy for the IGN homepage. I perused the day’s trending stories and found one whose headline seemed a little flat. I concocted an alternative that varied just by a word or two but was, I thought, snappier. Within seconds the test was live on IGN’s traffic, and within minutes the results were clear. My headline bombed.

I had officially been “slapped in the face by data,” as one developer put it: something of a rite of passage for A/B testers. The bigger slap, though, was the realization that my chosen profession was perhaps more quantitative and empirical than I’d imagined.

“It’s your favorite copyeditor,” says IGN co-founder Peer Schneider. “You can’t have an argument with an A/B testing tool like Optimizely, when it shows that more people are reading your content because of the change. There’s no arguing back. Whereas when your copyeditor says it, he’s wrong, right?” This comment stings retroactively, as forty-eight hours later I would cost his company umpteen clicks with my misguided “improvement.”

Conversations like this over the past months have prompted unexpected reflections on my own work. “So, like, how many A/B tests did you guys do when you were deciding the subtitle for your book?” a developer at one startup asked me. All of a sudden I felt the flush of shame. “Uh—none. We just all got together and discussed and picked one.”

“Huh,” said the developer, a look of curiosity and concern on his eyebrows.

Of course, what works for headlines and subtitles doesn’t work for novels, with their 90,000 moving parts. Indeed, developers seemed to treat me with sympathy and pity: As an author, I am expected to periodically disappear for 12 to 18 months and emerge with a massive and nearly finished product, virtually unseen before publication and unalterable afterwards. Its ultimate success or failure won’t be clearly measurable until years after its release, if even within my lifetime. For anyone in a data-driven culture, this is a nightmare scenario. And I confess there are days when I long for the tester’s certainty: the headline or ad-copy writer who takes three cracks at a sentence before 9:30 am, and by quarter of 10 knows once and for all which was best.

Ultimately, though, there are reasons to be grateful that life on the whole remains unamenable to the A/B test. The unholy thing about A/B testing is that it tends to treat users as fungible. Testing ad copy works because man-on-the-street X’s reaction is presumed to be a useful guide to man-on-the-street Y’s reaction. And when you do the test and the statistics are right, it is. But, in the political example, learning that a particular sentencing is excessive comes only after you’ve administered it to real people living real lives.

And as for finding the right words: Many of our most important letters, remarks, decisions, and questions are meant for an audience of one—a population size that admits no sampling. Where it counts the most—in family, in friendship, in love—we are operating by instinct, no A’s, no B’s, flying blind.

Tags: , , ,
Back to top

Mozilla’s Mitchell Baker on Being the Alternative to Microsoft, Google and Apple

Mitchell Baker. Photo: Joi/Flickr

Fourteen years ago, as a lawyer for Netscape, Mitchell Baker created the open source license that made Netscape’s code free. It was a fateful event for both Baker and the web: Baker ended up leading a small skunkworks project called Mozilla that was eventually spun out into a standalone foundation devoted to making the web better generally, and to offering an alternative to Microsoft’s Internet Explorer specifically. With its Firefox browser, Mozilla is now bigger and more influential than ever, and Baker still serves as its chair.

Meanwhile, the open source, open-web spirit of Mozilla lives on in thousands of projects: GitHub, Android development, HTML5 apps and in Firefox itself. Now, with a healthy 25 percent browser share, Firefox is in a fast-paced browser race with Google’s own open source browser and Microsoft’s vastly improved IE9.

Not bad for a scrappy non-profit that had to fight for the hearts, minds and desktops of the world’s computer users, nearly all of whom were deeply controlled by Microsoft’s monopoly.

For her efforts fighting for user software that doesn’t suck, Baker is being inducted into the Internet Society’s Hall of Fame in its inaugural year.

In this interview with Wired, Baker looks back on Firefox’s success and what it meant — and explains why she thinks Mozilla’s new push to create a mobile operating system to rival Apple’s and Google’s matters just as much as Firefox did.

Wired: Firefox 1.0 came out in November 2004. The release had a lot to do with Microsoft’s Internet Explorer not getting any better and being full of popups and malware, but there was also something like a movement aspect to Firefox. What did it feel like? What was that sort of explosion all about?

Mitchell Baker: In in 2001 and 2002 there was no competition in the part of the software that actual human beings touch, i.e., the client; i.e., the browser. In those days the server-side capabilities of the web were growing dramatically and there was a lot of innovation and a lot of things happening. We were on our way to Web 2.0 in those days. We didn’t know it yet. We were just starting to talking about AJAX.

But all of that went through a single client, Internet Explorer, and there was no competition for that. It’s a completely rational economic decision when you own the market and you have 97 to 99 percent marketshare, and you also have 99 percent marketshare of the OS below it, [...] it is a rational economic decision not to compete with yourself and improve your product. Nevertheless, the result for the consumer was an abusive setting.

Consumers were interested in getting to the web and the only way to get there was through this tool, which was insecure — one of the most risky pieces of software you could put on your machine! A vector for all sorts of terrible stuff for which there was no competition. And for which there was no rational economic model for competition.

There was no interest in the venture capital world in funding another browser. Netscape had died trying to fight Microsoft. Who would ever try and compete in that space, especially after the browser had been done away with as a separate product and combined with the operating system? So in that setting, many of us were eager to interact with the web but the only available tool for doing so was low-quality, poor-performing, and a security risk.

Wired: So what was so different about Firefox?

Baker: This particular piece of software — the browser — is really important to the user experience, and it can be radically different.

It doesn’t need to be integrated into a vertical silo. You can actually have some control of it and there are range of possibilities in this browser space that seems silly only because nobody’s done them and nobody else in the system has an interest in doing them.

So that all sounds really abstract and idealist and blah blah blah. But when you actually take those ideas and you build them into Firefox, and you put Firefox in front of people, then they have a great product experience and the stuff we’ve been saying suddenly make sense.

And what we found right after we released Firefox was a wave of mail saying, ‘Well my computer runs so much better. Like everything is better.’ So what was happening was that the user experience of the web was deteriorating but people didn’t really understand that or understand why. It wasn’t until we were able to put the product in front of people, that what we were saying became clear.

Wired: Was that wave coming and you guys just happened to have the right surfboard?

Baker: Mozilla has one foot in the Valley, Silicon Valley product technology, and partly one foot in the social enterprise space. So in the Valley, of course, you need the right product at the right time. And some piece of it is really, really hard work and good vision, and some piece of it is being in the environment and seeing what happens and being nimble. And some piece of it is the right time. So we had all of these.

So that idea of a movement for a better web, that’s what Mozilla is. But that’s not exactly what Netscape founded it to be, but that’s what we made it into.

Wired: In 1998 when the decision was made at Netscape to open-source the Mozilla code base, you wrote the license and then later, Jim Barksdale suggested that you take over that part of Mozilla. At that point were these ideas fully fleshed out or was it a more nebulous and it wasn’t clear what was going to happen with the code base?

Baker: Those ideas were not nearly as well fleshed out, but the set of people who have led and managed Mozilla and participated in Mozilla, have built something so much richer than anybody envisioned in 1998 or ’99.

Netscape was looking for a way to have an alternative in the marketplace to IE and Netscape’s goal of course, was to be a successful company. So their management was very forward-looking and was willing to consider and then implement a pretty radical solution at the time, which was open-source and free software.

But by the time Firefox had come along, we had expanded the set of people who understood that the browser was important and that Firefox really made a difference in internet life to include those who were more on the evangelism and outreach and adoption side. And so for an open source project, we were one of the early projects to go beyond software developers writing code to include evangelism and outreach within consumers

Wired: What sort of changes did we start on the open web besides people generally not being vulnerable to Active X hacks?

Baker: Today open source and open source in browsers was not surprising. But in that era, it was. And it was a radical change. IE6 was a bad experience for consumers, but it was a terrible for developers. Not only it was technically bad, but it was closed and you couldn’t do much with it.

In fact, Tim O’Reilly described Firefox as the “Oxygen for Web 2.0″.

Before that, with IE, everything was secret. All the innovation had to run through the Microsoft technology stack and the Microsoft business stack. And again, this is in particular to Microsoft, some characteristics that are closed systems, you know from top to bottom is controlled by one company has some convenience factor. But it has also some controlling and centralizing factor. That turned out to be not productive on the web and so Firefox broke all that open.

Wired: Can we draw a path from Firefox 1.0 to the HTML5 standard that makes it possible to do so much more in the browser?

Baker: I think HTML5 is one area where Mozilla has done very poorly at actually communicating what we have done. We were have been a central driving for HTML 5.0, long before it was popular — from even before Firefox hit. So that would be 2003, when we were just a new little foundation. We didn’t have a product yet. We were founding members of the HTML5 working group. W3C wasn’t interested in HTML 5.0, and didn’t think that it was necessary. They thought that the semantic web work was more important.

And so basically HTML 5 was not welcome. We were a founding part of the working group, which is what kept HTML5 alive.

The standards piece is very important to us, and we continue to do that. We have been a force in moving video to the web. We’re currently also a force in bringing in a bunch of web APIs, new APIs, so that you can access mobile devices through the web. I say through the browser but it doesn’t need to look or feel like a browser, so that the web has the capabilities to access accelerometers and cameras and all the things that mobile devices have that the old machines don’t.

Wired: We’re talking just a few days after the billion dollar purchase of Instagram, which hardly has a website. This is a little cliche, but there still is this battle between the open web and closed web and HTML5 and apps. Has the open web lost the war, as our cover story declared?

Baker: No. The web has not lost the war. I don’t think the issue is about HTML5 versus apps‚ because a lot of apps use HTML5 and so on. The question really is, there is the interconnected, distributed, broad nature of the web, and the app mode which in some ways is quite different. The web as a platform is the most powerful platform we have ever seen. In the last few years we’ve seen the rise of new, exciting platforms, iOS in particular, and Android. And so now were are in a setting where we don’t just have one platform called the web that people are interested in; we have multiple platforms.

We know that there’s a bunch of things about apps that people really like. They’re lightweight; they’re more focused on a single task; they’re not as broad as a browser and the whole web. You get them on your device and it feels really nice. For developers, there is a little bit of a tradeoff. Some aspects of development are much easier because you are just putting it on a client machine. On the other hand, you are deeply controlled by Apple or Google.

And so, what we are working on at Mozilla is to say, ‘How do we take the power and richness and vitality of the web as a platform and bring it to the apps world?”

When you are describing it to people — again it’s a little bit like the browser‚’Oh! That sounds nice but who would ever do that?’ and ‘The existing platforms are already entrenched and that all sounds a little abstract.’ Well, you know, we have seen that before and it’s hard to imagine, it’s hard to feel that those things could change, but, in fact, that’s what Mozilla is.

Wired: In 2005 you did an interview with Charlie Rose and then he asked you can were you thought you be in like 5 years and you said, ‘I don’t know. Maybe Mozilla or maybe something else,’ then as it turned out it‚ it’s Mozilla. What kept you there?

Baker: The main answer is that the importance of what Mozilla is doing hasn’t changed. The question of whether you or I or any person is actually going to have any control over our Internet life is still very real. It’s even scarier today — we are engaging in such a high degree of self-surveillance.

It is an open question whether we have any control over that. And if so, how and who would want to build that and who is going to do the oddball thing of trying to build that when it’s not exactly clear how do you make money out of it?

I do not want to live in a Big Brother world and I don’t want whether it’s a government and I don’t want to live in it even in a commercial setting even if I get some convenience back.

The web can be more than commercial organization; it can be more than Silicon Valley; it can be global; the web can be us; and you know, I might have the place in this world, too. So, that motivates me a lot. There is still a lot to do and I think the answer has been when I think of about what’s the best place to build the kind of web I want to live in, Mozilla continues to be that place.

Wired: But given that many people see Facebook as mostly their web these days, or spend a lot of time in Foursquare or Instagram‚ are they unhappy as they were in the IE6 days? It seems like a lot of the open-web movement, things like activity streams, have just petered out. People seem to be okay with closed networks.

Baker: I don’t think people are as unhappy as they were. But also keep in mind when we were building Firefox, we didn’t anticipate reaching the kind of marketshare that we did.

Our goal in building Firefox was to provide a better alternative, and ideally, to have enough impact to be able to move the industry. That’s our same goal right now. And so for us, our success criteria are that we offer an alternative; and that enough people use it that it’s viable.

Because at that point, you or I, we can use it or we can say, for this part of my life which is more sensitive than other parts, I have an alternative. I’m perfectly happy with everything that is going on for 90 percent of my life, but there’s 10 percent of it where I am not actually sure that that system works for me. And there is an alternative that’s technically excellent and the rest of the Web will recognize it enough that it works.

And to put in front of people the promise that things can be different‚ not just the promise — the example — because that is when you know how happy people really are. I think this sense that people are totally happy with the way things are, well, it’s not until we actually put a product in front of them and test it and see if that’s true.

Back to top

Kicking Nokia When It’s Down, Class-Action Suit Filed

Windows Phone has yet to reward Nokia investors

Nokia’s legal department is going to have a busy summer. Only a day after Nokia went on the offensive and filed lawsuits against HTC, RIM, and ViewSonic for patent infringements, a dissatisfied shareholder filed a class-action lawsuit against the struggling mobile company.

Robert Chmielinski filed a class-action lawsuit against Nokia and its CEO Stephen Elop and CFO Timo Ihamuotila in U.S. District Court for the Southern District of New York, claiming that the company and its executives misled shareholders. Specifically, the company committed fraud by telling shareholders that the Windows Phone platform would reverse Nokia’s slide in the smartphone market.

But since when did an optimistic outlook on a corporate bet equal fraud? According to the plaintiff, the key is that executives knew they were making misleading statements in order to artificially inflate the price of Nokia securities.

“During the Class Period, defendants had both the motive and opportunity to conduct fraud. They also had actual knowledge of the misleading nature of the statements they made or acted in reckless disregard of the true information known to them at the time,” the suit documents assert.

The lawsuit cites Nokia’s $1.7 billion Q1 earnings loss as evidence that Windows Phone failed to reverse the handset maker’s slide. It further claims that Nokia mismanaged the Lumia 900 release by shipping the device with a glitch, which forced the company to offer a $100 refund to customers.

While it’s true that Nokia suffered significant losses last quarter and did essentially make Lumia 900s free for a short period of time, it’s too early to say that Windows Phone isn’t working for Nokia.

Windows Phone hasn’t paid off for the company yet, but the Lumia line is less than a year old. And as evidenced in the chart above, Nokia’s shares were dropping before the company released its first Windows Phone, the Lumia 710. It will take time for Windows Phone to grow.

“As with any new platform [Windows Phone] needs runway to gather momentum and get all the right content,” Wayne Lam, IHS senior analyst, previously told Wired. “Nokia and Microsoft are no weaklings, they do have assets. We believe that there is a good chemistry there with that partnership, and ultimately long-term Windows Phone will be successful.” Yes, compared to the 35 million iPhones Apple sold in the first quarter of 2012, 2 million Lumia handsets isn’t a wild success, but it took Google’s Android operating system more than a year to gather any real momentum.

Nokia responded to the suit with a brief statement:

“Nokia has become aware of the filing of a securities class action complaint naming Nokia Corporation as a defendant, filed in the US District Court for the Southern District of New York on May 3, 2012. Nokia is reviewing the allegations contained in the complaint and believes that they are without merit. Nokia will defend itself against the complaint.”

Tags: , , , , , ,
Back to top

Evernote Expands Note-Taking Empire By Acquiring iPad App Penultimate

Photo: Ariel Zambelich/Wired

The shopping spree by cash-rich app company Evernote continues with the acquisition of Penultimate. On Monday, Evernote announced it would be acquiring the breathtakingly simple handwriting app for the iPad. A price was not disclosed.

The purchase makes a lot of sense. Evernote is a web and mobile application that lets users save notes and portions of webpages in one place, accessible from a multitude of web-connected devices. Penultimate, which was recently updated for the iPad’s Retina display, already features Evernote integration. Take notes with Penultimate, and they live in Evernote.

For those fans of Penultimate, Evernote CEO Phil Libin promises not to mess with perfection. “I’ve been in love with Penultimate since it first came out,” Libin told Wired. “The magic of the app is its simplicity, and we’re going to keep it as is: really simple, really pure.”

Earlier this month, Evernote raised $70 million in part to fund acquisitions like Penultimate and prep for a eventual public offering. That was on top of a $50 million round less than a year ago. Clearly, raising money is not a problem for Libin, and he’s not shy to spend it.

Last year, Evernote acquired five companies, most notably Skitch, a Mac app that let you take, annotate, and share screenshots. Although they’re not disclosing the financial terms, Penultimate is clearly Evernote’s largest acquisition thus far in terms of popularity and user base: For scope, in March Apple announced Penultimate was the fourth best-selling iPad app of all time.

Like Skitch, Penultimate will continue to exist as its own app, just with the increased manpower and funding of Evernote behind it to speed up progress. Evernote, meanwhile, will begin to incorporate some of “the beautiful digital ink” of Penultimate into its products, Libin said.

“Evernote does a bunch of stuff that’s a very natural technological fit for Penultimate,” said Ben Zotto, the founder of Cocoa Box, the company behind Penultimate. “Evernote has a really deep DNA around handwriting from early in the company’s history.”

With Evernote, Penultimate will be launching on additional platforms in the not-too-distant future, including Android tablets and potentially Windows 8 tablets. Evernote already has a presence on Windows 8 through the developer and consumer previews of the Windows 8 OS.

Tags: , , , , ,
Back to top
« OLDER POSTS