Amazon Blames Generators for Blackout That Crushed Netflix

Amazon said a backup generator failure caused its weekend outage. Photo: Flickr/4nitsirk

Amazon has published a more detailed explanation about the outage that knocked out a number of popular websites on Friday night, including Netflix, Instagram, and Pinterest. The culprit: a 20-minute power outage at a single Northern Virginia data center.

Problems started at 7:24 p.m. PDT when there was a “large voltage spike” on the grid used by two of Amazon’s data centers. When technicians tried to move to backup power, the diesel-powered generators just didn’t work properly at one of the data centers. “The generators started successfully,” Amazon now says, “but each generator independently failed to provide stable voltage as they were brought into service.”

Judging from Amazon’s explanation, the generators may have been powering up, but the switching equipment at the data center didn’t think they were ready for a switchover.

Then, to confuse matters more, the power went back on for a few minutes and then failed again, just three minutes before 8 p.m. Seven minutes later, the data center’s battery backups started to fail.

Then the data center went dark.

Continue Reading “Amazon Blames Generators for Blackout That Crushed Netflix” »

Google Shaman Explains Mysteries of ‘Compute Engine’

The Google Compute Engine was only announced on Thursday. But at least one software developer has already added the logo to his latte. Image: yukop/Flickr

Google started work on the Google Compute Engine over a year and a half ago, and it was all Peter Magnusson could do to keep his mouth shut.

Magnusson is the director of engineering for Compute Engine’s sister service, Google App Engine, and over the past 18 months, as he spoke at various conferences and chatted with various software developers about Google’s place in the world of cloud computing, he couldn’t quite explain how serious the company is about competing with Amazon’s massively popular Elastic Compute Cloud and other commercial services that seek to reinvent the way online applications are built and operated.

Google entered the cloud computing game back in 2008, when it unveiled Google App Engine, a service that lets outside software developers build and host applications atop the same sweeping infrastructure that runs Google’s own web services, such as Google Search and Gmail. Like Amazon’s cloud, this is a way of running online applications without setting up your own data center infrastructure. But it was difficult to tell whether the service was just one of those half-hearted Google experiments that would one day fall by the wayside. Though the service let you automatically accommodate an infinite amount of traffic — or thereabouts — it put tight restrictions on what programmers could and couldn’t do, and this seemed to limit its appeal.

Last fall, Google signaled its intent when it removed the “beta test” tag from Google App Engine and launched Google Cloud Storage, a separate service dedicated to housing large amounts of data. But all the pieces fell into place last week when the company uncloaked Compute Engine, a service that gives developers access to hundreds of thousands of raw virtual machines at a moment’s notice.

“Google Compute Engine gives you Linux virtual machines at Google-scale. You can spin up two VMs or 10,000 VMs,” said Urs Hölzle, the man who oversees Google’s vast infrastructure. “You benefit from the efficiency of Google’s data centers and our decade of experience running them.”

What this means is that developers and businesses can grab a vast amount of processing power and apply it to almost any task they want. Google is not only offering App Engine — a service that lets you build applications without having to worry about raw storage and processing power — it’s also giving you, well, raw storage and processing power. In other words, it’s going head-to-head with Amazon, the undisputed king of commercial cloud services that has long offered such raw resources as well as “higher level” services for building and running massive applications.

“We’re pairing Compute Engine with App Engine,” says Peter Magnusson. “But, increasingly, they will be able to work together.”

Google pioneered the art of the “cloud” infrastructure. But Amazon beat it to the idea of sharing such an infrastructure with the rest of the world. Six years after Amazon first offered its web services to outside developers and businesses, Google is still playing catch-up. But it’s intent on making up that lost ground.

Continue Reading “Google Shaman Explains Mysteries of ‘Compute Engine’” »

The Inside Story of the Extra Second That Crashed the Web

The leap second glitch that brought down several web operations on Saturday night can be traced to the Linux kernel. And it was fixed months ago. Sort of. Photo: foxgrrl/Flickr

When Saturday night’s leap second glitch hit Reddit, Jason Harvey didn’t realize it was the leap second glitch. He thought it was some sort of internet slowdown related to the massive Amazon cloud outage that brought down some of the web’s most popular services less than 24 hours earlier.

“It looked like the network was just moving really poorly,” says Harvey, one of the system administrators who oversee the operation of Reddit, the popular news aggregation and discussion site. “With Amazon going down, a network problem just made sense.”

But after about half an hour, Harvey and his team traced the problem to a group of their own machines running the open source Linux operating system. These servers had almost ground to a halt after failing to properly accommodate the “leap second” that was added to the world’s atomic clocks on Saturday night, as June turned into July.

Depending on how quickly the earth is spinning, the planet’s official time keepers periodically add an extra second to these clocks to keep them in sync with the planet’s rotation. This keeps us from drifting away to a place where sunsets happen in the morning, but it can cause problems with computing systems that plug into these clocks but aren’t quite agile enough to deal with that extra second.

In Reddit’s case, the problem could be traced to a glitch in the Linux kernel, the core of the open source operating system. A Linux subsystem called “hrtimer” — short for high-res timer — got confused by the time change, and suddenly sparked some hyperactivity on those servers, which locked up the machines’ CPUs.

Reddit was just one of several web outfits that were hit by leap second glitches just after midnight Greenwich Mean Time on Saturday, including Gawker Media and Mozilla, and these sorts of problems tend to pop up with every time there’s a leap second adjustment. In January 2009, for instance, the leap second reportedly caused problems with Sun Microsystems’ Solaris operating system and an Oracle software package.

“Almost every time we have a leap second, we find something,” Linux’s creator, Linus Torvalds, tells Wired. “It’s really annoying, because it’s a classic case of code that is basically never run, and thus not tested by users under their normal conditions.”

Continue Reading “The Inside Story of the Extra Second That Crashed the Web” »

Pages: 1 2 View All

Dell Continues Quest for Software Domination

Michael Dell is intent on pushing his company even further into the world of software. Credit: Dell/Flickr

Dell has agreed to purchase Quest Software — a Aliso Viejo, California-based outfit offering tools for managing a wide range of business software — as it continues to shift its operation toward the world of software.

The purchase — for approximately $2.4 billion — was announced on Monday morning.

Quest is something of an IT generalist, providing tools that help manage and monitor everything from databases, applications, and operating systems to virtualization software. Today, it claims over 100,000 customers, and Dell says the purchase will provide particular help with its own mid-sized customers.

Dell began life as a PC maker, but over the past several years, it has morphed into a company that deals with a wide range of business hardware and, yes, software. In February of this year, John Swainson — a former IBM and CA exec — joined Dell to run its new software group, and shortly thereafter, the company purchased SonicWall, a network security and data protection outfit, and AppAssure, which provides security for cloud services and other virtualized operations.

In April, the company bought three different software vendors in four days, including Wyse Technology, a desktop virtualization outfit, and Make Technologies, which helps businesses update legacy applications for use today.

“Given Dell’s thirst toward becoming a full scale IT company, if you’re heading in that direction — and we clearly are — then applications become very very key,” Suresh Vaswani, chairman of Dell India and executive vice president of the company’s services division, recently told Wired.

A Dell spokesman tells Wired that Quest will be “strong strategic fit” with Dell’s software group and provide additional tools for systems management, security, data protection,and workspace management. Quest also develops a portfolio of software for managing Microsoft’s Windows Server operating system, seeking to automate tasks and organize data.

“Quest’s suite of industry-leading software products, highly-talented team members and unique intellectual property will position us well in the largest and fastest growing areas of the software industry,” read a canned statement from Swainson.

Quest has a sales staff of 1,500 and employs about 1,300 software developers, and it pulls in almost $1 billion in annual revenue.

Last year, fellow IT giant HP has tried to completely shift the core of its business from hardware to software, but this backfired, resulting in departure of its CEO, other high-level defections and a sorts of other turmoil. Dell has taken a very different route, choosing to gradually augment its existing business with a wide range of software acquisitions.

Update: This story was updated to include the purchase terms.

Steam Punk Remakes Power Grid With Compressed Air

World’s Most Wired

Steam Punk

Danielle Fong

Danielle Fong wants to reinvent the power grid — using giant tanks of compressed air. Photos: Wired/Ariel Zambelich

Danielle Fong was 12 years old when her mother decided she should go to college. Danielle’s teachers didn’t agree. Though an aptitude test put her above 99 percent of students who had already graduated from high school, her teachers said the move to college would ruin her education. But her mother sent her anyway. “Why would I conceivably put my child through six more years of that bullshit?” remembers Danielle’s mother, Trudy Fong, who was 15 when she herself went to college. “I didn’t bring my kid into the world to have her tortured — and be treated like dirt for being brilliant.”

Little more than a decade later — after graduating from Canada’s Dalhousie University and then dropping out of the Ph.D. program at the Princeton plasma physics lab when she decided academic research was as broken as grade school — Danielle Fong is the chief scientist and co-founder of a company called LightSail Energy. Based in Berkeley, California, this tiny startup is built on an idea that’s as unorthodox as Fong’s education. LightSail aims to store the world’s excess energy in giant tanks of compressed air. The goal is to plug these tanks into wind and solar farms, so that they can squirrel away energy for times when it’s most needed, much like reservoirs store rain water. The wind and the sun are prime sources of renewable energy, but they generate power unpredictably. LightSail’s compressed air tanks, Fong and company say, will make the power grid that much more efficient — and ultimately make the world a greener place.

In 2010, Danielle Fong and LightSail took their compressed air storage idea to the U.S. Department of Energy’s Advanced Research Projects Agency, seeking a grant for their work. The agency turned them away, saying she and her team were unfit to manage a company, that the idea wouldn’t work anyway, and that her air compressor would likely explode. But like her mother, Danielle didn’t listen. Backed by $15 million in funding from green-minded venture capital outfit Khosla Partners and with a team of 32 employees, LightSail is pushing ahead with its plan to reinvent the power grid. Fong believes the potential market for compressed air tanks will exceed $1 trillion over the next 20 years. “People get skittish,” says Fong, who is now all of 24. “If you have your own resources and have a real effort, it doesn’t matter what the rest of the world thinks, in its knee-jerk, fight-or-flight response.”

“If you have your own resources and have a real effort, it doesn’t matter what the rest of the world thinks, in its knee-jerk, fight-or-flight response.”
— Danielle Fong

In a way, Fong is going back to the future. Compressed air tanks have been used to store energy as far back as the late 19th century. They were installed in cities across the globe, from Paris to Birmingham, England to Buenos Aires. Germany has been using the technology for the past 30 years, and a power company in Alabama opened a facility in 1991. The idea is a simple one: If you have a power source — whether it’s gas or coal or renewable sources such as wind — you can use the energy to cram air into a tank. When the air compresses, it heats up, as we all know from high school physics — or just from pumping up a bicycle tire. Then, when you need the energy at some point down the road, this stored heat can be turned back into power. It’s a bit like coiling and releasing a spring. The rub is that you lose power with each transfer, and you lose heat when the air is in storage. Because it’s less than efficient, compressed air storage never caught on in a big way. Current systems often lose more than 50 percent of the power originally put into them, since they use the released energy to run a generator — which only loses more power.

Since the 1700s, scientists have struggled to store energy in more efficient ways, working to refine everything from Galvanic fuel cells to modern-day batteries. The question is always the same: How do we build a system that lets us storage energy and then retrieve almost all of it? But Steve Crane — LightSail’s CEO and a geophysics Ph.D. — says Danielle Fong has cracked at least part of the code. “It’s a little arrogant to put it this way,” he says, “but I think that Danielle has succeeded where Edison and others have failed.” The trick? Fong added water. LightSail’s prototype sprays a dense mist into the compressed air tanks, and this absorbs the heat produced during compression. Water can store heat far more efficiently than air, and with this mist, Fong says, the prototype more easily stores and releases power. It heats up the tanks to temperatures that are only about 10 to 20 degrees warmer than the environment, as opposed to several thousand degrees. The tanks are still pressurized to about 3,000 pounds per square inch — and Fong hopes to increase that amount — but since the power is stored at lower-temperatures, it’s easier to insulate the tanks. Some compressed air storage systems sit deep underground, taking advantage of the earth’s natural insulation, but LightSail’s tanks sit above ground, which is less costly. When you want the heat back, you just reverse the process, spraying the warm water out of the compression tank as the air expands, and it drives a piston to reproduce the power. But in both storing the heat and spitting it out, you need just the right amount of water. LightSail has tested nearly 40 nozzle heads — not to mention various tank designs — in an effort to achieve just the right mix. According to Fong, her system doubles the efficiency of compressed air, from about 35 percent to roughly 70 percent.

Continue reading

Continue Reading “Steam Punk Remakes Power Grid With Compressed Air” »