What is Amazon EBS for?

• Using Amazon EBS
  • Yahoo! BCP
    • The Basics
    • Data Safety
• What is EBS for?

Using Amazon EBS

I was recently sent a link to Adrian Cockroft's blog entry Understanding and using Amazon EBS, along with a request for comment. In case of TL;DR, the blog posting in question is a lengthy analysis of how EBS appears to work (reverse engineered, since Amazon are tight-lipped about it), and how to work around that. This was of interest because various observers have noted unpredictable latency spikes, and no one can have failed to notice the outage in March 2011 that affected Reddit and many others.

For some time, I've been surprised at all the attention this gets, because I don't see why most of these users are using EBS at all.

For context, I'm talking about people who run web sites. I learnt a lot of technique for running websites 24x7 from my stints at Yahoo!. And Yahoo! doesn't use EBS. So what does Yahoo! do?

Yahoo! BCP

Yahoo! has rules for what is known internally as "BCP," or Business Continuity Planning. These dictate minimum deployments to guard against single points of failure, and ensure that Yahoo! properties stay up all the time.

The Basics

A property must have a presence in at least two geographically distinct colos (aka co-location facilities, or data centers). The colos are registered with an external geo-load-balancing DNS system. If one colo goes dark (because of a power failure or a network problem), the external DNS service stops returning that colo's A records. Service continues uninterrupted from other colo(s).

Within a colo, the number of servers for a service must be sufficient to handle the load if another colo becomes unavailable. In other words, in a two colo setup, each colo must have enough servers to handle the entire load.
Under normal operation, these are only at 50% capacity. In a three colo setup, each colo must have enough servers to handle 50% of the load; that is the amount they would have to handle if one of the three colos became unavailable.

In addition to this, whenever you calculate the number of servers required, you must add one. This is to account for a single server failing.

Data Safety

For a long time, most of Yahoo!'s infrastructure relied on MySQL. (This is beginning to shift over to NoSQL solutions, but they have the same general characteristics for the purposes of this discussion.) In order to guard against data loss, MySQL databases are replicated. This fits the above rules well; the servers are slaves.

If you put the above rules together, you can see that the minimum MySQL deployment is four servers: two in each of two colos. (If you only needed one server to meet the demand, you would have to have one more than that, and you need to have the same number in a second colo for a two colo setup.) One of those servers will be the master. Data is considered safe, because it is replicated to three other machines besides the master, two of which are in a remote location. There is no EBS.

So, having seen that, I'm always puzzled by web sites that are running in EC2, are spread across multiple availability zones, and are having issues because of EBS. Why are they using EBS at all?

If your data is replicated to at least three servers in at least two geographically distinct locations, why do you need EBS? Note that the choice of data storage technology isn't really an issue here. Most of the current NoSQL offerings offer replication capabilities that will achieve this just like MySQL does.

A couple of weeks ago, I was at the CassandraSF 2011 Summit, and I noted that Netflix have also come to the same conclusion. Adrian mentioned that Netflix is now using the instance stores on their VMs for storage (this came up on slide 26 of this deck.) He mentioned that because the data is replicated, he's not worried. The data is replicated to multiple remote locations, so it can be recovered and new storage cluster members can be added if an availability zone disappears off the map.

Note that if you read the EC2 documentation carefully, you'll find that rebooting an instance does not wipe out the instance store. Only explicit termination of the instance does that. But termination can happen for two reasons: you explicitly shut it down, or the instance actually dies because of a hardware failure. These are what you need to guard against. In the Yahoo! BCP case, replication covers it. There isn't any reason that shouldn't work in EC2 as well.

What is EBS for?

Having been doubtful of EBS' utility for some time, I've asked myself, "Why did Amazon build EBS?" Amazon aren't dummies, and they must have some reason for needing this themselves. After all, it was available right from the start when EC2 became available.

Put another way, I seek to reverse engineer the requirements that led to EBS' development.

Given that EC2 was the first virtual hosting to be available on an hourly basis, rather than monthly, and given how EBS works, I suspect the whole ecosystem came into being to meet internal needs that are transient.

For example, suppose you are a merchant who tracks your customers' buying patterns. There are certain things you might want to do periodically. You might have a weekly batch job that runs your recommendation engine. Based on the results, you blast out emails to your customers with the message "Here are some new items we think you might be interested in." But you don't want to keep all of the infrastructure you need for this up and running all the time -- that costs money, and the system would probably have a very low average utilization.

What if I told you you could have such a system on a temporary basis? Create your databases on a networked filesystem, and fire up the processes to do your periodic processing. When you're done, unmount the databases' file
systems, and shut everything down until you're ready to do it again. Meanwhile, I'll give you a low price for that, because when you're not using all of that, I'll use it for something else.

EBS sure sounds like a perfect fit for that, doesn't it? That's my best guess as to why they built it. It's not about data safety, as everyone seems to think; it's about transient usage.

Of course, if you're running a system where your data is not preserved via replication, you probably still want to use EBS. But if you are replicating your data to several other servers that are always up, what do you need EBS for?