It just so happens that I have been experimenting with Amazon Elastic Computing Services (EC2), documented in Computing in the Clouds with AWS over at The CEP Blog.  The server over at The UNIX and Linux Forums has been experiencing some very hardware-limited, high load averages recently. We thought we should take a look at moving the forum server up to the clouds.   

Then, a fellow system admin over at the forums suggested that maybe some rogue bots were causing high server loads; so I wrote a one-line command to do a bit of real-time spider hunting in the Apache2 logfiles.  Surprise!  I found there were a number of rogue, hungry spiders that would not follow our robots.txt directive not to crawl the site.   One of the bots was from Russia, one was from China, and another one was from Korea.  There were spiders from places I never heard of, all consuming precious  resources and denying our users!

So, I did what any Linux admin would do. I used iptables to block the networks of these rogue, hungry, spiders (sorry I was not very kind to these cyber creatures).  It probally comes to no surprise at this point in the story that four of the spiders were from the Amazon EC2 cloud.  Here is a sample of the output from iptables -L:

root@www:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all — ec2-67-202-45-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-243-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-197-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-213-0.compute-1.amazonaws.com/24

Well, imagine a not-so-distant future dystopian world where criminals or terrorists want to launch a massive denial-of-service attack against some critical infrastructure, like the root DNS servers, or an attack against major financial institutions, military or e-commerce sites.   

First, the bad guys create an instance of powerful operating system with a malicious network application, they test it, and they place it the cloud (without invoking the instance, paying a very small storage fee, no computing time fee) and they wait.   Then, at the precise moment of their planned attack, they launch 128 instances each with the equivalence of whatever is the mega-platform at the time, and just blast away at their attack target(s).    Even more damaging, they do this from many cloud computing infrastructures.  (Note: The cost of the attack is minimal because the criminals are only charged a few pennies an hour for each running instance and the attack runs an hour or two.)

My experience with cloud computing, which is still maturing, is that cloud computing has great promise for both good and evil.  The very real example of the “spiders from the clouds” is a harmless enough story of folks using a cloud computing infrastructure for web crawling, perhaps hoping to be the next Google billionaires. 

One the other hand, cloud computing brings with it an emerging and growing danger for the misuse of the power of cloud computing infrastructures.   The misuse could be malicious, or accidental, but never-the-less, the danger is real.

What an interesting world we have created!  Would would have ever dreamed 10 years ago that we could be attacked by ……

#include <horror_movie_sounds.mp3>

…. Spiders from the Clouds.

Reprinted by permission from The Attack of the Spiders from the Clouds by Tim Bass, CISSP

Amazon EC2 allows us to rent dedicated servers (instances) on-demand to run applications, such as the forums.  Then we can run and host on EC2 any Linux application; but unlike classic hosting where folks install your application and set up your server for you, Amazon Web Services provide only the infrastructure.

Here are some links about AWS:

• Amazon EC2 Getting Started Guide
• Amazon EC2 Developer Resources
• Frequently Asked Questions for Amazon EC2
• AWS Developer Resource Center
• EC2 Articles and Tutorials
• EC2 Solutions Catalog
• Firefox Extension for Amazon EC2
• EC2 Forum

Maybe you will elevate your event processing application to the clouds?

“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”

Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).

What does this mean?

It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only *if* you manually verify the host SSH key during your initial SSH connection.

OK, but I created my AMI before June 23rd - am I vulnerable?

According to Amazon, yes.  Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI.  The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.

How can I fix my pre-June 23rd AMIs?

Regenerate the SSH host key.  The exact commands will depend on your operating system (hint: ssh-keygen).

Who is to blame?

Either the creators of the original AMI or Amazon - depends how you look at it.  If Amazon created the public AMI then it could be argued they are responsible.  However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).

Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image).   That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?

Just to be clear, its not the fault of SSH - ’secure channels’ require proper key management and the need for unique host keys is well documented.

Are there any mitigating factors?

Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet).  You’ll still want to regenerate the ssh host keys sooner than later.

Is the Amazon environment vulnerable to Man-in-the-middle attacks?

I don’t know.  But that isn’t the real question - is the path between you and your AMI immune to MITM attacks and the answer is most definitely no.  If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-).  From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).

Why Didn’t Amazon Tell Me I’m Vulnerable?  They know from their logs what AMIs I use!

Didn’t they?  Whoops - naughty Amazon :P.

But seriously, Amazon are not responsible for the configuration of the public AMIs you use.  Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.

Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).

Not that I know of.   Right now you have to search forum posts and monitor documentation updates - which is time consuming and makes it easy to miss something.  I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management).   In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security  (for some customers, if they don’t “see it”, it doesn’t “exist”).

A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.

This happened recently to some Amazon S3 customers. There were complaints in the AWS forums about ‘S3 Corruption’. The first post in the forum was recorded at Jun 22, 2008 5:05 PM PDT (although in subsequent posts some people reported emailing Amazon prior to this):

we are having some serious S3 issues.

all data we store on S3 has gone through the same code path for months. starting a couple days ago a small percentage of the objects we are retrieving are not checksumming to the correct values. we hash and store objects by checksum and rehash the objects when we retrieve to ensure there is no data corruption. all the objects we’re having issues with were uploaded at approximately the same time period a few days ago.

we’ve stored 10’s of millions of objects in S3 and never encountered such problems. please let me know ASAP if you have any idea what could be going on here. thanks.

Amazon responded 6 minutes later (!) and started investigating. To troubleshoot they asked customers to email aws@amazon.com with the ‘Bucket-Name and few keys that you believe are having issues’.

Others weighed in reporting similar problems. Amazon provided status updates and on Monday Jun 23rd at 6:10pm PDT, provided the following explanation:

We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted. Based on our investigation with both internal and external customers, the small amount of traffic received by this particular load balancer, and the intermittent nature of the above issue on this one load balancer, this appears to have impacted a very small portion of PUTs during this time frame.

What are some of the takeaways?

• If you are directly using the AWS S3 API, make sure to calculate and send MD5 checksums along with actual data. Check status return codes - an HTTP 400 error code means ’something went wrong’ - respond appropriately.
• If you are relying on 3rd party tools to access S3, be sure to check with your software vendor that they are following the advice from Amazon to use MD5. If they are not then your data can get silently corrupted…
• Downloads, aka HTTP GETs, can also be affected. The thread in the forum continues and questions are asked as to whether the corruption caused by the loadbalancer was affecting both incoming and outgoing traffic. The conclusion was yes. If you are hosting media on S3, and the browser is using partial GET requests (to download in chunks) then the corruption will not be automatically detectable.
• If your business relies on Cloud Storage, are you prepared to wait a 36 hours for a resolution? This isn’t a swipe at Amazon, this is true for any provider. Check your SLA’s, check the trouble ticket resolution times, ask about availability of experts for troubleshooting etc.
• Cloud Providers will increasingly need to instrument their services such that they can ‘early detect’ negative operational events. In this case, Amazon has stated plans to use better logging and analysis to automate detection of unusual error patterns (i.e. anomoly detection).
• This incident - caused by an Amazon malfunctioning loadbalancer - did not make it onto the AWS status page at http://status.aws.amazon.com/. Taking Amazon at face value, this incident only affected a small number of transfers, relative to the total number of S3 transfers. But this begs the question, what level of outage or service problem needs to happen before Amazon will flag the issue on their status page? On a sidenote, based on the timestamps, 31 hours passed between the loadbalancer being taken out of service and Amazon providing the explanation on the forum.
• When Amazon update their S3 API documentation, it would be useful to have entries in the S3 API index for ‘checksum’, ‘MD5′, ‘integrity’ and ‘corruption’.
• Stepping back, will customers hold Cloud Service Providers to a higher standard than their own internal IT teams?

I’m sure there are more takeaways I didn’t cover. What say you?

###

Kudos for the heads-up on the S3 issue goes to my friend and colleague Jason Harper - network supremo and crypto-head. Thanks Jason!

I received an email today from Amazon Web Services (AWS) Support announcing new support offerings. One item that caught my attention is the new Service Health Dashboard.

The dashboard is pretty standard fare - traffic lights to show availability for each Amazon service with a historical view available at the bottom.

This is good and all but where is the security dashboard? I’d like to know their “security service” is operating normally. Are they “hacker safe” ;-)

I can dream right?

Users of salesforce.com are not dreaming when they surf over to trust.salesforce.com. In addition to the - dare I say it - “expected” service availability dashboard, they display recent security alerts to raise awareness.

Thats definitely a “good thing” - as far as it goes.

For larger organisations that already use security metrics to track the effectiveness of their security program, this isn’t going to cut it.

Cloud providers, are you listening?

What security metrics would you expect to see from your cloud provider?