Ok, so the headline of this article is probably a little OTT. But I’ve just had one of the most annoying, and expensive, experiences working with Amazon’s hosting.
Actually, it’s really only annoying because it was expensive.
What happened was this: I moved a sizeable set of sites over from my previous standard hosting to the Amazon EC2 set. It was LAMP based, but I’m not going to tell you which site, because it was built a long time ago and hasn’t had enough TLC recently.
The site was suffering from some pretty terrible response times. Using a top to find the slow process seemed to give me no clues at all, and yet the cloudwatch service was showing consistently high CPU usage.
The site is database driven, and that work is quite intensive. Each page on the site executes several queries, and there is a chatroom which drums the database on a regular basis. But the site doesn’t do much else. And so my culprit was obvious: the database.
I’d read about Amazon RDS and it seemed the perfect solution. There’s no denying that my database is relational, it’s almost the textbook relational example, and so RDS beat Simple DB any day of the week. It was cool to have my own EC2 instance running MySQL, but really, Amazon can be a bit complicated with backups and storage. The thought of carefree backups, and a consistent long-running database as I toggled between application servers definitely appealed. I have a new site to roll out? Easy. I run a new server in parallel, and switch the IP across when it’s tested. Dream scenario, right?
And so it seemed. The cost seemed a little high, but I had high hopes for the performance, and it was still less than our (3 year old) prior hosting. I followed the Amazon online setup guide together with someone’s step-by-step tutorial (since sadly the console did not support RDS yet).
Except that it wasn’t much faster. “Shit”, I thought. That was a waste of time. But that’s ok – I now can toggle between different instances of application server, and figure out the problem there without worrying about the database at all. The site was fast enough to leave it running for a week to “bed in”.
One week later, a spring in my step, time to get working on the application. This is working out well. Database still intact, Cloudwatch says CPU load still high, so the blame must be with the PHP not the MySQL. Ok, fair enough. And a quick check of the running costs, gives me $400, not bad for the …
No, wait, go back to that bit again.
Four hundred dollars? A WEEK?
WHAT. THE. DUCK.
That’s over twenty grand a year. Sucre.
My fingers have rarely moved as fast as they did to get that server switched back.
So what happened?
It turned out that I’d been charged for the bandwidth between the app server and the database. And in that week, I’d racked up a terabyte of data. That sounds like a lot, but like I say, this was a DB-heavy site. And actually, you never really look at the bandwidth to your DB do you? It’s not what I would consider “external bandwidth”, which is what Amazon charge for.
After a very stressful couple of hours, I had a medium size instance (high cpu) running, with a clean Ubuntu install and Apache PHP MySQL on top. Performance was back to proper levels (so the PHP just needed another processor), and the costs were back down.
I sent a message to Amazon asking what had happened, and whether I could have my money back. Answer: not on your nelly. You used the bandwidth, you pay the price.
I protest. “Come on”, I say, “I’m not sure what happened here”. I’m a loyal customer and they could at least be generous on this. A bit of faith in the newly-subscribed. Better than that, I paid up and reserved an instance to demostrate my commitment. Just drop the week’s charges, I asked.
My question was passed to management. Not on your nelly, they said. You used the bandwidth, you pay the price. Your bandwidth was cross continent, they say.
AHHHHHHHHH.
Now I get it. My servers were naturally in Europe. Obviously I’d omitted the step of putting my RDS instance in Europe too, and it had defaulted to the US. Bollocks. I moved 1TB of data across the ocean because of a silly oversight.
Of course, it would’ve been nice if the setup commands had mentioned this.
It would’ve been nice if they had added RDS to the console, so I could see this (seriously, they can develop a distributed database system, but haven’t got the time for a bit of HTML?)
It would’ve been much better if they’d alerted me as I switched from a $1,000 setup to a $20,000 setup.
Not much to ask.
And really, it’s not much to ask for my $400 back. Ok, so I got it wrong. But where’s the love?