A day in the life of atest failure

I love my job. Fact.

But some days are easier than others. If you’ve been reading my tweets this week, you’ll have sensed a certain frustration.

Don’t worry. None of this is your fault.

But here’s what happened.

Over the weekend it had occurred to me, while I was asleep at about 6am – seriously, this happens – it occurred to me that I’d forgotten to submit my latest test fixes before I merged my branch. This would screw up the branch for the weekend, for the whole team, and delay the deploy on Monday. Bad news.

As soon as I woke up, I popped open the laptop, preemptively apologised to all and sundry – as is the English way – and proceeded the fix the problem. It turned out my fixes only fixed half the problem. And it’s the weekend.

So, I cheated. I knew this code was in a hidden section, and wouldn’t be live when deployed. I could just comment out the tests, ensuring the branch would pass even if the code didn’t work properly.

First thing Monday morning, I returned to the problem. I looked back over the logs to find the faulty test and dutifully kicked off a test run.

And this was the start of my problem.

It passed.

I was a little surprised. There were failures last week, I could see there were failures. Maybe somebody else fixed it. I check the logs. Nope. I run the tests again.

It passed.

I should mention that running the tests isn’t instantaneous. It’s a complex test suite we’re running here. We’re currently migrating our server code away from Ruby (we’re always migrating to faster systems). In the interim, we’re running tests against both systems. The various compilation and startup sequences, combined with the real browser used, and the queues of other developers waiting on the system, gives us an average runtime of about 15 minutes.

It’s optimised to run a suite of tests, rather than individual tests, so both cases take around the same amount of time to run. In fact, we have some excellently comprehensive test suites with thousands of tests, and they can all be run in about 15 mins. Running single tests is still slow, because the task can’t be distributed.

From this point on, I’ll indicate the test timings like this: (15).

I can of course run a test locally. Again, this can take time. Usually less time – let’s say 5 minutes or so. It takes less time because I already have the servers running locally.

So my tests are passing. Usually, a cause for joy. In this case, a worry. Why did they fail last week? I dig out the code from last week, and run the tests (15).

It fails, as expected.

Well, that’s something. I diff the code. And here I win the Goof of the Week award. Prize chump right here. The tests are passing because I commented them out.


I go to lunch and try to apologise to my own brain cells for letting them down.

I kick off a real test run (15). It fails. Good. But weird – I swear these were working once. I test the Ruby code instead (15). It passes.

Interesting. So I’ve found a difference. This is weird because I haven’t actually changed any of the server code. It shouldn’t be possible to be different here.

What’s worse is that we can’t easily test the new code locally. It takes longer. I run them (10). They pass. I run the Ruby tests (5). They pass. I run the new tests again, to be sure it’s not random. Pass (10). Pass (10).

I spike the code full of debugger statements. Run the tests (15). More debugging (15). Add screenshots (15). Everything is the same between systems, except that the new tests are failing.

” is not true”. Thanks, error messages.

Hmm. I look back at the screenshots. While the systems show the same screens, there’s something I didn’t expect. When you’re a new user on Twitter, we show a little helper module at the top of the page. I’d forgotten we did that. It also loads asynchronously.

Aha. Here it is. The new code runs so much faster that this module loads earlier. This explains the difference. I go home, happy that I’ve established the problem.

Next day, I make allowances for the helper module. I pull out the debugging code, happily run the tests (15) one last time, and prep the branch for shipping.

It fails.

Bollocks. I unprep the branch and put the debugging code back in. I run the tests (15).

It fails. Same debugging result as before.

That’s weird. I was so convinced the module was the problem. I hack the code and remove the module entirely. Another run (15).

It passes.

Ok, so it’s the module. No question. I just got the allowances wrong. I tweak them, and run (15). Fail. I add debugging (15). Nothing. And again (15). No.

This is crazypants. I am so not happy.

I try to reproduce locally. Passes (10) every (10) time (10). No amount of debugging and breakpoints will help if I can’t reproduce the error.

I look back to the screenshots. Aha. No images in the module.

I knew the error was caused by an bad height calculation. And the missing images are (whips out an onscreen ruler), yes, the images are 137px – exactly the difference.


Our images don’t load on the testing machines because they’re insulated against the outside network. That makes sense. Locally, we can access the images, which must be why they pass. It’s perfect.

Furthermore, our testing machines are using Firefox, which won’t render broken images, while I’m using Chrome locally – which does. This would account for the height difference.

I curse and laugh, since this quirk of Firefox has bitten me many times in the distant past.

To fix the test, I simply break all the images – to be consistent. I prep the branch for shipping, throw it out to the test cluster one last time, and go and get a fifteenth cup of coffee.

It fails (15).

What. The. Fuck.
What. Thefuck.

How can this be? I had this nailed.

Worse than that, the tests is now passing occasionally. When I run the tests on the test cluster, I run them about sixty times in parallel. This ensures we don’t get “flaky” tests, which sometimes pass and sometimes don’t. Flaky tests are notoriously disruptive and hard to fix.

Sometimes, all you can do is try it yourself. I open a browser. I downgrade it to the exact same version as our test cluster. I’ve done this before, but I’m doing it again. I run the tests (10). I click around. I fiddle with the code. I break the images, to match. Refresh, refresh, refresh.

Hey, what was that?

I refresh again. And again. There it is.

Firefox won’t render a broken image – it hides it away completely. I knew that. But what I’m seeing here is a flash of broken image. There’s a flash of a broken image in this helper module, before Firefox hides it away.

If my code is making calculations just at that moment, it’ll get a different result than, say, half a second later.

I don’t get my hopes up again. I tweak the test. There’s actually a little css property:
This will prevent Firefox from hiding the broken images. I run the tests. All the tests (15).


I prep. I ship. I smile.
I move on to the next thing.

(btw, if testing tools is your kind of thing, and you want to help me make things faster, we’re hiring and I will buy you cookies)

Waiting for Bill

I throw another log onto the fire. I’m not sure we really need it – it’ll be warm enough all night long. Out here in Palm Springs, it’s only bearable when the sun sets anyway. The fire simply gives us something to watch while we wait.

Bill Gates is due for dinner. Most of the other guests have already arrived, so he should be next. Or the one after.

I poke the fire. Sparks fly. It’s been a few good hours now, and I still haven’t decided.

Do I shake his hand, or kick him in the fork?

This was about ten years ago. I’d never suggest or condone violence in any form these days. Hell, even then I wouldn’t really. So let’s moderate the language.

Do I shake his hand heartily, welcome him like an old friend, thank him for all he’s done for my career, and propose my next big idea? Or do I ignore him, snub him, turn the other cheek?

I should clarify my position. I don’t have dinner with billionaires. That doesn’t happen. I’m really not that kind of guy.

I’m here on holiday. Vacation, if you will. I’m visiting an old friend from London, who is originally from Perth, who is currently working as a hot air balloon pilot in Palm Desert. And in the evenings, he works as a parking valet at The Reserve, an exclusive golf club in Palm Springs. And, since I’ve got nothing else to do, I’m here to help.

I say “parking valet” to emphasise that I’m using the American meaning of “valet”. Someone who parks your car. Not the English meaning, which you might be familiar with from Downton Abbey, of someone who dresses you in the morning.

To make this slightly more bizarre, I can’t actually drive. No license. Never even sat behind the wheel. I’m opening the car door, welcoming them, opening the restaurant door, that kind of thing. Oh, and I get to drive the golf cart, ferrying my fellow valets back from the car park. This is a nice place – you expect your valet to be fast. This is a classy place – even the valets have a little nook with a fireplace.

When I arrived, they told me Bill Gates was booked for dinner. Yes, that Bill Gates. That one.

Obviously I was somewhat in awe. He was, I believe, the richest man in the world at the time. And I’d been working with his software for years. My job, my life, was founded upon it.

And yet, that brought so much trauma. I was out here for a much-needed break. I’d been fighting poorly implemented APIs, buggy applications, incomprehensible error codes, absent or irrelevant documentation, and that blasted comedy paperclip. In no small way did I hate this software as much as I loved it. I should snub this man for all the suffering he put me through.

But what a wasted opportunity. I should be pitching an idea. I’ve got maybe 20 seconds with the most powerful man in technology. I’ve got to plant a thought in his head, one that will distract him over dinner, so that he’ll emerge with a warm smile and an offer of partnership.

All I needed now was the idea.

Ideas are funny things. After a few pints, or several sleepless hours, your mind is full of amazing, wonderful, life-changing, world-changing, revolutionary ideas. When you’re put on the spot, nothing.

Fast forward to the future, and hindsight brings the answers. I could’ve pitched Twitter. Facebook. Myspace. The cloud. Groupon. Well, maybe not Groupon. Instagram. The iPhone. I could’ve begged him to continue development of IE.

If you were put on the spot, right here, right now – could you pitch? What’s your idea? Are you ready?

I wasn’t ready, but that was ok. Bill never showed up.

The other guys were fine with it. Apparently he’s not a big tipper.

Since then of course, he’s donated most of his wealth to charity. Seems like a generous tip to me. I’d heartily shake his hand. Or fist bump. Or whatever he’d prefer.

And he tweets. I build the software he uses every day. That’s pretty awesome.

Maybe it’s a good thing he didn’t show.

Apple ideas

Here are my predictions for Apple products this year.

No, hang on. Not predictions. More like hopes and desires. Wishes.

1) Embeddable products.

I have just not bought a Honda CRV. I’ve been driving one as a zipcar, and I love how it drives. We need a family car, so it’s an obvious choice.

And then I saw the screen on the website.

Screen Shot 2013-02-16 at 19.13.10

No way. It’s 2013. I’m not using a computer system from 1980. I’m just not.

You should be able to slide in an iPad mini, and have it control all the car’s features. This shouldn’t be exclusive to luxury cars. This should be across all cars, all brands, all prices.

The same is true for aeroplanes. We took a Virgin America flight a few months ago. One of the seats was empty (which was awesome), but why? The entertainment didn’t work. If they’d used embedded iPads, they could have just switched it out. And we’d have apps. And the games wouldn’t be shit. And they’d be interactive adverts for Apple products, with a 100% captive audience.

2) Touchscreen laptops

Ok, I get it. A touchpad works better on a laptop. But, having used a Surface for a bit, I’d still say a touchscreen is useful.

It’s useful when pairing to be able to say “that word there”, or “push this button”.

It’s useful to scroll, sometimes, when reading. Your right thumb just flicks the content down. Feels natural.

And they seem to be both ubiquitous and inexpensive. So why not?

3) A radical design change for iOS

I have an original iPod Touch. The iPhone wasn’t released in the UK for about a year after the US, but the Touch was. I’m still blown away by the completely unnecessary brilliance of the device. It’s so slim, so solid. The screen is amazing. The touchscreen was unlike anything anyone had seen before. It had an accelerometer, even though there was NO USE FOR IT until the App Store arrived. It still works.

But it looks exactly the same as the new ones. I know great design is forever, but is forever really longer than five years?

4) iTunes

Just break the videos out of iTunes. Call it iVideo or iView or something. It doesn’t make sense where it is.

While we’re there, let me opt into iCloud fully. I *do not want* to download everything. Ok? Ok.

5) TV

Jus do something amazing. Typing using a remote control is madness. Give me Kinect.

Don’t give me Siri. I don’t want to shout at my TV. Seriously, the tech pundits are idiots. Don’t go there. Let it go. Put it in a watch, where I don’t have to care about it.

6) Buy Sonos

Simply to acknowledge that their system is awesome in a way that Apple showed us was possible. And it needs iTunes.

That’s it. I’m happy with the products as they stand. They don’t need another iPad/iPhone/iPod idea. They make money. We love the stuff. Keep it going.

Oh, and I’d love a phone the size of an iPod nano. Now that would be amazing.

2012: This year I have…

My annual End of Year update. See previous years: 201120102009 and 2008.

This year we have:

  • Survived.

Last year I said I’d take it easy this year. That’s not so simple with a baby. But I think we’re slowly getting on top of things.

I have also worked solidly on twitter.com, making it faster and more robust. I’m giving the New Hire Orientation talk for frontend. I’ve got the hang of how things work. I interviewed eighty people. We moved offices. We got a roof garden and a proper kitchen. Twitter is all grown up.

We went to Texas, our first real trip with the baby. And we made it.

We had a weekend in Monterey.

Next year, I plan to buy a car. A new car. My first new car.

And I’d like to go home to England for a few weeks. See some old friends and new family.


I’m English.

England is vast. Despite holding about fifty million people, most of the land is still dedicated to farming, and enjoys large exports to the rest of the world.

We complain about the variable weather that always comes with island life, but it gives us acres of fertile soils. From the air, the land looks terraformed: neat rectangles of colour.

Fifty million people is also a huge populace, mostly living in cities. The cities are vast and sprawling, with thousands of years of history behind them. Your local church can be a thousand years old as easily as a hundred. You can travel from town to town, city to city, with barely a break in housing.

An hour’s drive is a good trip. In ten hours, you can cross the country, even the long way. Rail travel is good, expensive and sometimes unreliable, but generally good. Motorway travel is excellent. Speeds are fast, accidents are infrequent.

Wherever you travel, you are surrounded by people. At most, you could be half an hour from people, a few miles from people.

I live in America.

American cities feel like English cities. There’s that mix of all-comers, the blend of cultures thrown together. Buildings, housing, stacked closely together. People rubbing shoulders on the bus.

Further out, to the suburbs, there is still very little difference. The houses are generally a little bigger. They’re less likely to fit the upstairs/downstairs template of English life. They’ll have proper garages. They’re made of wood. The urban sprawl surrounds the cities, giving families some breathing space and distance from the bustling city nightlife.

Further out still, there is more difference.

Because America is absolutely, staggeringly, incomprehensibly, mindbendingly vast.

Looking out of the window at the Rockies as you travel here, you can see the breathtaking landscape still raw and scarred, like the surface of Mars. It’s enormous, it’s beautiful, and it’s empty.

Travelling from city to city, it’s easy to forget this. Close the window, close your eyes, and move seamlessly from one identikit airport to the next.

But out there, people live. They live off the land, they live different lives. Farm fields roll over the horizon, without limit. Farmers have planes.

How far can you be from other people? How far can you be from help? How far from the nearest mobile phone tower?

If your family lives on a farm, miles from anywhere, how can you feel safe? Who will be the next person to drive by? If trouble came knocking, how long before anyone came to ask if you are ok?

England has a wonderful position of security.

America is different.

I don’t support the gun lobby. I believe that fewer guns mean fewer deaths. But I’m disappointed in polarity of argument, and the lack of imagination in recent commentary. One should try to see both sides of the picture to find where to draw the line.

I’m not making a point one way or the other. I’m saying: think.
Use your imagination.

Common courtesy

Remember at the end of the 19th century, when we all got together and agreed to stop bowing and curtseying and saying “good heavens, whatever is the matter child?”

That was progress.

I know we’re more than a dozen years into the 21st century now, but it’s not too late. We can make a little progress of our own. Can we all just agree to stop acting out bollocks door-holding courtesies like anyone gives a shit?

It may be ok, if you’re out on a date. Open the door with a flourish. This is fine, don’t get me wrong. But it has no place in an office or on the street. Stop giving way – you are actually getting in everybody’s way.

And besides, everybody knows the only reason you’re doing this is to check out their arse. Definitely not an office thing.

It’s not even about women. I was in a lift the other day with seven other men. And somehow I had to push my way past all seven to get off, even though it was the last stop and they all had to get off too. Why? Because they were all trying too hard to give way to everyone else.

No, they weren’t looking at my arse. Shut up. They weren’t.

You know that awkward moment when someone’s holding the door for you, and you have to skip ahead at half a run, just to get there in time? Who is this helping? I am a grown up. I can open a door. Let. It. Go.

Every time there’s a fire alarm, you’ll see the insanity. As we all crowd through those emergency exits, the guy in front will hold the door for you. When you get there, you take it off them. But that’s not enough – he’ll want to hold it for the guy behind you too. And even the next guy behind is already reaching over you to try to hold it for you. And before you know it, your hands all end up touching, and then nobody can ever look each other in the eye again.

So, from this day forth, hold no doors, except for the following allowances:

  • person behind holding something with both hands
  • person behind in a wheelchair, or on crutches, or conspicuously pregnant
  • the door happens to be one of those truly lethal fast-swinging ones
  • person behind less than ten years old
  • you’re on a date
  • your front door.

As an additional note, that person who always shouts “there’s room for one more” in the lift, when there are plenty more lifts and honestly no room, shall henceforth be committed to the lift shaft without remorse.

White guys

I hate white guy jokes. I find them deeply offensive. I don’t understand why it’s considered ok to group “white guys” when the same jokes about “black guys” would probably get you fired.

Because they are the same. No, that’s not true. They’re worse. When people say “black guys”, it no longer means uneducated poor people. It’s changed, and usually refers to people who dress a certain way, act a certain way.

But “white guys” is back to the old definition. When you make “white guy” jokes, you’re playing on the stereotype of white guys being wealthy, educated and family oriented, and by your definition “black guys” or anyone else, is not.

The more you say these things, the more the stereotype is enforced.

I’m disappointed that in the recent election, it was only the democrats bleating about “white guys” for Romney. Why do that? I thought you guys were supposed to be above that kind of thing.

Don’t say “white guys”. Don’t tell “white people” jokes. Don’t tell me that’s “the whitest thing you’ve ever said”. It’s offensive to white people, black people, asian people, australasian people, gay people, straight people, tall people, small people, average people, weird people, male people, female people, all the people. We the people.

Rise above it.


Back in secondary school… No, wait, I’m in America. What do you call it? High school? Something like that? Ok.

Back in High School, I studied English Literature. Not by choice, you understand. I was compelled.

It was absolutely excruciating. Never has anything destroyed my interest in a subject quite like English Lit. I remember that we spent a whole term – that’s three months – on Act 1, Scene 1 of Macbeth.

Maybe my teacher felt overly familiar with that chapter, being something of an old crone herself. But we read it, we analysed it, we interpreted it, we acted it, we watched the movie, we went to the play (ok, the whole play, not just the first act) by the RSC. God, that was dull.

But still not as dull as Great Expectations. A book whose ironic title is the only wit to be found. I *hated* being forced to read this. I’d sneak a Pratchett book into class to read instead. How ridiculous that I had to sneak a book into an English lesson.

But despite hating Lit with a passion, I found myself becoming rather good at it. My secret was in finding that there were no limits. You could take the piss as far as it would flow.

I remember we were analysing a rather harmless poem about picking blackberries. My family and I do this every year down by the river. It has fond memories for me.

“What’s the poem about?”
“Blackberry picking”
“Go on.”
“What was the author intending?”
“What are the blackberries symbolising?”
“I dunno. Life?”
Really? Ok then, “the nature of the berry signifies the promise of the summer fruit, of the joys of life, and while it is fleeting and momentary, there are plenty more to be found, until the season, much like life itself, eventually comes to a close.”

I owned English Lit from that point onwards. My reinterpretation of the hundred words of Act 1 Scene 1 ran on for seventeen pages. Front and back.

So I aced Lit, and I found it useful later in life. In coursework assignments, it’s easy to bulk out an essay with rambling nonsense. Later still, I could knock up a system design specification in about a week. Two hundred pages (big headings). Happy client.

Trouble is, with all that fluff, there’s precious little content.

It is actually not at all a good skill to have.

Recently, a couple of years back, I changed this habit. I decided not to write any more documents that weren’t expressly for the purpose of conveying information. No more bulk, no more fluff. Get to the point, cut it back even further, redraft again for clarity, and then kill any words that aren’t absolutely needed.

It’s harder. Writing less takes longer.

The same is true of speaking. In a meeting, how often do you ramble on? Are you explaining what’s on the slide? Are you saying the same thing you said last week? Are you pointing out something that anyone smarter than a grapefruit already knows? Or something nobody cares about?

It often arises in interviews. You’re nervous, you’re keen to appear knowledgable. So you start waffling. I’ve cut my interviewing back to coding problems on a computer, because waffling wastes so much time. I won’t rank you down for waffling, I’ll criticise myself for not cutting you off.

And of course, I’m terrible at waffling in meetings. I’ll be trying to clarify something in my own head, by explaining again, something I already said last week. Or yesterday. Or five minutes ago.

So stop me. But don’t embarrass me. We’ll need a secret signal between ourselves.

When I’m rambling, just start checking your tweets on your iPhone. Preferably with a large bored sigh. I’ll stop.

And of course, I’ll do the same for you.

Lets do this.


I have a problem with spelling.

It’s a bit like anti-dyslexia. I’ve seen friends stare at a page of text, totally unable to see the glaring errors in front of them. I don’t have that. I get the opposite.

When I look at a page of text, errors jump out at me. They’re distracting. I’m not trying to proofread, not trying to find problems. It’s them. They find me.

Of course, I’m as vulnerable as anyone else when it comes to text I’ve written. It’s hard to proofread your own stuff.

But it’s bad enough that I find it a problem to read on the Kindle. As soon as I see an error, I’m pulled out of the book. The carefully constructed fantasy world is jerked out from under me. I want to feed back to the publisher, so I think about how I could do that. I can’t.

I’ve taken to just highlighting the word. Then I move on. For the rest of the chapter, I’m not reading, I’m just checking words for errors. The next chapter, I can start reading again. But I’m not back into the fantasy world for another three chapters or so.

So I bought this book recently, written by an ex-colleague. It’s good – it’s really good. I want to write back and say I love how he sets the scene, how he builds his characters through their actions rather than simply describing what they’re wearing. How he doesn’t fall for the new-author fate of rewriting the first three chapters straight from a thesaurus, just to appear intelligent; but at the same time, I love the simple choice of words. “Raucous chowder” is a wonderful way to describe Fisherman’s Wharf.

I’d say this, and more, but I wouldn’t be able to say it without adding the soul-destroying “but”. He doesn’t want to hear the “but”. Nobody cares about the “but”.

But there’s a rogue “if” instead of “it” in paragraph eight. And it’s killing me.

And nobody needs to know that.

So maybe I’ll just be quiet.

Exclusive hands-on review – one year in

Here’s my one-year review.

Obviously we’d had this product on order for a long time, and there was a lot of expectations. Would it live up to the hype? In the end it was lovely to see it arrive, right on time. The unboxing was something of a complicated process, and we had some concerns that the product would not fit back in the wrapper if we had to return it. But it does not come with any kind of warranty, so that may not be a problem. The packaging itself is beautiful, and well worth hanging on to.

From first glance, this product is stunning. It is clearly more attractive than all the other models on the market. Home run. The attention to detail is superb, nice to see some familiar features in there, and you just want to pick it up and play immediately.

On the down side, the preinstalled operating system was not really ready for prime time. It would go into sleep mode every hour or so, and require constant recharging. The charger is built into the packaging, which seems inconvenient.

We’re still waiting on the promised voice commands. Waiting a whole year for this obvious feature is frustrating, and I feel this should have been built in when shipped. The inbuilt apps are also not much use. However, we’ve seen more and more available as the product has matured, and each new one is a delight to install.

We are still having a number of problems with sleep mode, which seems to have a mind of its own. I think some sort of hardware switch would be more useful. Equally, the alarm can go off at all times of night, which hasn’t helped with scheduling.

While the initial outlay for the product was very low, it ships with a binding multi-year contract, and the monthly fees are very high. Be very sure this is what you want when placing your order. Accessories are also expensive, if you want to look the part.

We’ve tried a little networking, which seems to be going well. I can get the devices almost talking to each other, but we haven’t completed a full handshake yet, let alone transferred any meaningful data.

Despite this, viruses can still be easily caught, though you can get some anti-virus plugins. They are painful to install, but apparently worthwhile, and the minor viruses we’ve seen have been shortlived.

Ultimately, like any luxury good, this is a product requires significant customisation and time, but will grow with you. I think it’s well worth the investment of time and money, and I look forward to playing with it for years to come.

10 / 10