CSRF stories

I think I first learned about [CSRF](https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)) at an "@media" conference talk in 2008 given by Simon Willison ([Twitter](https://twitter.com/simonw)). In the middle of a Django presentation, he explained how it's possible to make a form that will post across to a different website, and that when you do so, the appropriate login details are also attached. This lets one website do _anything it likes_ with the user's credentials on another website. "This," he explained, "should make you think 'oh shit'". "Oh shit" I thought. The solution to this problem is a csrf "token". The idea is that you add a new field to all the forms on your website. The field is prefilled with the token: something that only your own website would know. So Simon's website has an evil form that posts { text: "I like marmite" }. But all the forms on your website post { token: 'secret', text: "I like vegemite" }. And you check the token before you accept the data. This works perfectly, and it's left to the developers to figure out a way to generate, distribute and check the secret. At Twitter when I joined we were using Ruby-on-Rails. It had a common library providing the form token, which I think it called "form_authenticity_token". However, the project we were working on was the "Follow button": a button that we wanted people to place on a good percentage of the entire internet's web pages. The traffic would be immense, and our Ruby-on-Rails system couldn't handle the load of serving those pages with their tokens. Our solution should have been Scala, as is well known, but at the time our first Scala service wasn't ready for prime-time, so we only had Rails to work with. So we generated the token on the client. None of us had tried that before, so we were a little dubious, but we ran it past our security team who seemed satisfied by the idea. For clientside CSRF, we just generate a random string using JavaScript and append it to both the form AND the cookie. While Simon's evil form can submit any form fields it likes, it's unable to set or read cookies from another site. This worked well for numerous years. More recently our security team moved our primary website off csrf tokens and onto Origin headers. These were more recently supported in browsers, and let you know the original domain of a request. If the origin is "twitter.com" then we can be pretty sure the request came from Twitter. This worked well until recently, when we received a bug report that the csrf protection was failing. This was hard to believe - nothing had changed for years in that code, and all of our focus was on the new website. Sure enough, the new website was the cause of the problem. The new website installs a serviceworker that intercepts requests, which means that it can cache pages for offline or performance reasons. Sure enough, it was intercepting Simon's evil form post and proxying it back to the server. Since it intercepted the request and issued a new one, it added the origin header for the serviceworker itself, completely defeating our old website's origin header csrf protection. The solution was simple: check the [mode](https://developer.mozilla.org/en-US/docs/Web/API/Request/mode) of the request before proxying it. I've recently been playing with CSRF again for the new website. The new website depends on a CSRF cookie which is provided by the server. If the CSRF cookie expires (it has a limited duration) the API requests will fail, but they will issue a new cookie as they do so, and our code would retry the request. Something I'd noticed is that the retried requests would often still fail, which should be impossible. Cookies are inherently complicated to handle. While the concept is simple, the api in the browser is not. There are volumes I could write on the inadequacies of the model. Happily, there are some beginnings of an idea to improve them, but it will likely take years to roll out. A lot of the problems stem from legitimate attempts at privacy protection: a browser or extension might hide or block a cookie it considers unworthy. Sadly this can interfere with security protections like CSRF, and there's no way for developers to detect or override them. After a few failed attempts to crack the problem without success, I turned back to clientside token generation, something I hadn't looked at for 6 or 7 years. Is it still a valid option? I found that I could easily generate a random string that would be accepted by our serverside CSRF validation. In code review, my colleagues asked whether a Math.random-based solution would be sufficiently secure? My initial answer was that it shouldn't matter - this isn't a login credential, it's just a random string. Upon investigation I discovered this confidence was misplaced - in several browsers it's possible to reconstruct the state machine of the random number generator, and by asking for a few random numbers you can perfectly duplicate the machine to be able to predict the next number to be generated. Mind blown. I switched my random string generator over to [window.crypto](https://developer.mozilla.org/en-US/docs/Web/API/Window/crypto), a new feature for generating cryptographically secure numbers. In production, this seems to work. When I find the cookie is missing or expired, I simply generate a new token and cookie and the requests succeed. While I struggled to reproduce the original retry-failure, I could mock up responses using [Charles proxy](https://www.charlesproxy.com/), and observed the drop in errors when my code was deployed. https://twitter.com/wongmjane/status/1151676723064295424 There was a spike in new errors, seemingly only from Xiaomi phones and tablets - they don't support window.crypto. I now let them fall back to the original retry logic. We'll still see some errors, but significantly fewer than before. If you find a CSRF issue on the site, please do report it responsibly. The best place is [hackerone.com/twitter](https://hackerone.com/twitter), where there's also the chance of rewards and recognition.

Read

2018: What I did this year

My annual review of the past year. One thing I said I needed to do last year was work on my shoulders. I get a lot of pain in my shoulders when I sleep. I can’t sleep on my right side because my right shoulder just starts hurting, I can’t sleep on my left side because my right shoulder starts hurting, so I’m sleeping on my back which means I’m snoring (and sometimes my shoulder hurts). I looked into getting a new mattress. We’ve had this one since we got engaged about ten years ago, so we’re due. But, we have a British bed, and sizes here are different from there, and there’s quirky weird laws around exporting mattresses, so it’s problematic. Haven’t done it. So I opted for some physio instead. I got my shoulder MRI’ed, which showed nothing, and the doc recommended physio, but that’s too complicated to set up. So I’ve just been doing some pushups before bed each night. And sometimes lifting some small (12lb) weights. It has really helped! I can feel the effects when I lie down, and there’s no pain in the morning. I’m proud of myself for keeping that up all year. Another thing I said I’d do is stop getting older. But, sure as clockwork, I turned 40, so that’s a fail. I’ve been trying to keep a little fitter, but I’m getting no younger. To celebrate the occasion, I invited the folks from work round to my house and we had an awkward social event in the sweltering 100F heat of Walnut Creek for my birthday. It was good. Speaking of work, I’ve settled into being an IC. Coding is just much easier than working with people and I need easy right now. I’ve got this feeling that I’d like to switch back into management at some point, but it’s not imminent. I’d rather improve my coding skills for now. Last year I started off working on Settings. My thinking was that they’re the unglamorous part of the website and as we rewrite and refactor the website they’re the most likely to be forgotten. I did a lot of work to assess what needed fixing, both from the API and the website, and build out what was needed. The next big project was GDPR compliance, and the settings work turned out to be essential for that effort, so it turned out really well. The new API endpoints I’d made could be used by the native mobile clients as well, so now (finally) you can set all your settings on all the clients! I also worked on a privacy/security problem during this time, for which I also wrote a blog post on the twitter engineering blog. That was a fun project to get into and learn about. For another GDPR project, I was asked to look at cookies across the company. As a long-time employee, I figured I could use my historical knowledge here. We had a huge log of 600k different cookies that were sent to Twitter, which needed classification and ownership. We managed to find owners and roll out the allowlist filter at the front of the service without any incident. It was surprisingly smooth. A small thing I was proud to do last year was Text Size. As the team worked on various accessibility fixes, I realised that I was struggling to read the text on the site myself. My eyes are pretty good - I’m just getting old. So I added a text size option. I’m not sure how many other people are using it, but personally I find it invaluable. 🤓 Enough work We took a month off and went to England! We stayed with my parents in their new house in the sticks; we journeyed up to the cousins in Melton Mowbray, we took a short trip to Poland, we went to London to see the Harry Potter play, we went to Waldringfield to mess about in boats and camp in tents. It was a lot! Probably too much. It was only our third trip back since we moved here eight years ago, and the temptation to do everything is too strong. I’d like to do the Waldringfield part again, maybe just with Jack, because it’s a part of my childhood I really value. We bought a new car. The old one was only five years old, but it was having battery issues and getting tetchy. The car had electrical issues from day one, and honestly I’m happy to see the back of it. We looked at different options and settled on something similar from a different manufacturer. The biggest shock of last year was losing friends. I found out from tweets that Cindy had died. It was the kind of shock that just stops you dead. I couldn’t keep working - I just went home to tell my wife. We met Cindy through Matt when we first moved out to SF, they were incredibly friendly and helpful - we still use their leftover cutlery. We’ve not been in touch for years, so didn’t know she was sick. I still can’t quite come to terms with it. And then we lost Dave Goody. He ran the coffee shop in Fort Mason, which we stopped at every day when Jack was a baby. The place was never too busy, so he always had time to give us a warm welcome and a chat. He marked Jack’s height on the wall as he grew. We’ll always remember him fondly. And we also discovered that my favourite great aunt had terminal cancer too! What is happening? I thought this kind of thing wouldn’t hit for another 20 years or so. I thought we were still in the “everyone’s having babies” phase that follows the “everyone’s getting married” phase. I miss that phase. Next year: - take Jack back to London - try some new things - write some blog posts - work out if nine years at Twitter is something I really want to do O_O

Read

Blog Update

Ah, shit. It’s the end of May. I’d meant to write a blog post at around the end of last year, when I provide my typical end-of-year update. But I didn’t. I should do that. I had this smart idea of trying one new thing every day for each month. I knew that one of those new things would be blogging, so I delayed 😱 my end-of-year post until then. But it hasn’t happened yet. First month started well. I did Keto. The next month I needed something a little easier, so I got myself a watch and decided to wear a watch every day. The third month I was considering the blog posts, but I got really sick at the start - so I decided not to shave for the month, which required no effort. Negative effort really. And then it was April, and I still wasn’t ready to blog, and I didn’t have the energy to do anything else either (sick kids and wife can really take it out of you), so all of a sudden it’s the end of May and I’ve got nowhere. Bah. Topics I was considering: - What I did this year (2018) - Refactoring Twitter. Some things I’ve noticed over my 8+ years working there. - Thoughts on the trend from internet companies to give you what they say is good for you, rather than what you want. - A (probable) rant about how tests both set you free and hold you back. - Some thoughts on customer service for small teams within large companies. - Monorepos and source dependencies - a simple summary of what that means and why. - Some thoughts on skills and expertise of leaders. None of these are particularly well formed in my head, but they’ve been running around in there for a year or three. It’ll be good to set them down in type. I write this blog more to compose my own thoughts than to provide reading material for others, and I’ll continue to do that. _Update:_ Ah shit, it’s August already. Time to publish this thing.

Read

Microservices and websites

You build a startup quickly, scale the tech as far as it goes. You’ve got one codebase, and one scary-as-hell deploy. You’re known for failure at least as well as you’re known for your product. So you break up your service into microservices. Each can be deployed independently. Each can be owned by a team who understands the whole of the thing, and so the changes therein. Responsibility and accountability are delegated to those teams. It’s a proven pattern. And very successful. At Twitter, we run dozens of microservices. We have macaw-users, serving the user information apis. We have macaw-tweets, serving tweets. We have macaw-timelines, serving timelines. It all makes a lot of sense. The tweets service can roll out a deploy, observe issues with their endpoints, roll back and fix - all without bothering the other services. This is an ideal way to scale an api. Your mobile apps can use this api. Maybe with JSON, or something more efficient. If an endpoint fails now, a single team is paged and the service recovers. Users might see a momentary error with some actions. The mobile app teams needn’t even know; they worry about their own code deploys, not the reliability of their apis. A website is a different animal. One web page is the combined output from many of your apis, many of those services. And for speed of delivery, you are rendering those web pages on the server. For server-side rendered web pages, the output of your apis is combined inside your own datacenters. This is happening on your own servers. The microservice model starts to fail. If a service you depend upon is experiencing issues, your web pages are seeing that issue and throwing error pages themselves. Your service may fall below SLA before theirs does. Suddenly, you’re a single point of failure, and so primary oncall for all microservices. Farewell sleep. It gets worse. Since your service is inside the DC, you use a more efficient api layer with compile-time dependencies. When you compile your project, you need to build every other service. Of course your build system has caching, but what does that matter when every backend change forces a recompile on you? It gets worse. The other services start to focus on mobile. They build business logic into the json formatter. It’s not as crazy as it sounds - of course they use shared objects for json. And checks at the edge are the simplest. But now you’re missing them on web. And, oh, resources are tight. Build it yourself or miss out. It gets worse. Your integration test suite needs fixture data in code. But the mobile clients just depend on manual QA and json fixtures. Effective integration testing falls away, and deploys gets even more scary. Development grinds to a halt. This is where we were. Let me tell you where we went, how we made things better. We shipped a PWA. A progressive web app. The app is entirely rendered in JavaScript in the browser. We use the same apis as the mobile clients. We’re a mobile client. We can write just one language: JavaScript. We can write, deploy, test and be oncall for our own app, and stop worrying about others. We can use the new api formats and features as soon as they’re ready. We did it. Or did we? Another route we could have taken is to rethink this from the perspective of the user. To the user it doesn’t matter whether an api call failed in the client or the server. A failure is catchable and can be well handled. We had an SLA on the server because we could easily measure it. Should that be different on the client where we can’t? There’s no doubt in my mind that our service could have been coded more defensively, mitigating the vast majority of pages and alerts. So why did we not? Partly because our service isn’t built on a robust web framework. It’s a migration of our ruby-on-rails system to Scala. This was necessary for the complexity of a migration while operating at scale. The framework is optimised towards api development, which again makes sense, as most services are apis. It’s logical that our graphs, monitoring and alerts are also configured this way. APIs are not websites. What’s more, it’s because websites are built with static assets (JavaScript, CSS) that don’t split well against service-based concerns. So our website could not be fragmented into micro web services without some mismatch in the asset versions delivered by the various services as they deploy. And finally it’s because web engineers have a tricky enough job writing good quality modular scalable robust crossbrowser user experiences. We should be allowed our focus and not have to worry about the intricacies of server code in a foreign language. Try hiring a CSS developer with Scala skills. I’m sure Google has some. Good luck. So we went PWA and we’re not going back. Was it necessary? Maybe, maybe not. Was it worth it? Hell yes. I shall sleep well tonight.

Read

Granny

The first time my mother-in-law came to stay with us, she was sleeping on the floor. It was awful. Not just on the floor, but on the living room floor. We were living in a small one-bedroom flat in San Francisco, we hadn’t been there long, and the move had cost us most of our meagre savings. She’d lie there and watch me eat dinner. It’s hard to move away from your family. San Francisco is about six thousand miles from my home. I have all kinds of family back home: parents, brothers, aunts, uncles, cousins. My wife has far fewer. I don’t know if that makes it easier for her, or harder. Anyway, we don’t see people often. It’s a little sad. Kids just make the problem worse. We can’t just nip home for the holidays now. We need four plane tickets. We need car seats, luggage, and a week to cope with jet lag. Tantrums are frequent. Vomit is inevitable. We’ve been home three times. Three times in eight years. Luckily, we get visitors far more often. My brothers have been really good about visiting. My parents too. And of course, my mother-in-law. The thing is, nobody is going to visit just for dinner. Nobody flies six thousand miles to just watch the kids while I pop out for a beer help yourself to whatever’s in the fridge thanks love. No. They need to stay for a few weeks. Just to make it worthwhile. And a few weeks in a hotel is kind of pricey. Even in an AirBnB. It’s more than the cost of the flight. So she’s lying on the floor, watching me eat my dinner. For weeks. It’s not just that. You see, I’m pretty sure she’s a vampire. No, really. It’s not just that she calls people “darlingk”. She once went to a playground and announced, in her thick Polish accent, “Look at aall the lovvely cheeldren!”, and I swear she licked her lips. She never seems to eat anything but ice cream, which suggests that food isn’t her main source of nutrition. She’s at least a thousand years old. She never seems to sleep. And when she goes outside in the daytime her eyes glow red and she begins to disintegrate. Ok, I made that last one up. It’s really intense having family over to stay for a long period. It’s also really intense going to stay with them. You go from the occasional monthly phone call with nothing to say, to good morning good afternoon good evening good night. IN YOUR FACE, TWENTY FOUR SEVEN. Ugh, it’s unbearable. I even get frustrated with my mum, and she’s the nicest and least offensive person on the planet. Anyway so we got this new place with a granny flat. We didn’t mean to. It’s just a place that had a beautiful backyard. And the granny flat became a playroom for the kids, and an office for dad (me), and yes we realise those two concepts don’t colocate brilliantly, but our schedules mostly don’t match and it’s fine. But for this past visit, we’ve made it a granny flat. It’s granny’s flat. Sometimes (ok, every day) we send the kids up there to play with granny in her playroom. I set her up a sofa and television. It’s like she’s living next door, it’s not like she’s IN MY FACE, watching me eat. She visits us every day too. And it’s been brilliant. She’s been brilliant. She can come to stay any time. (Not any time of course, visas don’t actually work that way, which is either a shame or probably a good thing, because you don’t want to push it do you?) Any time though. Welcome, Granny.

Read

A day in the life, January 2018

Five thirty. The baby is making noises. My wife murmurs something I can’t hear while I stretch. It’s my turn. I get up and go to the cot. He sees me and instantly turns away. He wanted mummy not daddy. After five minutes of trying to sleep, he realises he’s awake. I pick him up, take him to the chair and rock him on my shoulder. After half an hour, he gets quiet asleep. I put him back in the cot and wait the thirty seconds until he wakes again. My wife says he needs food and goes to get him some. I wait. She wasn’t going to get him some, she was turning on the heating and going back to bed. I take him to the living room where the baby bottles are. I take him back and feed him. I get him down to sleep. It’s six thirty. At seven, the alarm rings. I turn it off, then take five more minutes. I get up again, head for the kitchen to make tea and boil eggs. I turn the boy’s light on, to get him used to the idea of morning. Halfway through my tea, my wife staggers in. She’s been getting up four times a night for sixteen months now, and with the recent teething it’s been worse. She’s got a bad cough that won’t go away. She starts to say something, but the baby monitor is louder: he’s awake. She goes to get him. He’s groggy but up. I take the baby and go to wake the boy. He’s reasonably amenable, especially when reminded there’s oreos and milk waiting for him (his daily nutritious breakfast). I tell him he can watch iPad if he gets his clothes on first. He does. Ten minutes later, they’re getting ready to leave for school. I sniff something on the baby as I’m putting him in the car seat, so I whisk him off to clean up the poo. The boy is being packaged up ready. I deliver the baby to the car, and head back inside for a moment of peace. My morning can begin. Shower. Brush teeth. Clothes. Check bag. Into the truck. Drive two slow, boring miles to the station. Park the truck. Into the station. Five minutes for the bart. Not bad. Bart seat is only mildly smelly. Not bad. Into the city in forty five mins. Reading Armada on kindle, not very good. No visible poop or visible needles in the station, surprising. A wet walk to the office. Drizzle. I’d normally head up to nine for some eggs, but I did have one of the boiled eggs earlier, and I have a suspicion there’ll be doughnuts. I head for coffee on five. Wash hands, latte, glass of water. Head to my desk, unpack the laptop. What on earth was I doing yesterday? Today I needed to get a real answer for a question for auditors. I ask someone, but I know that some of what they’re saying isn’t true. I find someone else, we verify our answer. I stick it in the sheet, all good. I’ve got a few branches to land from yesterday. Landing branches is slow, and some of them need the tests to be rerun for certainty (I’d spent much of the previous day unblocking the test pipeline). It’ll take a couple of hours. I head to the other building where the team is running a hackday. We need to convert every file from CSS files to JavaScript styles. It’s a lot of tedious work, so it’s best done as a team, in a day, in a room, with doughnuts. There’s a prize for the most files converted. I’m not here for the prize, I’m here for doughnuts. I have a doughnut. I pick up a ticket and work on it while landing branches. Twelve o’clock and we need to get out of the room. It’s basically impossible to book a good meeting room for a whole day. We head to lunch. I’m doing tacos, because they’re always ok, and I don’t mind boring. I often have tacos. I sit with the team until they bring up D&D. I go back to work. I keep tabs on a login issue. I think it’s more complex than they realise. I have a suspicion that an error page or asset is pointing to the wrong site and logging us out. They haven’t figured it out yet. I’ll keep watching. Another trivial ticket I’ve been watching hits 100 comments without real progress. It’s time to go rogue. I remove the disputed image from the page altogether, and then remove the links to the page itself. That’ll solve the problem, but it’s two more branches to land. Two more sets of tests to run. It’s four thirty. I’ve had an extra brownie and some sweets to keep me going, but I’m still feeling sleepy. I dive into an api migration I needed to get a handle on this week. I can’t get it to work, I chat to the team who make the api. They give me a repro case. Feels brittle, but it’s something to work with tomorrow. Five o’clock, and it’s time to run. Wife texted me earlier, she’s going to the doctor at seven thirty for her cough. I need to get home. Drizzle is wetter. Back to the Bart. Bit smellier this time. Fine though. I read the book but can’t get into it. Back to the truck. Driving through the rain. Stop at home, bring the bins in, grab the mail. Open the door. “Daddy!” The boy comes running for a hug. He hasn’t done this since he was three, but he started again this week and I love it. The baby is excited and doesn’t know what to do with himself. He hands me the mail I put down on the bench while I drop of my jacket and bag, then follows me to the bedroom to change trousers and wash hands and face. He gets a big hug. There’s no food ready or anything to really make, so I reheat some dried-up pizza from the weekend. I eat while reading tweets. The baby is shouting “mama, mama” at me, like he does. He’s smelly again, so I go and cleanup the poo again. A little has leaked, so my wife gives him a bath. I make pasta for the boy, my wife leaves for the doctor, I sit the baby down and try to feed him food from a pouch. He has a little. I tell the boy to turn off the iPad, and we have a genuinely pleasant few moments, us three, at the table. We head upstairs to the playroom. We play James Bond. I’m the bad guy spy. The baby climbs on furniture to turn on the disco lights like he always does. We play until eight. Let’s go down. Mummy texts to say the doctor was good, she’s coming home. Put your pyjamas on. Put your pyjamas on. Put your pyjamas on. And so forth. Teeth are brushed, rinsed. A wee is had. Pyjamas are on. I take the boy in to read, but we don’t read, we play Space Mazes. It’s eight forty. Up to bed, lights off. Quiet. Quiet. The baby is still playing outside the door. I sit with the boy until he sleeps. It’s nine thirty. He should sleep soon, and then I can steal his hot water bottle and go to bed too. Sometimes I have some decaf tea before I do, dipping some chocolate biscuits into it. Not today though. Doughnuts were had. Good night.

Read

2017

The end of the year once again. Last year, I vaguely remember feeling quite upbeat. Although I apparently had a cold, and yes, Trump had been elected and Brexit had been voted for and yadda yadda yadda. From a personal perspective, I was a new dad, I’d just had a huge break from work, I was going to the gym regularly, and the baby was basically just sleeping and eating, so I had plenty of time to Netflix and chill. As babies get bigger, they become more work. You can’t let them out of sight for a moment, they’ll toddle off somewhere and start eating cables, or pulling things off tables, or climbing on pianos, or falling down staircases. They’re full of boundless energy, excited to be alive, and very cute as well. It’s exhausting. When I look back to my end-of-year assessment for the similar timeframe of the older boy, I see it says “we survived”. I feel similarly. So, I must keep on top of the energetic baby and the rambunctious six year old and the wife who wakes up four times a night to feed and to hold, with never a moment off (no babysitters, no nearby relatives - we do things the hard way). In consequence, I quit the MBA course I was taking, I stopped going to the gym, I stopped running, I stopped watching much Netflix, and I haven’t chilled for more than about five minutes straight all year. When I returned to work at the start of 2017, I decided to re-think my job. I had two prime motivations: one, I loved the heart and soul that my stand-in was bringing to the team and knew I couldn’t match it when my mind was on babies; two, I wanted to get involved in the abuse-handling efforts as much as possible. So, a week after returning, I handed off my team and scouted around the company for new roles. I offered myself as an EM to the abuse efforts, but it was one role they said they didn’t need filling. I considered becoming a PM for some web-related efforts, but that role is very different and I didn’t feel I’d be accepted by the team for that position yet. Ultimately, I realised that the abuse efforts were going to be long-term problems (possibly forever) rather than short-term fixes, and the best I could contribute would be to help push for a migration to a better stack, allowing us to make those changes more easily. I was also in a good position to make rapid fixes to the existing stack for any quick fix work. And so, once again, I’m an IC. (Individual contributor) It’s been fun. This year I’ve done some legal fixes (ok those weren’t fun), a lot of settings fixes, a world of bug fixes, and my favourite project was working with one of our interns to deliver Nightmode on desktop. It’s been the most positively received project I’ve ever seen. My Intern, David, also overdelivered. He threw in some code to make sure your colours in nightmode and daymode provide enough contrast to be visible, using proper accessibility-based contrast maths. Much respect. I’d always planned to hop back to management again, but I’m currently reconsidering. It’s felt good to have colleagues again, to hang out and have fun with people who don’t report directly to you. Management can be a lonely job. We’ll see. I stopped going to the gym because I just can’t find the time. If I cut time from work, then I’d need to make it up. If I cut time from home, then I get less time with the kids and my wife will go crazy. If they could only put a gym car on Bart, then it’d be doable. Anyway, my shoulders are killing me, and I can’t sleep comfortably, so in 2018 I’ll have to find time somewhere. We went to see the total eclipse! It was AMAZING! Like a hole in the sky! If you’ve only seen a partial eclipse, even 99.9% partial, then you just haven’t seen it. You need totality. We flew up to Oregon for the day with the kids for it. We had my parents visit for a couple of weeks. We had my mother-in-law visit for a couple of months. It is incredibly important to balance enough time to help and get to know the kids, with the need to not kill each other. The new (bigger) house helps. Mostly helps. This year the brewery closed up. I’m still pleased I invested in a friend (to help him open), that operated honestly, and made some really very good quality beers. But sometimes you build it and they don’t come. I’ll miss the money I invested (which was significant), but I’ve learned some lessons (should I ever get any more money for investment). I didn’t invest in Bitcoin, which hit nearly $20k this year. If I’d invested some years back, I could be a multimillionaire, which riles me slightly. BUT, I didn’t invest because I’m an immigrant and didn’t want to be accused of money-laundering (which was a concern at the time), AND I didn’t have much money back then, AND if I’d bought then I’d have used Mt Gox, which would’ve lost my investment, AND if I’d had any I would have sold at $10, at $100, at $200, at $500 and so on. NO WAY would I have the guts to hold. I feel much better about losing my money to the brewery than I would have about losing money to Bitcoin. This year: * Total eclipse. Wow. * Hawaii again (yay) with mother-in-law. * New baby is full of beans and friendly as anything * The boy started learning the alphabet for real in kindergarten in September. Now he’s reading Dr Seuss to himself. It’s fantastic. * My wife continues to be beautiful and indefatigable. And patient. * Twitter is still the best employer I’ve had (7 years now). * California is still better (sunnier) than London. Next year (2018): * I need to settle my job. Decide if I’m management or code. Or both. Or something else. * I need to fix my shoulders. Seriously, I can only sleep on my back, and when I do that I lose feeling in my hands. It’s ridiculous. * With a zero-ending birthday impending, I need to stop getting older by \*any means necessary\*. It’s gone on too long. * I need to go home. I want to visit the friends that remember me, any family that still recognise me, and see the places we used to be. * I need to blog more. I have some ideas. * I need to work on my skills more. Instead of just doing work, I need to invest some time in growing my skills. Maybe I can link that to the blog thing. * I qualify to apply to be a US citizen.

Read

The last great-grandparent

When Jack was born, he had five great-grandparents. That's five more than I had. But by the time Max was born, only one was still with us. And now, he too has passed away. Hamish was probably my favourite. He was a sailor, a conscientious objector, a dancer and a fool. He taught me to sail, a beautiful little sailboat called Sally Brown on the river Deben. By the time he passed it onto us, it was probably the only sailboat without an engine left on the river. He was teetotal, and wouldn't even enter the pub. He didn't fight in the war, and suffered for it. I would love to have learned more about this time and this choice. The most I heard was "why should I go and kill some foreign johnny I don't have any disagreement with?" He worked as a manual labourer. He reckoned he could carry a ton of grain. He did ten long-arm pull-ups each morning until his doctor told him that wasn't normal for a 75 year old, and that he should probably stop. He was a fool, a dancer dressed in scraps of many colours in a Morris dancing troupe. I never saw him fool, but I've seen a tape. As a tall man, he gave it dignity. He was the second cousin of Hugh Laurie. He was a printer, producing flyers and cards on his cast iron manual printing press hidden in his garage. Granny would draw the pictures and cut the lino. They were married for more than 60years. As I remember him, he was retired with Granny in Waldringfield, where we keep the boats. They didn't have a television. They didn't eat out. They lived by the wind and tide, the Archers, their friends, and that story Hamish would tell about eating turkey stock instead of apple sauce in his porridge. I miss summers on the boats. The boats are all still there, even little Percy, but we've moved away.

Read

2016

Ok, it’s the end of the year. I’ve got a cold. I hate having colds - they attack me particularly badly, and I get all gross and covered in tissues and snot, and all I want to do is sit around and eat ice-cream and… Hang on. … Ok. Ice cream. Better. So, 2016. One of those years when lots of things happen. We started the year in London. I spent a couple of days back at my old university in Warwick on a management course. I came back to work all charged up and ready to fix everything. Didn’t manage to fix everything of course. Some things are more complicated than that. But we made progress. In happy news, the brewery opened and Randy is successfully producing excellent beers. It makes me very happy to have invested in a real tangible company, making high quality product. The brewery has become my local establishment, for my rare evenings off. We had another baby! We make very good babies. This proves that the first was not a fluke. They are not easy to make or maintain, but are a great investment. One thing Twitter did this year was change their leave policy. Now, both parents can take 20 weeks paid leave in the first year. That’s been absolutely incredible. Not only has it helped maintain my wife’s sanity through the early months, it has helped my five-year-old feel like he’s not been usurped. This time is probably the most time I will ever have with my family. It has been amazing, and I’m very thankful to Twitter for their support. In November, I clocked 6 years with them. Best place I’ve ever worked. Britain voted to leave Europe. That was something of a surprise, but also kindof understandable. It doesn’t mean the country is full of racists, it just means the politicians have failed to connect with the electorate. Most Brits don’t vote for their European representative (MEP), nor have any idea who they are or what they do. When votes became inconvenient (see Lisbon treaty), the politicians simply avoided them. It is no surprise to me that voters felt disenfranchised. I hope they figure something out that works better. America voted to elect Trump. That was something of a surprise. It doesn’t mean the country is full of racists, it just means these politicians have failed to connect with the electorate too. There’s an anti-establishment vibe. I’m not as pessimistic about this one as other people are, because a) I felt the way you feel now about Bush winning in 2004 and the world didn’t end then (ok, hundreds of thousands of people died, but the world didn’t end); and b) I don’t think Trump (whose opinions are unknown and variable) is as bad as many republicans (whose opinions are known and repugnant). We literally have no idea how this will go. I talk about this stuff occasionally on Twitter or this blog and never really make my positions clear. I’m happy to correct bullshit from either side (tbh lies about Trump are as annoying as lies about Clinton), and maybe that causes confusion, so here are my stances: Democrat(US)/Labour(UK) but not as far as the communist hippy crowd. I believe in universal healthcare, but I’m not opinionated about implementation. I support LGBTQ rights. I support diversity in all its forms, and acknowledge that I need to do more. I want more women in tech. I'm against bullying, online and offline. I think immigration is a complex issue without simple solutions, but I think employers looking for H1Bs should be required to provide US scholarships. I support public transport, but I love my truck too. I believe there are no gods. I'm pro-choice in the first two trimesters. I prefer Chrome to Safari, but I'm not entirely sure why. I'm enjoying craft beers, and bored with wine. I like vegetables, because they're what my food eats. I think Snowden should be exonerated, and I think Manning should only be released due to mistreatment before trial. I think gun control is a good idea when the laws make sense. I don't understand term limits for the presidency, because hey, why not choose the best person for the job? Ok, I think that's enough. Oh, we bought a house. I love it. It's old and knobbly and makes no sense, just like us. It's perfect. We have deer and turkeys and squirrels and trees and grass. We went to Tahoe for Christmas. The drive was terrible and we all got the flu. So, yeah, Merry Christmas. I ran a half-marathon! But I hadn't done any training at all, and it pissed it down all the way, and there was a big hill so I had to walk a bit, well a lot really. It wasn't much fun. I have been going to the gym a bit. Hopefully I'll keep this going into the new year as I go back to work. It's been good to stretch out. Last year I got wisdom teeth done. This year, gum grafts. They're really not fun. They tear a strip of gum off the top of your mouth and stitch it onto your teeth. Bleah. And there will be more of that to look forward to next year. I watched the entirety of Friends. I'm not sure if this is a worthy accomplishment, but when you're feeding the baby for 30mins every 2 or 3 hours, you need something to watch. I also developed a thrombotic hemorrhoid by sitting on my arse all day, which was not fun. And I think that's the year. Next year, get back to work. Do something awesome.

Read

What I love about working at Twitter

I've never been the one with the fancy words and clever insights. My writing is either hastily-scribbled thoughts to myself, or simple language and simple meaning to convey a message. If you're looking for the poetic literature on this subject, I recommend my colleague Jeremy's Medium post, "[Why I Love Twitter](https://medium.com/@jdrishel/why-i-love-twitter-83085dbef4c1)". Twitter is the first company I've worked for whose product I use every day. I've never had that before, and I never realised how significant that is, before this job. It's incredibly satisfying to work on a website you depend on yourself. It becomes a labour of love. The Twitter site always appears to be very simple. There's reliably someone on HackerNews each week claiming they could build it in a day, and before I came here, that could well have been me. Since joining, I've learned of the tremendous complexity that underlies essential services, like Twitter, Facebook and Google, to keep them running 24/7. Why is 24/7 just hours and days? It's not enough. Every second of every year, you expect Twitter to be there when you ask for it. And at scale! We serve _hundreds of millions_ of users! Our Google Analytics numbers are just ridiculous to browse through. (I've wasted hours in those dashboards - hours). I love working at scale. After 16 years building websites, I've learned that the most fun work is _realtime_, _scale_, and _user-focused_. The kids say they love Machine Learning and Big Data these days, but they're honestly missing out. I love the breakneck pace of developing for users, the tension of stability, and the challenges of scale. When I read the news (BBC), nearly every story mentions Twitter or a Tweet. Everything. Politicians making announcements, or campaigning, or discussing foreign affairs. Celebrities doing whatever it is they do. Every single obituary. It's the perfect way to get a short soundbite for a story. It's the source of news on the ground. It's a direct line to the people who matter. I love that we have this impact. Nearly everyone, whether they use it or not, has an opinion on Twitter. You should do this, you should do that. Sometimes we agree, and there's a technical constraint. Sometimes we agree, and we just haven't got time for that yet. Sometimes we agree and it gets done! I love being on the inside and being able to sway those priorities. This is, after all, a product I use every day. For me, one of the perks of the job is little tweaks. I'm not a coder any more, I'm just a manager. But that doesn't stop me making the _slightest_ tweak here, the _slightest_ tweak there. Just when I've got a free afternoon. Maybe I'll speed up the page load. Maybe I'll add another column. It all gets approved by others. There's nothing underhand. And it's all small, but I love that power. When I joined Twitter, I felt incredibly unworthy. I assumed the coders there would be gods, writing these books and building these sites. What I found were humans. Ok, one or two are the exceptional kind, who could rebuild your templating engine in C over the weekend just for kicks, saving the company millions of dollars. But there are also many wonderfully talented normal humans doing great work. And I was proud to join them. It's been more than five years since I joined, but the company, the work, the job, the product, the environment, the world. It all keeps changing. A few years ago everyone said we were doomed. Then they said we were amazing. And now they say we're doomed again. The faces at the top sometimes change. Sometimes often. But the team I work with, the team I build and run. This team is a constant. They're always doing the best they can to serve our users. To keep the site running, the service alive. To make it better every day. That is what I love about working at Twitter.

Read