Top all-time · #704
John Allspaw
Personal blog · kitchensoap.comsystemscareer
John Allspaw writes Kitchen Soap, a personal blog at kitchensoap.com about systems and career. Since 2009, 92 different Hacker News readers have linked it 104 times, most often “On Being A Senior Engineer”. The last post was in 2017, but the archive is still widely read.
Often recommended · 26 of 39 sampled comments recommend it?
Why HN links it
HN readers link John Allspaw's blog for its essays on senior engineering, root-cause analysis and incident post-mortems, such as the Knight Capital counterfactuals.
- Recommended for · 26 comments
- Called a classic, a great read and the best take on the subject; one says it is much better than a rival analysis.
Summarised by Claude from a sample of 39 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- On Being A Senior Engineer28 people · 28 mentions · first linked 2012-10-25
- On Being a Senior Engineer (2012)13 people · 17 mentions · first linked 2017-11-03
- Each Necessary, but Only Jointly Sufficient (2012)6 people · 6 mentions · first linked 2017-11-13
- Counterfactual Thinking, Rules, and The Knight Capital Accident5 people · 5 mentions · first linked 2013-10-30
- There is No Root Cause4 people · 4 mentions · first linked 2012-02-10
- On Being A Senior Engineer3 people · 3 mentions · first linked 2012-10-26
- Counterfactual Thinking, Rules, and the Knight Capital Accident3 people · 3 mentions · first linked 2019-04-01
- On Being a Senior Engineer2 people · 2 mentions · first linked 2015-02-28
- Owning Attention (Considerations for Alert Design)2 people · 2 mentions · first linked 2013-07-22
- /2 people · 2 mentions · first linked 2011-02-22
What Hacker News says?
Perhaps more accurate is that there is no one root cause, but a bunch of contributing factors. I had a quick search to find my favourite article on this one, it might be https://willgallego.com/2018/04/02/no-seriously-root-cause-i... as it demolishes Five Whys but https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... is a classic
Comment linking /2012/02/10/each-necessary-but-only-jointly-sufficient/
on “Amazon tells employees to return to office five days a week”Submitted On Being a Senior Engineer (2012) as a story · 155 points
HN discussionIt's a sign of poor management that someone has to be fired when something goes wrong, outages are learning situations for all involved, and it is widely held that the person who took the action that caused an outage is not responsible, but that all involved are responsible. See John Allspaw's Swiss Cheese Theory : http://www.kitchensoap.com/2012/02/10/each-necessary-but-onl... . [ Edit: I guess it's not Allspaw's m…
Comment linking /2012/02/10/each-necessary-but-only-jointly-sufficient/
on “Count to ten when a plane goes down”Submitted On Being A Senior Engineer as a story · 77 points
HN discussionAs always it depends on the organization or team. I have worked on teams where the title was nothing more than a title and didn't relate to their skill level or work ethic. One of my favorite posts on the subject comes from John Allspaw: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi... An important part of the equation is a companies engineering structure. Camille Fournier has a great set of slides ab…
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “What Makes a Senior Software Developer?”Submitted There is No Root Cause as a story · 77 points
HN discussionThere is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...
Comment linking /2012/02/10/each-necessary-but-only-jointly-sufficient/
on “Your nines are not my nines”Submitted On Being A Senior Engineer as a story · 48 points
HN discussionWhich is unfortunate because tmarthal's link is quite good: > Here is a great article on what it means to be a Senior engineer: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi... > The long answer is maturity. Then the author goes to explain what maturity brings to the discipline of software engineering.
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Engineers suck at finding the right jobs”Submitted On Being a Senior Engineer as a story · 26 points
HN discussionHere is a great article on what it means to be a Senior engineer: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi... The long answer is maturity. Then the author goes to explain what maturity brings to the discipline of software engineering.
Comment linking /2012/10/25/on-being-a-senior-engineer/?utm_source=statuscode&
on “Engineers suck at finding the right jobs”Submitted On Being a Senior Engineer as a story · 16 points
HN discussionThat's a very good point, John Allspaw from Etsy has a great post on that topic: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi...
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Engineers suck at finding the right jobs”Submitted A Mature Role for Automation: Part I as a story · 10 points
HN discussionEvery time it comes to “root cause analysis” (quoting makes sense in this context) I think of https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...
Comment linking /2012/02/10/each-necessary-but-only-jointly-sufficient/
on “CenturyLink 911 outage was caused by a single network card sending bad packets”Submitted The Infinite Hows (2014) as a story · 6 points
HN discussionThere is a really, really interesting breakdown of this document on kitchensoap.com [0] The writer breaks down why this document is not a post-mortem despite superficial similarities. >Again, the purpose of the doc is to point out where Knight violated rules. It is not: 1) a description of the multiple trade-offs that engineering at Knight made or considered when designing fault-tolerance in their systems, or 2) how…
Comment linking /2013/10/29/counterfactuals-knight-capital/
on “How to lose $172k per second for 45 minutes (2013)”Submitted Learning from Failure at Etsy as a story · 6 points
HN discussionHindsight is a helluva drug. The SEC report cannot be viewed as a “postmortem.” https://www.kitchensoap.com/2013/10/29/counterfactuals-knigh...
Comment linking /2013/10/29/counterfactuals-knight-capital/
on “The $440M software error at Knight Capital (2019)”Submitted Multiple Perspectives on Technical Problems and Solutions as a story · 5 points
HN discussionHOWTO: Be more productive http://www.aaronsw.com/weblog/productivity On Being A Senior Engineer http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi...
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Articles Every Programmer Should Read”Submitted On Being a Senior Engineer (2012) as a story · 5 points
HN discussionThat moral seems related to a meme floating around that Mean Time To Recovery is more important than Mean Time Between Failures. I think it came from, or through, John Allspaw: https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-ty... It's popular with a lot of people i know, because it's one of those ideas that's counterintuitive but believable, and so makes you feel really smart for knowing it.
Comment linking /2010/11/07/mttr-mtbf-for-most-types-of-f/
on “The Servers Are Burning”Submitted Deployment is just a part of dev/ops cooperation, not the whole thing as a story · 5 points
HN discussionThis is by far the best article I've ever read on the subject. http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi...
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Ask HN: When are you considered a “senior” programmer?”Submitted How Complex Systems Fail: A WebOps Perspective as a story · 5 points
HN discussionI think probably one of the most valuable things you can do with your junior developers is to set expectations around how to ask good questions. Your goal is to help juniors strike the right balance between spending time figuring something out on their own versus spending too much time and going way off track. You also don't want your juniors to ask too many questions that they should have first spent some time inve…
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Ask HN: Strategies for mentoring junior developers?”Submitted On Being a Senior Engineer as a story · 4 points
HN discussionA great read on this topic: http://www.kitchensoap.com/2012/10/25/on-being-a-senior-engi...
Comment linking /2012/10/25/on-being-a-senior-engineer/
on “Ask HN: What non-technical skills make a senior dev and how to develop them?”Submitted Counterfactual Thinking, Rules, and The Knight Capital Accident as a story · 4 points
HN discussion
