Top all-time · #756
Brave New Geek
Personal blog · bravenewgeek.comprogrammingsystems
Brave New Geek is a personal blog at bravenewgeek.com about programming and systems. Since 2013, 85 different Hacker News readers have linked it 157 times, most often “You Cannot Have Exactly-Once Delivery”. It is still writing: the last post was on 2025-05-08, and you can follow it by feed.
Sometimes recommended · 14 of 40 sampled comments recommend it?
Why HN links it
HN readers link Brave New Geek, Tyler Treat's blog, for explanations of distributed systems and messaging: exactly-once delivery, message queues, NATS, distributed locking and retries. Commenters cite it as a reference on those topics.
- Recommended for · 14 comments
- Commenters call posts a good overview, comparison, explanation and series, and say a multi-cloud article is very well written.
Summarised by Claude from a sample of 40 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Introspections of a software engineer
Most-linked posts?
- You Cannot Have Exactly-Once Delivery8 people · 8 mentions · first linked 2019-04-24
- You Cannot Have Exactly-Once Delivery8 people · 8 mentions · first linked 2015-03-25
- Measuring Latency (2015)6 people · 6 mentions · first linked 2020-08-24
- Why We Use Go5 people · 5 mentions · first linked 2015-05-20
- Dissecting Message Queues5 people · 5 mentions · first linked 2014-07-09
- SRE Doesn’t Scale4 people · 4 mentions · first linked 2021-10-06
- Dissecting Message Queues – Brave New Geek4 people · 4 mentions · first linked 2015-09-05
- Smart Endpoints, Dumb Pipes3 people · 4 mentions · first linked 2017-06-30
- GCP and AWS: What’s the Difference?3 people · 3 mentions · first linked 2018-07-17
- You Cannot Have Exactly-Once Delivery Redux3 people · 3 mentions · first linked 2017-06-30
What Hacker News says?
Note that NATS is currently pub/sub, which is a "if a tree falls in the forest" situation. Messages don't go anywhere if nobody is subscribing. So it's awesome for realtime firehose-type use cases where a websocket client connects, receives messages (every client gets all the messages, although NATS also supports load-balanced fanout) for a while, then eventually disconnects. NATS is ridiculously fast [1], too. Ther…
Comment linking /tag/nats/
on “How We Built r/Place”Submitted You Are Not Paid to Write Code as a story · 328 points
HN discussion> Whenever the plumbing does not guarantees EOIO but the business demands it, it gets pushed into the app layer where TCP (retries and acks) is reimplemented, to various success levels. Interesting! You're saying the exact opposite of what Tyler Treat is saying: > Even with smart middleware, problems still leak out and you have to handle them at the edge—you’re now being taxed twice. This is essentially the end-to-e…
Comment linking /smart-endpoints-dumb-pipes/
on “Delivering Billions of Messages Exactly Once”Submitted Building a Distributed Log from Scratch, Part 1: Storage Mechanics as a story · 278 points
HN discussionI think this article[0] is very well written and clearly explains the problems with provisioning resources in multiple clouds. Yes, there are tools that allow you to do it, however you give up most of the reasons for going into the cloud. Switching between cloud providers is much easier than going from on-prem to the cloud. So, once your systems are "cloud" ready you should be able to avoid vendor lock in naturally.…
Comment linking /multi-cloud-is-a-trap/
on “In killing Inbox, Google takes another swipe at its most passionate users”Submitted Building a Distributed Log from Scratch: Data Replication as a story · 217 points
HN discussionI'm just going to leave a direct quote from Rob Pike: """ The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy fo…
Comment linking /go-is-unapologetically-flawed-heres-why-we-use-it/
on “The Value in Go’s Simplicity”Submitted SRE Doesn’t Scale as a story · 162 points
HN discussionSee the other comment but this is a good post/series: https://bravenewgeek.com/building-a-distributed-log-from-scr...
Comment linking /building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/
on “On SQS”Submitted Building a Distributed Log from Scratch: Sketching a New System as a story · 140 points
HN discussionNats streaming isn't just a persistence layer to NATS. It's an entirely different system that basically acts as a client to NATS and then records messages it sees. Basically think of how you would design a persistent queue on top of the ephemeral NATS pub/sub and that's what NATS streaming is. Here's a good post (and series) about distributed logs and NATS design issues: https://bravenewgeek.com/building-a-distribut…
Comment linking /building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/
on “On SQS”Submitted Building a Distributed Log from Scratch: Scaling Message Delivery as a story · 130 points
HN discussionIf you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/
Comment linking /tag/amazon-sqs/
on “On SQS”Submitted Sometimes Kill -9 Isn't Enough as a story · 119 points
HN discussion1)Highscalability http://highscalability.com 2)BraveNewGeek - http://bravenewgeek.com 3)Antirez(redis) - http://antirez.com/latest/0 4)Aphyr - https://aphyr.com 5)GitHub's Engineering Blog - both old and new https://github.com/blog/category/engineering http://githubengineering.com 6)Facebook Engineering https://code.facebook.com/posts/ 7)Twitter Engineering https://blog.twitter.com/engineering 8)Code as Craft - Etsy…
Comment linking /
on “Ask HN: What tech blogs, podcasts do you follow outside of HN?”Submitted You Cannot Have Exactly-Once Delivery as a story · 113 points
HN discussion> Are you saying that this is bad? It doesn't seem so. It's not "bad" per se, but it's not what it seems on the surface. ACKing a message doesn't mean that the message will not be received more than one time. As long as the smarts for recognizing recycled messages are embedded in the application logic, all will be fine. Here's a good explanation written about SQS, but applies to all message technologies that work ba…
Comment linking /you-cannot-have-exactly-once-delivery/
on “Message DB: Event Store and Message Store for PostgreSQL”Submitted Why We Use Go as a story · 107 points
HN discussionYou can get exactly once processing, but not exactly once delivery. https://bravenewgeek.com/you-cannot-have-exactly-once-delive...
Comment linking /you-cannot-have-exactly-once-delivery/
on “Falsehoods programmers believe about TCP”Submitted Service-Disoriented Architecture as a story · 101 points
HN discussion> In a reasonably well-behaved datacenter environment, the timing assumptions will be satisfied most of the time – this is known as a partially synchronous system [12]. But is that good enough? Please consider this answer [0] (which I personally understood as - YES, real systems are always partial sync): "Asynchronous systems have no fixed upper bounds. In practice, systems tend to exhibit partial synchrony, which i…
Comment linking /category/distributed-systems-2/
on “How to do distributed locking”Submitted Fast, Scalable Networking in Go with Mangos as a story · 96 points
HN discussionI'm totally in agreement with this. Both processes should also be idempotent (you should be able to pull multiple times without side effects, and push and pull should be able to happen at the same time without side effects). When everything is working well, the 'push' does all the work, and though the 'pull' runs every few minutes/seconds/whatever it never has anything to do. This same thing applies to time-based ev…
Comment linking /you-cannot-have-exactly-once-delivery/
on “What do you mean by “Event-Driven”?”Submitted So You Wanna Go Fast? as a story · 95 points
HN discussionA follow up from a reader on this article: http://bravenewgeek.com/smart-endpoints-dumb-pipes/
Comment linking /smart-endpoints-dumb-pipes/
on “How do you cut a monolith in half?”Submitted Distributed Systems Are a UX Problem as a story · 89 points
HN discussionThis is probably the most detailed analysis of retry techniques that I've seen. I really appreciated the circuit breaker and deadline propagation sections. But this is why I've pretty much abandoned all connection-oriented logic in favor of declarative programming: https://en.wikipedia.org/wiki/Declarative_programming Loosely, that means that instead of thinking of communication as client-server or peer-to-peer remo…
Comment linking /you-cannot-have-exactly-once-delivery/
on “Good Retry, Bad Retry”Submitted Dissecting Message Queues as a story · 79 points
HN discussionYou almost never want to use average as your metric when dealing with time. For time, the 95 or 99 percentile latency is going to be measure of how many requests out of a hundred a less than your metric. Esp when the thing you are measuring is a piece of larger system, which is the crux, because of how a resource in turn requests k-more resources. Each request has an equal chance of being over the latency percentile…
Comment linking /everything-you-know-about-latency-is-wrong/
on “Sanic – Python 3.5+ web server that's written to go fast”Submitted Benchmarking Commit Logs: NATS Streaming vs. Apache Kafka as a story · 75 points
HN discussion
