Top all-time · #1208
Frank McSherry
Personal blog · frankmcsherry.orgsystemsprogrammingdata
Frank McSherry's personal blog at frankmcsherry.org covers systems, programming and data. Since 2014, 56 different Hacker News readers have linked it 95 times, most often “Scalability, but at what cost? (2015)”. The last post was in 2015, but the archive is still widely read.
Sometimes recommended · 6 of 40 sampled comments recommend it?
Why HN links it
HN readers link his 'Scalability! But at what COST?' post and the COST paper, which show single-threaded code on one laptop can beat cluster systems. Mostly cited as a reference in big-data discussions.
Summarised by Claude from a sample of 40 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Scalability, but at what cost? (2015)29 people · 34 mentions · first linked 2015-01-16
- At what scale is a distributed system required? [pdf]19 people · 23 mentions · first linked 2015-01-22
- Bigger data; same laptop7 people · 10 mentions · first linked 2015-02-05
- Scalability but at What Cost? [pdf]5 people · 5 mentions · first linked 2019-03-03
- /graph/scalability/cost/2015/01/15/COST.html4 people · 4 mentions · first linked 2020-10-13
- Columnarization in Rust, part II2 people · 2 mentions · first linked 2014-12-18
- Differential Dataflow for real-time graph analytics2 people · 2 mentions · first linked 2015-05-12
- The impact of fast networks on graph analytics, part 12 people · 2 mentions · first linked 2015-07-08
What Hacker News says?
This post: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... "Scalability! But at what COST?" In it Frank McSherry uses a handful of data structure tricks to make graph algorithms like connectivity and PageRank run progressively faster on his laptop, beating distributed cluster implementations. It's actually a version of their HotOS paper with Isard and Murray. Admittedly it's the opposite of obscure,…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Ask HN: What are some cool but obscure data structures you know about?”Submitted Scalability, but at what cost? (2015) as a story · 165 points
HN discussionIt's been formalized past that into the COST methodology of assessing distribution solutions: Configuration that Outperforms a Single Thread. If that configuration doesn't exist, just ditch the solution in favor of a single thread. :) http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... It just gets more hilarious the more you read the measurements of all these…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “HN comments are underrated”Submitted The Cost of Scalability in Graph Processing as a story · 107 points
HN discussionThanks for writing this. I think many of your observations are correct (and agree with them). I'm not as sold on your solutions though. Here's some thoughts: Abstractions are a correct way to do things when we don't know what "correct" is or need to deal with lots of unknowns in a general way. And then when you need to aggregate lots of different abstractions together, it's often easier to sit yet another abstractio…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Ideal OS: Rebooting the Desktop Operating System”Submitted Bigger data; same laptop as a story · 80 points
HN discussionhttp://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Microsoft open-sources Graph Engine”Submitted Columnarization in Rust as a story · 75 points
HN discussionIt reminds me of https://www.frankmcsherry.org/graph/scalability/cost/2015/01... - you pay an extremely high cost upfront without even noticing - for the promise that at some point, you'll be able to scale out to any scale, with no extra human effort and for costs proportional to the scale. I have not observed enough uses of serverless I can draw conclusions from, but if it's anywhere like hadoop style scaling, 95%…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “The Serverless Revolution Has Stalled”Submitted Abomonation: terrifying serialization as a story · 47 points
HN discussion"Big data" frameworks are very good at throwing a lot of hardware at problems. See eg the classic big data vs laptop treatment: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Working for a startup makes less sense”Submitted The impact of fast networks on graph analytics, part 1 as a story · 17 points
HN discussionA classic from 2015 along the same lines: Scalability, but at what COST? http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Command-line tools can be faster than a Hadoop cluster (2014)”Submitted Worst-case optimal joins, in dataflow as a story · 16 points
HN discussionSee also: Scalability! But at what COST? [1] [2] > The COST of a given platform for a given problem is the hardware configuration required before the platform outperforms a competent single-threaded implementation. COST weighs a system’s scalability against the overheads introduced by the system, and indicates the actual performance gains of the system, without rewarding systems that bring substantial but paralleliz…
Comment linking /assets/COST.pdf
on “Command-line Tools can be 235x Faster than a Hadoop Cluster (2014)”Submitted The impact of fast networks on graph analytics, part 2 as a story · 14 points
HN discussionSee his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http:/…
Comment linking /graph/scalability/cost/2015/02/04/COST2.html
on “Redis re-implemented in Rust”Submitted Differential dataflow in Rust as a story · 14 points
HN discussionLittle secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Redis re-implemented in Rust”Submitted Columnarization in Rust, part II as a story · 5 points
HN discussionThis reminded me of Frank McSherry's article "Scalability! But at what COST?" (2015) - https://www.frankmcsherry.org/graph/scalability/cost/2015/01... Where he shows a single laptop beating a spark cluster.
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”Submitted Differential Dataflow for real-time graph analytics as a story · 4 points
HN discussionThis idea was the subject of a paper at a major systems conference. The paper is called "Scalability! But at what cost?" - It goes well beyond this simple example above to explore how most major systems papers produce results that can be beaten by a single laptop. Here's the paper and the blog post describing it. http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/graph/scalability/cost/2015/01…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “Command-line tools can be faster than a Hadoop cluster (2014)”Submitted Scalability but at what COST? [pdf] as a story · 3 points
HN discussionThere's even a standard model for that kind of thing: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “How to Call B.S. On Big Data: A Practical Guide”Submitted At what scale is a distributed system required? [pdf] as a story · 3 points
HN discussion“the performance of the underlying hardware has increased so much in the last decade that the simplicity of SQLite has become a more important tradeoff” Thanks for articulating this so clearly! It's going to be interesting also to see how much simplicity is realized from new thread-per-core designs leveraging io_uring to get rid of multithreading, locks and memory barriers scattered throughout the control plane. The…
Comment linking /graph/scalability/cost/2015/01/15/COST.html
on “How SQLite Helps You Do ACID”Submitted Scalability but at What Cost? [pdf] as a story · 2 points
HN discussionVery nice, reminds me of a quote from this very related paper: You can have a second computer once you’ve shown you know how to use the first one. –Paul Barham Scalability! But at what COST? https://www.usenix.org/system/files/conference/hotos15/hotos... http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... Interesting it also involves graph processing, which Bane's comment did. The point is that distribut…
Comment linking /graph/scalability/cost/2015/02/04/COST2.html
on “Ask HN: Your Favourite HN Comment?”Submitted Scalability but at What Cost? (2015) as a story · 2 points
HN discussion
