Top all-time · #909
Adam Drake
Personal blog · adamdrake.com · on HN as adamdrakedatacareer
Adam Drake's personal blog at adamdrake.com covers data and career. Since 2018, 72 different Hacker News readers have linked it 112 times, most often “Command-line tools can be faster than a Hadoop cluster (2014)”. The last post was in 2019, but the archive is still widely read.
Sometimes recommended · 7 of 40 sampled comments recommend it?
Why HN links it
HN readers link Adam Drake's 2014 post "Command-line Tools can be 235x Faster than your Hadoop Cluster", which benchmarks a shell and awk pipeline against a Hadoop cluster on one machine. Commenters cite it often as a reference.
- Recommended for · 7 comments
- Called a wake-up call and a wonderful analysis; described as a nice, interesting article showing command-line pipelines beat Hadoop; a "favourite" and a good re
Summarised by Claude from a sample of 40 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Adam Drake is an advisor to scale-up tech companies. He writes about ML/AI/data, leadership, and building tech teams.
Most-linked posts?
- Command-line tools can be faster than a Hadoop cluster (2014)62 people · 77 mentions · first linked 2018-05-23
- Enough with the microservices – Adam Drake3 people · 4 mentions · first linked 2018-10-08
- Big Data, Small Machine2 people · 6 mentions · first linked 2018-06-01
- The Biggest Smallest Website2 people · 2 mentions · first linked 2018-07-05
- Always Be Shipping – Never Do Feature-Freeze2 people · 2 mentions · first linked 2018-07-06
- Three Things Your People Need to Know2 people · 2 mentions · first linked 2018-06-06
- Novel Results Considered Harmful2 people · 2 mentions · first linked 2019-02-11
What Hacker News says?
Spot on. I usually summarize it with clients as "you have to show people you can lead yourself before you can lead others" and that usually gets the idea across. If you wouldn't follow yourself as a leader, then you have some personal work to do. To your second point, one definition of leadership is "getting people to want to do what you want them to do." That's also a pretty reasonable definition, and the way to do…
Comment linking /three-things-your-people-need-to-know.html
on “How to be a Manager – A step-by-step guide to leading a team”Submitted Command-line tools can be faster than a Hadoop cluster (2014) as a story · 463 points
HN discussionAbsolutely. People often don't appreciate or feel comfortable with the fact that when in a leadership role, the relationship changes. New leaders often focus too much on being liked, or having their decisions liked, over doing what is best for the team and its members. Being a popular and well-respected leader is the outcome of good decisions, support, and relationships, not a consequence of trying to do the popular…
Comment linking /teammate-to-team-lead.html
on “How to be a Manager – A step-by-step guide to leading a team”Submitted Command-line Tools can be 235x Faster than a Hadoop Cluster (2014) as a story · 438 points
HN discussionTeammate to Team Lead is a topic that I run into a lot with the companies I advise. I have a small post on the topic, which I'm planning to expand: https://adamdrake.com/teammate-to-team-lead.html There are quite a few things, some of them subtle, that really change once you're in a leadership role. The frequent example I see is that people don't realize that if they are in a leadership role, then there will be deci…
Comment linking /teammate-to-team-lead.html
on “How to be a Manager – A step-by-step guide to leading a team”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) as a story · 362 points
HN discussionMilitary organizations have been studying how to lead people under a variety of conditions since effectively the dawn of time. Many people make the mistake of fetishizing military leadership, but I absolutely thing there is a lot to be learned by studying the techniques of the organizations that have been building leadership skills the longest. I talked a bit about this https://adamdrake.com/command-and-control.html…
Comment linking /command-and-control.html
on “How to be a Manager – A step-by-step guide to leading a team”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) as a story · 240 points
HN discussionThis is great. It reminds of that Adam Drakes’ 2014 article “Command-line Tools can be 235x Faster than your Hadoop Cluster” [1] was the wake up call I needed. 1. https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “You Are Not Google (2017)”Submitted Novel Results Considered Harmful as a story · 17 points
HN discussionThere was also the "Hadoop vs. unix pipeline running on a laptop"-story a few years back, a more modern take: https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Is something bugging you?”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster as a story · 5 points
HN discussion> This find | xargs mawk | mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation. https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Nginx to Be Acquired by F5 Networks”Submitted Command-line Tools can be 235x Faster than a Hadoop Cluster (2014) as a story · 4 points
HN discussionI don’t have an example, but this article comes to mind and you may be able to find an example in it https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Shell script best practices, from a decade of scripting things”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster as a story · 4 points
HN discussion> Pipe this into that and through seven pieces of software that sound like glibgcd, add 8 arcane flags and in the end you have some kind of textfile that would've just as easily been made in some handmade program? What exactly is the selling point for devs? No, that's exactly the selling point. Yes, you could write some hand made program. And piping software together IS a kind of hand made program. It's just going t…
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Software I’m thankful for (2021)”Submitted Always Be Shipping – Never Do Feature-Freeze as a story · 4 points
HN discussionHere's some articles * https://blog.jpalardy.com/posts/why-learn-awk/ (also discussed on HN: https://news.ycombinator.com/item?id=22108680 ) * https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Awk in 20 Minutes (2015)”Submitted Big Data, Small Machine as a story · 4 points
HN discussion> This find | xargs mawk | mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation. https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Kubernetes Is a Surprisingly Affordable Platform for Personal Projects”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster – Adam Drake as a story · 3 points
HN discussionRelated: Pipelines can be 235x faster than a Hadoop cluster https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “The Beauty of Unix Pipelines”Submitted Don't Categorize, Prioritize as a story · 3 points
HN discussionhttps://adamdrake.com/command-line-tools-can-be-235x-faster-...
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Cold Showers”Submitted MAC Addresses, UDIDs, and Privacy as a story · 3 points
HN discussionI wouldn't underestimate how much a modern machine with a bunch of RAM and SSDs can do vs HDFS. This post[1] is now 10 years old and has find + awk running an analysis in 12 seconds (at speed roughly equal to his hard drive) vs Hadoop taking 26 minutes. I've had similar experiences with much bigger datasets at work (think years of per-second manufacturing data across 10ks of sensors). I get that that post is only on…
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Big data is dead (2023)”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) as a story · 2 points
HN discussion> I mean if you're doing data science the data is not always organized and of course you would want multi-processing Not necessarily - I might not want it or need it. It's a few TB, it can be on a fast HD, on an even faster SSD, or even in memory. I can crunch them quite fast even with basic linear scripts/tools. And organized could just mean some massaging or just having them in csv format. This is already the same…
Comment linking /command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
on “Big data is dead (2023)”Submitted Command-line Tools can be 235x Faster than your Hadoop Cluster as a story · 2 points
HN discussion
