Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. If you’ve been forwarded this email, you can subscribe here.
After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.
This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:
Today, we cover:
**Algorithmic programming speedrun.**While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.**Eight years of infra at Shopify.There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.“Napkin math” as a superpower.**Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they werefarmore expensive than necessary.**A new product without VC funding & Cursor as customer no.1.**After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.
Disclaimer: turbopuffer is a season sponsor of the podcast, but as with all our deepdives, this article is independent of podcast sponsorships.
A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.
While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.
After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.
Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm, and dynamic programming; one of the more tricky concepts to master for algorithmic programming.
I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls:
“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.
Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”
Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.
**Aware of how little he knew about programming, Simon doubled down to catch up. **He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:
As Simon recalled:
“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”
The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:
“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”
But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.
Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:
A five-line scriptto take screenshots and upload them to Imgur, all from the command line (2010)Setting up a Ruby HTTP serveron NGINX using the libraryUnicorn(2010)Why Simon favors SQLover ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)
Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings. The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.
With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke, which went viral on Hacker News, making it to the front page of the site with many comments.
It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.
**In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. **As he told me:
“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.
In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.
For example, if someone at work mentioned TCP, I assumed that surely they would know
exactlywhat’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”
Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:
**Ruby on Rails and databases:Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layerSharding and cutting over:**as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.**Multi-data centers:expanding database footprint from one data center (DC) to multiple DCsSplitting up key services:**Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.
**Simon built a framework to simulate networking conditions called toxiproxy. **The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:
As Simon wrote about this problem at the time:
“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”
This was the inspiration that led to toxiproxy. As Simon told me on stage:
“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.
Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”
Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!
**The biggest benefit of a long tenure at a company was learning to write software that ages well. **Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.
While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?
He sought the actual numbers, wrote a script to collect them, and created a table with the data:
**Simon started to memorize key numbers with flash cards. **He wanted to be able to instantly recall all important numbers. As he told me:
“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.
For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.
This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”
**Napkin math enabled him to challenge design decisions based on benchmarks that had issues. **As he told me:
“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.
For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.
So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.
Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”
**Simon went even deeper into “napkin math” after leaving Shopify. **He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:
“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!
Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could…
