Career advice for young system programmers

Much of the advice I see online targeted at “developers” is really bad. And the funny thing is: it is bad advice, even if it is completely true.

That’s because in developers’ minds, it is very easy to say that a “developer” is someone who does the same things I do. But in practice, there are many kinds of developers, focusing on drastically different problems.

Much of the advice online takes the form of “who needs this anyway?”. As a quick detour, that is one of the reasons I appreciate ThePrimeagen so much: he is the rare kind of influencer that is always happy to take a step back, and invite his audience to ask “ok, let’s understand the problem and the domain first”, as opposed to “whoever is doing this or that is doing it wrong”.

It is hard online for system programmers

In common discourse, the main groups of developers are “frontend” and “backend” developers. But even those are just a part of the whole. A part I was never interested in since my early days in the late 1990s.

There are other developers. The ones writing databases, operating systems, compilers, and foundational building blocks that, if they do their jobs right, end up becoming almost invisible to the average programmer.

Advice for systems programmers

Lots of the advice that you see on Twitter (X?), is especially bad when it comes to systems level software. There’s a certain level of “move fast and break things” that you would never accept from your OS Kernel. The whole discussion of whether or not you should write unit tests sound frankly quite stupid from the point of view of a compiler author or some core Open Source library. I could go on.

My goal in this article is to give some pointers to young engineers that want to build a career doing systems programming, based on my personal experiences. People in different circumstances may have different experiences and you should listen to them too.

Do Open Source

Barriers to entry in systems programming are usually higher. Writing a Kernel, a Compiler, or a Database takes years. You can write toy versions of those, but that’s still a large surface area that will take at least months to complete, and you’re now in an environment that is so unrealistic, that you’re not really learning much.

It is possible to get a job somewhere and learn, but that’s much harder. We live in a society that requires 5 years of experience for tools that only exist for 3 years for entry level jobs. Finding a job that will give you access to world class systems programming challenges, albeit possible, is challenging.

Thankfully, lots of cornerstone systems software have their code wide open. If you are interested in compilers, you don’t have to struggle imagining how very basic things could work in your toy compiler. You can just go and play with LLVM.

Pick a project that you like, that is industry-relevant, and is well known for its high standards. For me, in the early 2000s, that was the Linux Kernel. Today, I am the founder of Turso and libSQL, that you can contribute to.

But there are many other projects I can recommend, led by people that I know personally and can vouch for, like ScyllaDB and its companion Seastar Framework (C++), RedPanda (C++), TigerBeetle (Zig). The list is very data oriented because that’s what I know best, but there are a variety of projects out there that will fit your interests.

Do meaningful Open Source work

For this to work, “doing Open Source” needs to be more than a stamp. It’s easy to get your name in a project contributor’s list with trivial commits. But if your goal is to learn, at some point you have to break past this.

Getting started with a big project can be daunting, and you may feel there’s no good place to start. The good news is that when I say “do Open Source”, that doesn’t mean you need to show your Open Source work to anybody.

Remember I said toy projects didn’t help much? That’s because the time-to-value is huge. But a toy project inside an established project? That helps!

In my early days I focused more on modifying Linux than contributing to Linux. As an example, I wrote a really basic, and frankly useless, filesystem. You could write to files, but they always returned a static string.

I wanted to understand filesystems. With my toy fs, I could get started on a real kernel. I learned about the Linux VFS layer, modules, core interfaces and Linux Kernel concepts, and more importantly: the variety of unpredictable ways in which a program that is in complete control of the hardware can just break, and developed the experience on how to debug and fix those issues.

A couple of months later, I made my first (very small) contribution to the ext4 filesystem.

Does Open Source really help you get hired?

The brutally honest reality is that I receive so many applications for people interested in working at our company, that your usual bullet list of technologies mean very little. Yes, interviews should help sifting candidates, but even doing an interview uses up time and resources.

Open Source does help. But these days, getting your name listed as a contributor in an Open Source project is very easy. So your contributions have to be to be meaningful.

Hiring V

We recently hired someone from our Open Source community. I figured it would be helpful to share a bit of that story from our perspective, in the hopes it can help you too!

When we released our private beta in February 2023. Avinash, from Bangalore, India (he goes by V), joined it right away. This immediately set him apart: I receive hundreds of standard job applications over a month, and I don’t have time to go through all of them. But I always have time to talk to my users.

Our Discord community has lots of people who join, ask a question, and then ask for a job. I am never interested in talking to these people. But that wasn’t v’s case. He was genuinely interested in what we were doing. He kept engaging with us consistently, and giving feedback on what he’d like to see next.

Not only was he giving us good feedback — that’s easy to do — he started contributing to our tooling. In fact, he quickly became the third top contributor to our Go client. Our Go client is not the most active project in the world, but his contributions were beyond trivial.

Then came the tipping point. We have an experimental open source initiative to replace the SQLite default storage engine with an MVCC implementation. That’s hard stuff. V started contributing to that too. He filed and fixed a bug that caused the storage engine to corrupt data.

Here’s where it gets interesting: our implementation was correct, as described by the original research paper we based the implementation on. He found a bug in the paper, contacted the authors, who realized the mistake and updated the paper. Then he fixed the implementation.

No other image on the internet could do justice to what I thought at the moment, except for the one and only, Tiffany Gomas.

Companies have a hard time finding great talent. And being a good match for a company goes beyond your coding ability. Are you aligned with the vision? Do you understand what we’re trying to do? Because of that, hiring someone is always a risk, even if a person excels in a technical interview (which as I said earlier, are costly to run on their own).

It later became known to us that V was looking to switch jobs, and we had just opened a position for our Go backend.

The only disagreement we had internally was whether or not we should even interview this guy. My position as the CEO was: why would we even waste time interviewing him? What do we hope to learn? Just send the offer!

I guess people are set in their ways. It feels strange to hire someone without an interview. And because companies sometimes like to give people take home assignments, Sarna, one of our engineers, brilliantly crafted a humorous take on why it was indeed, pointless to even interview V:

And so, we hired V.

Summary

If you are interested in the plumbing of our software industry — systems level programming, much of the career advice online won’t apply to you. I hope in this article I could give you some useful advice on how to stand out from the crowd.

I showed you a real example of someone who did stand out, and got hired by us. If you want to chat with V, you can join our Discord channel today!