Introduction

Our star runner tonight needs no introduction.

This book was born out of my desire to understand Rust better and to learn it while creating something useful.

Being interested in many different things, such as coding, guitar playing, and painting, I find myself constantly in the enviable position of a beginner, eager to sail on an ocean of new concepts.

When I approach something new, whether it's a new programming language or a painting technique, I always realise how true it is that learning happens through experience. It is definitely useful to watch YouTube videos where great coders explain advanced concepts, but it's when I'm in front of an empty file in my editor that I'm forced to apply what I've seen and make it work with my own hands.

A key principle is that our brains retain information better when acquiring it required effort. And while it's perfectly fine to copy and paste commands from StackOverflow (or LLM models) to solve one-off situations, I personally want to learn to use tools and languages properly and to be effective without constantly needing to look up solutions.

So, why a book on coding, if I just stated that we learn by doing and not by reading?

As the author, writing this book was an excellent opportunity to organise my notes, review decisions, and deepen my understanding of certain topics.

For anyone else, the book might be a springboard for getting used to the language before you start coding on your own. Or it might be used as a comparison, to see an alternative solution to the same steps you already went through.

But there is another reason. When we learn something new, we need someone to copy from. Copying is not a shortcut; it's the first step. Once you have absorbed enough patterns and techniques, you naturally start doing things your own way. This book is meant to be that example you can replicate, taking your time to dig deep into the frequent references to the Rust documentation, until you feel confident enough to take a different path.

Having written, edited, corrected, written again, tested, and read every paragraph at least 3 times, I can assure you that it was an amazing journey, full of discoveries and satisfaction. It will also be full of typos and mistakes, though. Forewarned is forearmed.

I hope you will enjoy it as much as I did!

CodeCrafters#

The idea of writing a Redis clone is anything but original. It originated from one of the most interesting projects I encountered in the past decade, CodeCrafters. This company, founded in 2022 by Sarup Banskota and Paul Kuruvilla, provides a very simple service: a set of well-paced challenges that guide you to implement an important piece of software from scratch.

Are you into databases? Here is the Build your own Redis challenge. Do you want to create your own language? No problem, head to Build your own Interpreter. Want to dig into network protocols? Build your own HTTP server and Build your own DNS server to the rescue. And the nice thing is that you can solve it with your favourite programming language, and see the solutions of other coders.

It's TDD applied to integration tests, where CodeCrafters provides the code that tests features of your software, and you provide the code that implements such features.

You don't need CodeCrafters to enjoy the book, but I will follow the challenge "Build your own Redis" strictly, so you might benefit from having an account. You can get discounted access using the code below which gives you a free week of access to the platform and a discount of 40% if you decide to subscribe to the paid tier.

Why this book comes for free#

The first reason I started writing a technical blog was to share with others my discoveries, and to save them the hassle of going through processes I had already cracked. Moreover, I always enjoy the fact that explaining something forces me to better understand that topic, and writing requires even more study to get things clear in my mind, before attempting to introduce other people to the subject.

Much of what I know comes from personal investigations, but without the work of people who shared their knowledge for free I would not have been able to make much progress. The Free Software Movement didn't start with the Internet, and I got a taste of it during the 80s and 90s, but the World Wide Web undeniably gave an impressive boost to the speed and quality of this knowledge sharing.

So this book is a way to say thanks to everybody who gave their time to write blog posts, free books, software, and to organise conferences, groups, meetups. This is why I teach people at conferences, this is why I write a technical blog, this is the reason behind this book.

That said, if you want to acknowledge my effort with money, thanks! However, the best thing you can do is to become part of this process of shared knowledge; experiment, learn and share what you learn.

What's new in the second edition#

You are currently reading the second edition of the book. This is the list of the improvements:

• Major review of the code. Added comments and fixed compiler warnings in early stages.
• Major refactoring of chapter 2 "The RESP protocol".
• Added chapter 8 "Replication - Part 2".
• Added chapter 9 "Transactions".
• Improved PDF rendering of code with coloured diffs and code markers.
• Published HTML version of the book.

Code comments#

In the source code of the project available on GitHub and linked in every section of the book you'll find many comments, that hopefully will integrate what is explained in the book. However, I want to spend a minute discussing the real reason why I heavily comment code.

In the past, I always thought that good code is self-explanatory, and I don't disagree with this statement. However, after many years spent writing, reading, and debugging code, it's clear to me that this is not the full picture.

First of all, code cannot always be "good", as in well-architected, layered, idiomatic. Clearly, sometimes we need to sacrifice beauty for performance, or implement algorithms in languages that give us a lot of power but are less visually pleasing. We all know that in all those cases the rate of comments vs code reaches extremely high levels.

However, those are arguably edge cases. It's much more important, in my opinion, to discuss the fact that programming languages are, as the name says, languages. And even the best coders are not native speakers of such languages. Thus, it's always an additional burden, for a human being, to read code. On top of that, programming languages express the "what/how", but rarely the "why". Even in the era of AI, where LLMs can read thousands of lines of code in seconds and figure out connections, components, and pathways, adding comments can't but clarify what is being done. After all, LLMs teach us that context is king.

All these reasons led me in recent years to a change in style. I tend to comment my code heavily, often line-by-line, and this is the style you will find in this book. Comments are available in the source code on GitHub, but have been copied in the source code boxes shown in the book as well. You are clearly free to either ignore the comments or to change them, as you are free to change everything I do in this project.

Source code#

The book has been written using Mau and rendered with Pelican.

The source code of both the project and the book is available in this GitHub repository.

Feel free to create issues in that repository if you spot any mistakes or if you want to ask for clarifications.

Typographic conventions#

Here are some typographic conventions used throughout the book.

Regular text appears like this, while inline code is displayed like this. When a library or a Rust concept is mentioned for the first time, a link to the official documentation will be added in this form [docs].

When the text mentions concepts that are not directly connected with the project at hand but are worth further investigation an aside will contain details or links.

The code of the project is presented in a box like this

fn main() {
    println!("Hello!");
    println!("Hello, world!");
}

Lines highlighted in red are removed from the file, while lines highlighted in green are added to it. Often, ellipsis (...) will be used to replace long pieces of unchanged code.

When the text discusses specific lines of code, they might be highlighted as follows.

fn main() {
    println!("Hello, world!");
}

or they might be marked with a callout like this

fn main() {
    println!("Hello, world!"); 1
}

which is then usually referred to in the code like this 1.

Sometimes, functions can be too long, so only a small part of them will be shown. In such cases, changes will be surrounded by enough context to make the diff clear.

fn main() {

    println!("Hello, world!");
    println!("This function is really long!");

In this case, the change happens in the function main (highlighted), next to the existing println!.

In certain cases, it's useful to show code that represent an alternative solution or a suboptimal one. In such cases, the code will be shown in a box with a magnifying glass icon to signal that this is not exactly what we are going to implement.

fn main() {
    println!("Hello, world!");
}

If the code shown is not correct, either because it doesn't compile or for other reasons, it will be shown like this.

fn main() {
    println!("Hello, world!");
}

The output of commands run in the terminal, such as the compiler, the test suite, or other shell commands is shown in a box like this.

$ run -a --command

This is the output of the command in a terminal

The code shown in the book can be used to complete the CodeCrafters challenge "Build your own Redis". Whenever the code passes a specific stage a box like this will provide details.

Finally, the code of the project is available in a repository and the current step will be highlighted with a box like this

Set up the development environment#

As the project consists in a sort of reverse engineering of Redis, I decided to call it "Sider", which is Redis written backwards. To follow the project you need an editor (pick your favourite one and make sure you have support for Rust) and the Rust compiler. You can install Rust in your system following the official guide. To be able to use CodeCrafters you also need to configure a Git repository, but you can follow the instructions in the challenge page.

Create the Rust project (CodeCrafters)#

If you are using Codecrafters, follow the instructions they provide. You will clone a repository provided by them that contains the Rust project and a script to run the server.

Create the Rust project (manual)#

To create the project, I recommend using Cargo

$ cargo new sider
$ cd sider

This will create a directory called sider that contains the standard files and directories of a basic Rust project. You can test that everything works correctly with

$ cargo run

Now open the file src/main.rs with your favourite editor and remove the demo code.

Also, create the file spawn_redis_server.sh that contains the following code

exec cargo run \
    --quiet \
    --release \
    --manifest-path $(dirname $0)/Cargo.toml \
    -- "$@"

As you can see, this is just a wrapper around cargo run and will be useful to run tests. Make sure the script is executable

chmod +x spawn_redis_server.sh

How to build the code and run tests#

Since we are using Cargo, you can build the code with

$ cargo build

and run unit tests with

$ cargo test

Make sure you build code often to check it. When working with a compiled language, the compiler is your best friend, and this is true in particular for Rust.

How to test code (CodeCrafters)#

If you are using CodeCrafters, tests will be run every time you commit and push. Make sure you read the official documentation, in particular if you want to use the CodeCrafters CLI.

How to test code (manual)#

You can also run the CodeCrafters tests manually on your machine. Clone the tests repository and change the file internal/test_helpers/pass_all/spawn_redis_server.sh

#!/bin/sh

find "." -type f -name "*.rdb" -exec rm {} +

exec /YOUR/PROJECT/FULL/PATH/spawn_redis_server.sh "$@"

You can then run the tests locally using the provided Makefile. There are several tests that correspond to the base challenge and to the extensions. For the base challenge run

$ make test_base_with_redis

This will however run all the tests in the base challenge, so if you want to check only the steps you completed so far you can copy that rule into a custom one and add steps while you progress, e.g.

test_base_with_redis_prog: build
	CODECRAFTERS_REPOSITORY_DIR=./internal/test_helpers/pass_all \
	CODECRAFTERS_TEST_CASES_JSON="[\
		{\"slug\":\"jm1\",\"tester_log_prefix\":\
                 \"stage-1\",\"title\":\"Stage #1: Bind to a port\"},\
		{\"slug\":\"rg2\",\"tester_log_prefix\":
                 \"stage-2\",\"title\":\"Stage #2: Respond to PING\"}\
	]" \
	dist/main.out

Please note that the last element in a JSON list cannot end with a comma.

test_base_with_redis_prog: build
	CODECRAFTERS_REPOSITORY_DIR=./internal/test_helpers/pass_all \
	CODECRAFTERS_TEST_CASES_JSON="[\
                ...
		{\"slug\":\"rg2\",\"tester_log_prefix\":
                 \"stage-2\",\"title\":\"Stage #2: Respond to PING\"}, \ 
	]" \
	dist/main.out

Once you added the new target, you can run the tests with

$ make test_base_with_redis_prog

About the book#

The cover photo is by Avinash Kumar and can be found on Unsplash.

The website theme is a custom version of PrettyDocs designed by Xiaoying Riley.