In addition, there’s a flood of non-technical takes on the BGP due to many people being interested in the non-technical parts of Bitcoin and other cryptocurrencies. It makes it hard to research the details of the problem from a Computer Science perspective.

At this point, I believe we all know, at a very high level, what the BGP is and its main conclusion: An analogy between software processes/nodes and the Byzantine generals trying to reach a consensus about something. Consensus is a hard problem, but it’s especially hard when the nodes, or the generals, can act adversarially. In other words, the other participant nodes of the network can stop responding due to a crash, but, worse, they can respond with wrong information (think bit flipping or other bugs) or with maliciously crafted information, as an adversary of a protocol would.

And then, after a lot of details in the paper, Leslie concludes that a Byzantine fault-tolerant system needs at least $3m + 1$ nodes to support at most $m$ faulty, or traitor/adversarial, nodes. Easy to work with the conclusion: if you have 4 nodes, you can only support 1 bad guy. If you have 2 bad guys in the system, then you’re gonna need 7 nodes, where 5 are honest. And so on. It’s important to note that this result is specifically for synchronous networks. This is a pretty important assumption. It changes a lot once we move to an asynchronous network model, which was solved in later works such as Practical Byzantine Fault-Tolerance by Miguel Castro and Barbara Liskov.

However, this post isn’t only about going over these results. I would like to dive into the details. Not only that, but we’re gonna write code to test some of the ideas on Leslie’s paper. I’m a big fan of building something to truly understand that thing. So let’s get started.

Context and goals are everything

One thing that helped me understand the proposed algorithm was to constantly go back and remember the main goals of the solution proposed by Leslie.

This snippet of the paper captures the high level perfectly:

We imagine that several divisions of the Byzantine army are camped outside an enemy city, each division commanded by its own general. The generals can communicate with one another only by messenger. After observing the enemy, they must decide upon a common plan of action. However, some of the generals may be traitors, trying to prevent the loyal generals from reaching agreement. The generals must have an algorithm to guarantee that

A. All loyal generals decide upon the same plan of action. The loyal generals will all do what the algorithm says they should, but the traitors may do anything they wish. The algorithm must guarantee condition A regardless of what the traitors do. The loyal generals should not only reach agreement, but should agree upon a reasonable plan. We therefore also want to insure that

B. A small number of traitors cannot cause the loyal generals to adopt a bad
plan.

After some discourse around those two points, he lays down a very important piece for us, the interactive consistencies:

Byzantine Generals Problem. A commanding general must send an order to
his $n - 1$ lieutenant generals such that:

IC1. All loyal lieutenants obey the same order. IC2. If the commanding general is loyal, then every loyal lieutenant obeys the
order he sends.

Conditions IC1 and IC2 are called the interactive consistency conditions. Note
that if the commander is loyal, then IC1 follows from IC2. However, the commander need not be loyal.

To solve our original problem, the i-th general sends his value of v(i) by using a solution to the Byzantine Generals Problem to send the order “use v(i) as my value”, with the other generals acting as the lieutenants.

There’s so much packed in this single passage. Personally, while writing the code to solve the BGP, I found myself very confused about many things, most of which were answered by referring back to the passage above.

First, Leslie introduces the concept of commanding general. If we have 4 generals, at the beginning of the algorithm we’ll promote one of them to be the commanding general, sending orders to the other 3 generals, now called lieutenants.

Second, as the IC1 says, we care about all loyal (or non-faulty) generals (not the commanding general) following the same order. If we have 4 generals, 1 being a loyal commanding general, 2 loyal generals, and 1 traitor general, it doesn’t matter what the one traitor does; we care that the 2 loyal generals follow the order given by the loyal commander. Eventually, one can realize that’s the goal of most distributed consensus mechanisms.

Third, if the commanding general is loyal, then the order that all loyal generals are following is the same order given by the loyal commanding general. In the case of a traitor commanding general, all hope is lost.

Now, here’s one of the most, if not the most, important parts of this whole thing:

To solve our original problem, the i-th general sends his value of v(i) by using a solution to the Byzantine Generals Problem to send the order “use v(i) as my value”, with the other generals acting as the lieutenants.

This is a recursive solution. Here’s a concrete example: If we have 4 generals, 1 being the commanding general, once the commander sends their value $i$ to the other 3 lieutenants, each lieutenant will act as a commander in the next iteration of the algorithm, but considering only $n-1$ generals.

On the first step of the algorithm, this is what we have:

Then, once the commanding general sends their order, there will be 3 more steps. Each of the 3 lieutenants will act as commander and send the message they received from the commanding general to the other 2 lieutenants. Meaning that the original commanding general is out of the picture for now:

At the end, we have something like this:

And by doing this, something magical emerges: if you have $3m +1$ generals, you can tolerate $m$ traitors. Meaning:

1. All loyal generals will follow the same order (IC1)
2. If the commanding general is loyal, then every loyal lieutenant obeys the order he sends.

We haven’t seen the proper specs of the algorithm, but that paints the picture of what we want to achieve and the overall idea of how we want to achieve it.

The oral message algorithm

The first solution to this problem, as Leslie named it, is the Oral Message (OM) algorithm. Which will do what we described above.

A few relatively easy to understand assumptions are then made:

1. Every message is delivered correctly;
2. The receiver of the message knows who sent it;
3. The absence of a message can be detected (can be a simple timeout).

The OM algorithm is defined as $OM(m)$, and that’s something that confused me, probably for silly reasons. It wasn’t clear the correlation between $m$ the number of traitors in the system and, as it is in the case of the algorithm, $m$ the level of recursion in the OM algorithm. So, just to clarify, the number of traitors that the algorithm can tolerate is the same as the recursion level in this algorithm. They’re the same $m$.

Leslie then formally describes the OM algorithm very succinctly:

1. Algorithm $OM(0)$:
   1. (1) The commander sends his value to every lieutenant.
   2. (2) Each lieutenant uses the value he receives from the commander or uses the value RETREAT if he gets no value.
2. Algorithm $OM(m)$, $m > 0$:
   1. (1) The commander sends his value to every lieutenant.
   2. (2) For each $i$, let $v(i)$ be the value Lieutenant $i$ receives from the commander. Lieutenant i acts as the commander in Algorithm $OM(m - 1)$ to send the value $v(i)$ to each of the $n - 2$ other lieutenants.
   3. (3) For each $i$, and each $j \neq i$, let $v(j)$ be the value Lieutenant $i$ received from Lieutenant $j$ in step (2) (using Algorithm $OM(m - 1)$). Lieutenant $i$ uses the value $majority(v_1,…v_{n-1})$.

It’s a short algorithm, but there’s so much packed into it. I lost count of how many times I had to re-read this thing while implementing the code.

The base case ($OM(0)$) is actually straightforward. The commander at a given round sends their value to every lieutenant general, and each lieutenant uses that value (e.g., either retreat or attack). But here’s a nice detail: by using the value, it doesn’t necessarily mean that the lieutenant will execute based on that value, not until the end of the algorithm. It will simply hold on to that value either as their decision (came directly from a commander) or as a set of all values received from other lieutenants, which will then be used to make a decision based on the majority in the that lieutenants' set of values.

Things become slightly harder to visualize and reason once we think about the actual recursive rounds ($OM(m)$).

First, the commander of that round will send their value to every lieutenant. “Their value” here deserves some attention. What value? Where did they get this value? If they’re the commanding general (the first commander), it’ll be the value that we want all lieutenants/nodes to agree upon. But if a lieutenant general got “promoted” to a commander in this round, it’ll be the value they received from the previous commander.

Then, in the actual recursive step, after the current commander sends their value to every lieutenant, each lieutenant, which doesn’t include the current commander will be “promoted” to a commander and execute the same algorithm, but now decrementing the $m$ (the recursion level), running $OM(m-1)$. By “to send the value $v(i)$ to each of the $n - 2$ other lieutenants,” Leslie means that the new commander sends the message to the lieutenants and doesn’t include itself and the previous commander.

A tricky bit here that may sound obvious but, once overlooked, can trigger some confusion is that this recursive step happens for each lieutenant in the round. This means the step (3) of the first round won’t be executed until all current lieutenants have acted as commanders and passed the $v(i)$, which might include more and more recursive steps!

It might help to visualize this workflow as a tree, which actually is how some solutions are implemented:

And then, in step 3, after each lieutenant went into the recursive steps, a given lieutenant $i$ will use, as its decision, the majority value in the set of all the values they have received.

In other words, a commander in a certain round “spreads the word” and then waits to hear back from each lieutenant, then uses, as their own decision, the majority of these values they received from each lieutenant. This is a clever technique to overcome false information, i.e., “I could’ve heard something fake, so I’ll wait to hear back from the others before making my decision.”

Coding the Oral Message algorithm

Note: you can skip ahead and check the final code here

So what does the code look like? Let’s start with the smallest abstraction unit, a simple message that will hold a boolean value:

#[derive(Debug, Clone, Default, Copy)]
struct Message {
    attack: bool,
}

A general lieutenant is also pretty simple to represent:

#[derive(Debug, Clone, Default)]
struct General {
    id: usize,
    messages: Vec<Message>,
    is_traitor: bool,
    decision: bool,
    total_messages_received: usize,
}

Each general holds a list of messages they will receive and eventually will hold a final decision.

We’re going to use RNG to make m Generals with is_traitor set to true.

Also finally, total_messages_received is a field that could help with debugging and show some cool properties later on. Not really necessary for the algorithm.

Receiving an order is also very straightforward:

// Note: this is inside a `impl General {}` block
fn receive_order(&mut self, msg: Message, from: usize) {
 if self.messages.is_empty() {
  self.decision = msg.attack;
 }

 self.messages.push(msg);
 self.total_messages_received += 1;
}

A bit of detail is that if the list of messages is empty, we’re gonna set the first message’s decision as this General’s decision.

Then, making the decision based on the majority is also very simple:

fn decide(&mut self) {
 let msgs = self
  .messages
  .iter()
  .map(|m| m.attack)
  .collect::<Vec<bool>>();


 let attack = msgs.iter().filter(|&attack| *attack).count();
 let retreat = msgs.iter().filter(|&attack| !*attack).count();

 if attack > retreat {
  self.decision = true;
  return;
 }
 self.decision = false;
}

Count number of trues and falses and decide based on the majority.

That’s pretty much it for the General. Oh, actually, there’s one more thing just to help us simulate faulty Generals; we need something to scramble orders if a given General instance is a traitor! If it’s not a traitor, then just return the original message.

fn next_order(&self, idx: usize) -> Message {
 if self.is_traitor && idx % 2 == 0 {
  if self.decision {
   return Message { attack: false };
  } else {
   return Message { attack: true };
  }
 }
 Message {
  attack: self.decision,
 }
}

Also, to make things more chaotic, we not only invert the order but only invert the order if a certain index is even. Pure evil. Now onto the hard bits!

The Oral Message algorithm will be built around the following struct:

struct OMAlgorithm {
 // Both these fields will be used later on
    is_first_commander_loyal: bool,
    original_order: Message,
}

Now, Rust is going to make our lives a bit harder. Generals will hold a list of Messages, and this will be constantly updated as a General receives orders, and since this will be a recursive algorithm and we’ll be passing lists of references to Generals around to achieve memory safety, Rust must hurt us deeply. That’s okay, we’ll cope.

Since Generals will be passed around and mutated, we’ll have to use the Rc<RefCell<T>> pattern: Rc is a pointer with shared ownership while RefCell provides interior mutability. This means a given Rc<RefCell<General>> is shared, and each shared owner gets to mutate the contents (update their list of orders). The effect of mutating the contents will be seen by all of the shared owners of the outer Rc because the inner data is shared.

Sounds simple once it’s all laid out, but it took me a while to get to this solution, especially when you could do it much more easily in order languages, such as Go. But data races, though… Ugh, painful.

Also, it would be relatively easier if we were implementing an actual protocol. Each general would be its own process, and they would communicate over RPC or HTTP endpoints; we wouldn’t need to keep managing all these references.

So this is how the implementation will look like at a high level

impl OMAlgorithm {
    fn om_algorithm(
        &self,
        generals: &[Rc<RefCell<General>>],
        commander: Rc<RefCell<General>>,
        m: usize,
    ) {
        if m == 0 {
            // OM(0):
        } else {
            // OM(m):
        }
    }
}

Where in the OM(m), m > 0 block we’ll be calling om_algorithm() recursively.

Let’s tackle the base case first:

impl OMAlgorithm {
    fn om_algorithm(
        &self,
        generals: &[Rc<RefCell<General>>],
        commander: Rc<RefCell<General>>,
        m: usize,
    ) {
        if m == 0 {
            // OM(0):
            // The commander sends his value to every lieutenant.
            for (idx, general_rc) in generals.iter().enumerate() {
                let msg_to_relay = commander.as_ref().borrow().next_order(idx);

                let mut general = general_rc.borrow_mut();

                general.receive_order(msg_to_relay, commander.as_ref().borrow().id);
            }
        } else {
            // OM(1)
        }
    }
}

Note that we do:

let msg_to_relay = commander.as_ref().borrow().next_order(idx);

To decide whether we’re relaying the original message or a faulty one, in case the commander of this round is faulty.

Then, we mutably borrow a General from the list of Generals and call its receive_order. And that’s it; that’s the whole base case, as stated by Leslie:

OM(0): The commander sends his value to every lieutenant

Then, the $OM(m), m > 0$ case has three parts:

1. First, the commander of the round sends a message to every lieutenant
   1. In our code, this means the commander is the commander: Rc<RefCell<General>> in the function argument of this specific invocation of om_algorithm(), and the lieutenants are the list we’ve received in this function call, i.e. the generals: &[Rc<RefCell<General>>]
2. Second, for each General lieutenant in generals: &[Rc<RefCell<General>>], we’ll be calling om_algorithm, again, but now passing in list of Generals without the current commander, and the new commander being this General (the current one in the “for each”)
   1. This has interesting behaviors, as it’s the recursive step. It might take some time to click.
3. Lastly, and putting it in the simplest way: for each general in this scope’s generals, we’ll “ask” them to decide the final order based on the majority each has in their local list of orders.

It’s much easier to see in code, actually.

impl OMAlgorithm {
    fn om_algorithm(
        &self,
        generals: &[Rc<RefCell<General>>],
        commander: Rc<RefCell<General>>,
        m: usize,
    ) {
        if m == 0 {
            // OM(0):
            // The commander sends his value to every lieutenant.
            for (idx, general_rc) in generals.iter().enumerate() {
                let msg_to_relay = commander.as_ref().borrow().next_order(idx);

                let mut general = general_rc.borrow_mut();

                general.receive_order(msg_to_relay, commander.as_ref().borrow().id);
            }
        } else {
            // OM(1) (1):
            // The commander sends his value to every lieutenant.
            for (idx, general_rc) in generals.iter().enumerate() {
                let msg_to_relay = commander.as_ref().borrow().next_order(idx);

                let mut general = general_rc.borrow_mut();

                general.receive_order(msg_to_relay, commander.as_ref().borrow().id);
            }

            // OM(1) (2):
            // For each i, let v(i) be the value Lieutenant i receives
            // from the commander, or else be RETREAT if he
            // receives no value.
            // Lieutenant i acts as the commander in Algorithm OM(m-1)
            // to send the value v(i) to each of the n-2 other lieutenants.
            for general_rc in generals {
                let next_commander = general_rc.as_ref().borrow();

                let mut new_generals: Vec<Rc<RefCell<General>>> = vec![];

                for general in generals {
                    if general.as_ref().borrow().id != next_commander.id {
                        new_generals.push(Rc::clone(general));
                    }
                }
                drop(next_commander);

                // Sending message v(i) received from the commander
                self.om_algorithm(&new_generals, Rc::clone(general_rc), m - 1);
            }

            // (3) For each i, and each j != i, let v(j) be the value Lieutenant i
            // received from Lieutenant j in step (2) using Algorithm OM(m - 1),
            // or else RETREAT if he received no such value.
            // Lieutenant i uses the value majority (vl, ..., vn-1 ).
            // Note: honestly, this was the most subtle part of the whole algorithm...
            // Each general is updating their beliefs based on what they heard so far.
            for general in generals {
                general.borrow_mut().decide();
            }
        }
    }
}

And that’s it! It’s short, but it has a lot of details that will be more apparent the more you look at and reason about the code.

But, now, how do we decide whether the algorithm was successful? That’s a step that can be tricky to debug if you get it wrong. To decide that, we have to refer back to the Interactive Inconsistencies laid out by Leslie at the beginning of the paper:

IC1: All loyal lieutenants obey the same order. IC2: If the commanding general is loyal, then every loyal

fn was_successful(&self, generals: &[Rc<RefCell<General>>]) -> bool {
    // Here's how Lamport defines success:
    // IC1: All loyal lieutenants obey the same order.
    // IC2: If the commanding general is loyal, then every loyal
    // lieutenant obeys the order he sends.

    // Decision of each loyal general
    let mut loyal_decisions = vec![];

    for general in generals {
        let general = general.as_ref().borrow();
        if !general.is_traitor {
            loyal_decisions.push(general.decision)
        }
    }

    // Was there consensus amongst all loyal generals?
    let loyal_consensus = loyal_decisions.windows(2).all(|w| w[0] == w[1]);

    // If not, we've broken IC1
    if !loyal_consensus {
        return false;
    }

    // If we had consensus among loyal generals and the first commander
    // is loyal, does the consensus match the original order?
    // If it doesn't, we've broken IC2.
    if self.is_first_commander_loyal {
        // Then the loyal consensus should match original order
        if self.original_order.attack != loyal_decisions[0] {
            return false;
        }
    }

    // IC1 and IC2 hold
    true
}

In short, was_successful is just following Leslie’s 2 Interactive Consistencies.

Now, all we have to do is set up a nice environment for some experiments, which include:

1. Creating many generals;
2. Turning m generals into traitors;
3. Selecting one of these generals as the initial commanding general;
4. Running the algorithm with this setup;
5. Checking if it was_suceessful().

Also, it would be nice to run this experiment many times, letting some RNG pick different combinations of traitors and commanding generals. This is simple enough, though, so I’ll just lay out the code:

fn main() {
    // Configuration params
    let num_of_generals = 4;
    let num_of_traitors = 1;
    let m = num_of_traitors;
    let num_of_experiments = 10;

    for _i in 0..num_of_experiments {
        let mut generals: Vec<Rc<RefCell<General>>> = vec![];

        // Create all generals, set `is_traitor` as false initially
        for i in 0..num_of_generals {
            let g = Rc::new(RefCell::new(General::new(i + 1, false)));
            generals.push(g);
        }

        // Pick `num_of_traitors` random generals and set them as traitors
        // Note that this could also be the first commanding general
        generals
            .choose_multiple(&mut rand::thread_rng(), num_of_traitors)
            .for_each(|traitor| {
                println!("Traitor: General #{}\n", traitor.as_ref().borrow().id);
                traitor.as_ref().borrow_mut().is_traitor = true;
            });

        let order = Message { attack: true };

        let mut om_algorithm = OMAlgorithm {
            is_first_commander_loyal: true,
            original_order: order,
        };

        // Randomly pick first commanding general
        let first_commander = rand::thread_rng().gen_range(0..num_of_generals);
        let first_commander_rc = Rc::clone(&generals[first_commander]);

        first_commander_rc.borrow_mut().decision = order.attack;

        // Remove first commanding general out of the initial list of lieutenant generals
        generals.remove(first_commander);

        if first_commander_rc.as_ref().borrow().is_traitor {
            om_algorithm.is_first_commander_loyal = false
        }

        // Start the algorithm
        om_algorithm.om_algorithm(&generals, first_commander_rc, m);

        // Calculate total messages sent for analytical purposes
        om_algorithm.get_total_messages(&generals);

        // Check whether it was successful
        let successful = om_algorithm.was_successful(&generals);

        println!("successful: {:?}\n", successful);

        if !successful {
            println!("Found a case where consensus isn't achieved\n");
            break;
        }
    }
}

Now we can play with different parameters and see how it affects the results. For instance, with 4 generals and 1 traitor ($m = 1$):

Traitor: General #4

I'm commander #3. Sending message: true. m: 1

General #1's first order received: true
General #1 receiving order (true) from commander #3

General #2's first order received: true
General #2 receiving order (true) from commander #3

General #4's first order received: true
General #4 receiving order (true) from commander #3

### general #1 acting as commander ###

I'm commander #1. Sending message: true. m: 0

General #2 receiving order (true) from commander #1

General #4 receiving order (true) from commander #1

General #1 decision after round OM(0): true

### general #2 acting as commander ###

I'm commander #2. Sending message: true. m: 0

General #1 receiving order (true) from commander #2

General #4 receiving order (true) from commander #2

General #2 decision after round OM(0): true

### general #4 acting as commander ###

I'm commander #4. Sending message: true. m: 0

General #1 receiving order (false) from commander #4

General #2 receiving order (true) from commander #4

General #4 decision after round OM(0): true

deciding for general #1
Deciding majority for general #1: [true, true, false]
deciding for general #2
Deciding majority for general #2: [true, true, true]
deciding for general #4
Deciding majority for general #4: [true, true, true]
General #3 decision after round OM(1): true

total_messages: 9

successful: true

Note that this includes many more log messages than the code shown above. I thought it was really cool seeing the log messages as it helped a lot to visualize how the algorithm is behaving, just like a tree! You can find this version on the GitHub repo.

Let’s try 4 generals and 2 traitors ($m = 2$). Leslie states that it would be impossible to succeed given these assumptions (sync network model, permissioned, no cryptography), as we wouldn’t follow the $3m + 1$ bound. In other words, we can only succeed if we have $f \geq 3m + 1$, where $f$ is the number of generals and $m$ is the number of traitors. If we use $m = 2$, we’ll have

$$ f \geq 3*2 + 1 $$

$$ f \geq 6 + 1 \
$$

$$ f \geq 7 \
$$

We would need at least 7 generals in order to survive 2 traitors.

Let’s see how the algorithm behaves with 4 generals and 2 traitors:

Traitor: General #3

Traitor: General #4

I'm commander #1. Sending message: true. m: 2

General #2's first order received: true
General #2 receiving order (true) from commander #1

General #3's first order received: true
General #3 receiving order (true) from commander #1

General #4's first order received: true
General #4 receiving order (true) from commander #1

### general #2 acting as commander ###

I'm commander #2. Sending message: true. m: 1

General #3 receiving order (true) from commander #2

General #4 receiving order (true) from commander #2

### general #3 acting as commander ###

I'm commander #3. Sending message: true. m: 0

General #4 receiving order (false) from commander #3

General #3 decision after round OM(0): true

### general #4 acting as commander ###

I'm commander #4. Sending message: true. m: 0

General #3 receiving order (false) from commander #4

General #4 decision after round OM(0): true

deciding for general #3
Deciding majority for general #3: [true, true, false]
deciding for general #4
Deciding majority for general #4: [true, true, false]
General #2 decision after round OM(1): true

### general #3 acting as commander ###

I'm commander #3. Sending message: true. m: 1

General #2 receiving order (false) from commander #3

General #4 receiving order (true) from commander #3

### general #2 acting as commander ###

I'm commander #2. Sending message: true. m: 0

General #4 receiving order (true) from commander #2

General #2 decision after round OM(0): true

### general #4 acting as commander ###

I'm commander #4. Sending message: true. m: 0

General #2 receiving order (false) from commander #4

General #4 decision after round OM(0): true

deciding for general #2
Deciding majority for general #2: [true, false, false]
deciding for general #4
Deciding majority for general #4: [true, true, false, true, true]
General #3 decision after round OM(1): true

### general #4 acting as commander ###

I'm commander #4. Sending message: true. m: 1

General #2 receiving order (false) from commander #4

General #3 receiving order (true) from commander #4

### general #2 acting as commander ###

I'm commander #2. Sending message: false. m: 0

General #3 receiving order (false) from commander #2

General #2 decision after round OM(0): false

### general #3 acting as commander ###

I'm commander #3. Sending message: true. m: 0

General #2 receiving order (false) from commander #3

General #3 decision after round OM(0): true

deciding for general #2
Deciding majority for general #2: [true, false, false, false, false]
deciding for general #3
Deciding majority for general #3: [true, true, false, true, false]
General #4 decision after round OM(1): true

deciding for general #2
Deciding majority for general #2: [true, false, false, false, false]
deciding for general #3
Deciding majority for general #3: [true, true, false, true, false]
deciding for general #4
Deciding majority for general #4: [true, true, false, true, true]
General #1 decision after round OM(2): true

total_messages: 15

successful: false

Found a case where consensus isn't achieved

Well, I guess Leslie was right after all?

Change num_of_generals to 7, and it will succeed. Pretty cool, uh?

Where to go from here

Just sign the message, for god’s sake!

The problem and solution above are really fun and all. But, come on, it’s 2022, and we have cryptography. Why are we allowing messages to be forged by the traitors?

Removing the ability to forge messages simplifies the BGP a lot. The message looks like v:j:i where v is the value (e.g., either attack or retreat, true or false), signed by j, and then signed by `i'.

The implementation of this approach is really dependent on the actual business logic of the system, so I won’t be doing that now. But from the A1-A4 assumptions on the original paper, we can simplify the phrasing that states that to implement a BGP with signed messages, one must guarantee that:

• A1: Every message that is sent is delivered correctly;
• A2: The receiver of a message knows who sent it;
• A3: The absence of a message can be detected;
• A4: A loyal general’s signature cannot be forged, and any alteration of the contents of his signed messages can be detected. Anyone can verify the authenticity of a general’s signature.

And the end result with PKI (private-key infrastructure, cryptographically secure signatures) is that, instead of $f \geq 3m + 1$, it can tolerate up to 50% (or $n/2$) of faulty nodes, hence the famous 51% attack. A much better result by modifying the algorithm slightly to include signatures. Cryptography is awesome, isn’t it?

The coding exercise above was really nice to build intuition on the problem and its first solution, but the rabbit hole is really deep. One can continue by implementing signatures on the solution described here. Then, slowly relaxing other assumptions (synchronicity, permission model), making the problem and solution harder but more powerful and practical.

From a practical perspective, VMs allow users to safely run programs in an isolated environment: the emulated machine.

Why build a Virtual Machine from scratch

So, why build one when there are already so many great VMs out there? Simply put: To better understand how computers work. Personally, whenever I need to understand something complex enough that reading by itself won’t give me a deep understanding, I must build it. That which I cannot build, I cannot deeply understand.

I’ve found that learning how to build a simple VM is a great and seemingly underrated way to learn the basic philosophy behind assembly and assemblers.

Also, Justin Meiners and Ryan Pendleton already put out a great piece on building an LC-3 VM. I’m writing this document as a complement to this work. I go into a little more details and some of the underlying ideas I found tricky to understand. And as an alternative to writing an LC-3 in C, I’m doing it in Rust.

You can find the final source code here.

How to read this document

This guide is relatively long. VMs aren’t simple, and there is a lot of knowledge about it that, as I went through it, I found to be sort of “hidden,” or “heavy assumed to be known,” and that made understanding much harder for me.

For instance, I studied bitwise operations (such as shifts) a long time ago, I know the concept, but I have never had the chance to use it directly. And because of that, many tricks used in systems-level programming went straight over my head – these are “tribal” knowledge that people assume to be trivial, and therefore it is barely touched in these other documents.

I wanted to follow a different approach, a first-principles approach. I try to go over the most fundamental principles and tricks used to build a VM, with little prerequisite knowledge in systems programming other than entry-level Rust knowledge.

I’ve learned a lot along the way – systems programming is super fun. That said, let’s get to it.

The VM abstraction

The virtual machine will have two main components: registers and memory. It’s much simpler than it sounds.

Memory is where we will store a program’s binary representation; it’s a bounded array of bytes. Yup, that’s all there is to it, a good old array that will contain instructions and the indices are the addresses.

When loading an LC-3 program into memory, we do it at the address/index 0x3000 (12288 in decimal), the program section of the VM’s memory. Everything before this index is reserved for other things we’ll talk about later. Here’s the memory layout for the LC-3 memory:

Registers will store data that our opcode (Operation Code) will operate on. Physically, think of registers like a type of memory that’s much closer to the CPU than, say, the RAM.

Here we have 7 of these general-purpose registers. On top of that, we have two special purpose registers that I’ll talk about below.

Controlling the program flow with the Program Counter (PC) Register

The program counter (PC) stores the address of the next instruction in memory to be executed. In other words, it will point to an index/address in our array that represents the memory of the VM.

Manipulating this PC register is how we “walk” through each instruction and perform loops, conditional branching, and jumps; simply changing the index/address will affect the execution of the next instruction. For instance, in a loop instruction, we change the PC to go back a few “steps” until we meet a specific condition.

But how do we check for these logical conditions? Enters our next special purpose register.

Checking for logical conditions with the Condition Flags (Cond) Register

And talking about conditions brings us to the next special register: The condition flags register that tells us about the last computation. This register checks for conditions to perform control flow tasks (loops, ifs, etc.). It’s a straightforward one, and it might not immediately make sense how we use it, but all it does is store the signal (negative, zero, or positive) status of the last operation that happened within the machine. We’ll go over it in more detail later!

Structuring the project

This is how I’m structuring the project:

LC_3 git:(master) ✗ tree src
src
├── hardware
│   ├── instruction
│   │   └── mod.rs // Instruction-related code
│   ├── mod.rs // Glue-code for all the virtual hardware
│   ├── register
│   │   └── mod.rs // Register-related code 
│   └── vm
│       └── mod.rs // High-level VM-related code (uses the registers/instruction)
└── main.rs

This is dead simple to represent in Rust; In src/hardware/register/mod.rs we have the abstraction for our registers:

pub struct Registers {
    pub r0: u16,
    pub r1: u16,
    pub r2: u16,
    pub r3: u16,
    pub r4: u16,
    pub r5: u16,
    pub r6: u16,
    pub r7: u16,
    pub pc: u16,
    pub cond: u16,
}

impl Registers {
    pub fn new() -> Registers {
        Registers {
            r0: 0,        // general purpose register
            r1: 0,        // general purpose register
            r2: 0,        // general purpose register
            r3: 0,        // general purpose register
            r4: 0,        // general purpose register
            r5: 0,        // general purpose register
            r6: 0,        // general purpose register
            r7: 0,        // general purpose register
            pc: PC_START, // program counter
            cond: 0,      // condition flag
        }
    }

    pub fn update(&mut self, index: u16, value: u16) {
        match index {
            0 => self.r0 = value,
            1 => self.r1 = value,
            2 => self.r2 = value,
            3 => self.r3 = value,
            4 => self.r4 = value,
            5 => self.r5 = value,
            6 => self.r6 = value,
            7 => self.r7 = value,
            8 => self.pc = value,
            9 => self.cond = value,
            _ => panic!("Index out of bound"),
        }
    }

    pub fn get(&self, index: u16) -> u16 {
        match index {
            0 => self.r0,
            1 => self.r1,
            2 => self.r2,
            3 => self.r3,
            4 => self.r4,
            5 => self.r5,
            6 => self.r6,
            7 => self.r7,
            8 => self.pc,
            9 => self.cond,
            _ => panic!("Index out of bound. "),
        }
    }
}

Notice how we’re initializing the PC as: pc: PC_START,, where PC_START is 0x3000. That’s because, as we’ve seen in the specs, this is where the space in the VM’s memory reserved for the user’s program starts.

Also, notice that we’re explicitly using u16 (unsigned 16 bits) instead of the usual u8. We’ll talk about this soon enough!

Then, for our condition flag register, its goal is to store the information about the latest computation, which we will then use to test logical conditions. It’s a simple Enum, but it contains some tricks:

enum ConditionFlag {
    POS = 1 << 0, // Positive
    ZRO = 1 << 1, // Zero
    NEG = 1 << 2, // Negative
}

The << operator is a left bit shift operator. It’s simpler than it looks: n << k means we’re moving (or shifting) the bits in the binary representation of the number n by k. for instance, 1 in binary representation is 0000000000000001.

1 << 2 means we’re shifting the bits twice to the left so that the 1 at the end of 0000000000000001 will effectively move to the left, twice. It ends up being 0000000000000100, which in decimal representation is 4. Thus, 1 << 2 == 4. So why are we storing 1, 2, 4 here? Glad you asked. In binary, with 3 bits only:

• 1 == 001
• 2 == 010
• 4 == 100

So we’re playing with the possible conditional flags settings! Because the condition instruction will be nzp (neg, zero, pos) and only one can be set at a time, it will either be 001 (positive set nz1) 010 (zero set, n1p) 100 (negative set, 1zp). And these three binary values are 1, 2, and 4 in decimal base. How cool is that?!

Now we’ve gotta add one more method to our Registers implementation: one to update the condition register based on the last operation on a given (general purpose) register:

pub fn update_r_cond_register(&mut self, r: u16) {
    if self.get(r) == 0 {
        // Note that `9` is the register number 9, 
        // which is the cond register
        self.update(9, ConditionFlag::ZRO as u16);
    } else if (self.get(r) >> 15) != 0 {
        // a 1 in the left-most bit indicates negative
        self.update(9, ConditionFlag::NEG as u16);
    } else {
        self.update(9, ConditionFlag::POS as u16);
    }
}

What does the LC-3 compiled binary look like?

Now that we have the skeleton of our VM’s abstraction somewhat ready, let’s take a step back and see how an LC-3 compiled binary looks like.

Under examples/ we have an already compiled hello_world program for the LC-3. Let’s try reading it and seeing what it looks like:

let data = fs::read(args[1].clone()).expect("Unable to read file");

println!("data: {:?}\n", data);

Output:

data: [48, 0, 224, 2, 240, 34, 240, 37, 0, 72, 0, 101, 0, 108, 0, 108, 0, 111, 0, 32, 0, 87, 0, 111, 0, 114, 0, 108, 0, 100, 0, 33, 0, 0]

As expected: gibberish.

Okay, so, quickly taking a look at the fs::read docs we see that its signature is: pub fn read<P: AsRef<Path>>(path: P) -> io::Result<Vec<u8>> meaning, it returns a vector of 8 bits.

However, LC-3 operates with 16 bits per instruction!

We have to change how we’re reading the file to read it expecting 16 bits instead of 8.

Let’s see how the human-readable hello world program in LC-3 assembly looks like:

.ORIG x3000                        ; this is the address in memory where the program will be loaded
LEA R0, HELLO_STR                  ; load the address of the HELLO_STR string into R0
PUTs                               ; output the string pointed to by R0 to the console
HALT                               ; halt the program
HELLO_STR .STRINGZ "Hello World!"  ; store this string here in the program
.END                               ; mark the end of the file

The first line (.ORIG x3000) means that the first 16 bits read from the compiled binary tell the VM the position in memory where the program starts.

Back to reading and loading the compiled binary, but now expecting u16 instead of u8:

// Note the `read_u16`, forcing it to read as u16 and not u8.
let base_address = f.read_u16::<BigEndian>().expect("error");

let mut address = base_address as usize;
loop {
    match f.read_u16::<BigEndian>() {
        Ok(instruction) => {
            println!("address: {:?} instruction: {:?}\n", address, instruction);
            address += 1;
        }
        Err(e) => {
            if e.kind() == std::io::ErrorKind::UnexpectedEof {
                println!("OK")
            } else {
                println!("failed: {}", e);
            }
            break;
        }
    }
}

Endianness

Note that when we’re reading the 16 bits chunks, we tell Rust we want it as BigEndian: f.read_u16::<BigEndian>().

Without diving too deep into it, endianness is the order or direction in which the “machine” reads binary data. You can either read from left to right or right to left. The order is specific only to byte ordering and not bit ordering.

For instance, if we have only one byte, 0000 0010 (2 in decimal), it will be the same in big-endian or little-endian. However, the ordering is affected once we have more than 1 byte.

Big-endian means we’re storing the “big end” first, then the rest. Little-endian means storing the “little end” first. This big/little end thing means the most significant byte, the byte in a binary value that holds the biggest position.

In 00000010 (2), the left-most zero is the most significant one; changing it to 1 would cause the most significant change to the number, it would turn it into 10000010, which is 130 in decimal.

Changing the right-most zero would cause the smallest change to the number: 00000011 is 3.

Let’s consider a large 2 bytes number: 65,535, in binary: 1111 1111.

Since here we have two bytes, one is the most significant, the other is the least significant:

MSB  LSB
1111 1111

And now we reach the crux of this Big Endian vs. Little Endian understanding: Big Endian will first store the Most Significant Byte (MSB). Little Endian will store the Least Significant Byte (LSB) first.

Different machines, languages, implementations will use different endianness; such is life. LC-3 uses Big Endianness. That’s why we’re using f.read_u16::<BigEndian>().

Now let’s go back to the code above.

Running the code above passing the hello world binary file:

address: 12288 instruction: 57346
address: 12289 instruction: 61474
address: 12290 instruction: 61477
address: 12291 instruction: 72
address: 12292 instruction: 101
address: 12293 instruction: 108
address: 12294 instruction: 108
address: 12295 instruction: 111
address: 12296 instruction: 32
address: 12297 instruction: 87
address: 12298 instruction: 111
address: 12299 instruction: 114
address: 12300 instruction: 108
address: 12301 instruction: 100
address: 12302 instruction: 33
address: 12303 instruction: 0

Note that it’s showing 12288 as the base address, 12288 in hex is 0x3000, exactly how it’s in the original specs for the LC-3; how cool is that?!

We have to load the instruction 57346 in memory at the index 12288, 61474 at 12289, and so on.

Do you want to see something even cooler? In the hello world assembly, the first instruction is

LEA R0, HELLO_STR ; load the address of the HELLO_STR string into R0 PUTs

LEA (we’ll see what that is later) is the name of the instruction/opcode, let’s see the binary representation of that instruction in the spec:

Note that the OpCode is 1110.

Let’s see the output of our println! again:

address: 12288 instruction: 57346

So, load at 12288 (0x3000 in hex) the instruction 57346, this instruction in binary: 1110000000000010. Hey! The opcode is right there: 1110 000000000010.

The rest of the binary is the necessary information to perform the LEA operation, but you see that the OpCode is there. How cool is that?!

Now we have to load all these instructions coming from a program into the VM memory:

pub struct VM {
    pub memory: [u16; MEMORY_SIZE],
    pub registers: Registers,
}

impl VM {
    pub fn new() -> VM {
        VM {
            memory: [0; MEMORY_SIZE],
            registers: Registers::new(),
        }
    }
 
 pub fn write_memory(&mut self, address: usize, value: u16) {
        self.memory[address] = value;
    }
}
 

Note that we’re using a constant MEMORY_SIZE that we’re defining below. This constant is the maximum amount of memory we’re giving to our VM. We’ll be defining it as 65535, which is the max value for 16 bits.

And then our main.rs looks like this

let base_address = f.read_u16::<BigEndian>().expect("error");

// Here we're loading the program in memory
let mut address = base_address as usize;
loop {
    match f.read_u16::<BigEndian>() {
        Ok(instruction) => {
            vm.write_memory(address, instruction);
            address += 1;
        }
        Err(e) => {
            if e.kind() == std::io::ErrorKind::UnexpectedEof {
                println!("OK")
            } else {
                println!("failed: {}", e);
            }
            break;
        }
    }
}

In plain English: we start at the base address 0x3000, load the first instruction at this position, increment the address, load the next instruction, and so on.

Executing the program

Once we have a program loaded into the VM’s memory, running it is a matter of walking through the memory positions using the value stored in the Program Counter (PC) register.

use vm::VM;

pub const MEMORY_SIZE: usize = std::u16::MAX as usize;

pub fn execute_program(vm: &mut VM) {
    while vm.registers.pc < MEMORY_SIZE as u16 {
        // Read instruction
        let instruction = vm.read_memory(vm.registers.pc);

        // Increment program counter
        vm.registers.pc += 1;

        // Extract op_code and execute operation
        instruction::execute_instruction(instruction, vm)
    }
}

This leads to two important functions: read_memory and execute_instruction.

Reading the memory is pretty straightforward, at least at this stage:

pub fn read_memory(&mut self, address: u16) -> u16 {
 self.memory[address as usize]
}

Executing the instruction is about:

1. Detecting which instruction a given u16 value is, i.e., what’s the OpCode?
2. Matching it against all possible OpCodes
3. Running the matching OpCode and extracting the operands from the whole instruction.

The first two parts are pretty simple:

pub fn execute_instruction(instr: u16, vm: &mut VM) {
    // Extract OpCode from the instruction
    let op_code = get_op_code(&instr);

    // Match OpCode and execute instruction
    match op_code {
        Some(OpCode::ADD) => add(instr, vm),
        Some(OpCode::AND) => and(instr, vm),
        Some(OpCode::NOT) => not(instr, vm),
        Some(OpCode::BR) => br(instr, vm),
        Some(OpCode::JMP) => jmp(instr, vm),
        Some(OpCode::JSR) => jsr(instr, vm),
        Some(OpCode::LD) => ld(instr, vm),
        Some(OpCode::LDI) => ldi(instr, vm),
        Some(OpCode::LDR) => ldr(instr, vm),
        Some(OpCode::LEA) => lea(instr, vm),
        Some(OpCode::ST) => st(instr, vm),
        Some(OpCode::STI) => sti(instr, vm),
        Some(OpCode::STR) => str(instr, vm),
        Some(OpCode::TRAP) => trap(instr, vm),
        _ => {}
    }
}

Note that each OpCode is incredibly well specified in the original spec docs. Our goal is to implement one by one!

Extracting and handling our first instruction

Okay, how do we know which instruction is a u16 piece of data?

Let’s consider the first assembly instruction in our hello world program: LEA, it has, as expected, 16 bits:

And we want to take an instruction, grab the first 4 bits (the OpCode) and confirm it’s an LEA OpCode: 1110. Given the first instruction, if we print its decimal and binary representation, we get:

println!("instruction: {:?}\n", instruction);
println!("instruction in binary: {:#b}\n", instruction);

Output:

instruction: 57346
instruction in binary: 0b1110000000000010

Okay, so the OpCode is right there: 1110, how do we extract it from the whole instruction?

A common technique is to use bit shifting (>> or <<) to turn 1110000000000010 into 0000000000001110, which is the OpCode representation with the full leading zeroes!

That means we can right-shift the original number 12 times by using >>. Go on, count, and you will see that moving each element in that binary representation from its original position (1110000000000010) to the right, 12 times, will lead to 0000000000001110.

So if we do this:

println!("instruction: {:?}\n", instruction);

println!("instruction in binary: {:#b}\n", instruction);

println!("instruction >> 12: {:?}\n", instruction >> 12);

println!("instruction >> 12: {:#b}\n", instruction >> 12);

We get:

instruction: 57346

instruction in binary: 0b1110000000000010

instruction >> 12: 14

instruction >> 12: 0b1110

Now, here are all the OpCodes from the specs:

We now have to match the extracted 4 first bits of an instruction against the list of all OpCodes:

pub enum OpCode {
    BR = 0, // branch
    ADD,    // add
    LD,     // load
    JSR,    // jump register
    AND,    // bitwise and
    LDR,    // load register
    STR,    // store register
    RTI,    // unused
    NOT,    // bitwise not
    LDI,    // load indirect
    STI,    // store indirect
    JMP,    // jump
    RES,    // reserved (unused)
    LEA,    // load effective address
    TRAP,   // execute trap
}

pub fn get_op_code(instruction: &u16) -> Option<OpCode> {
    match instruction >> 12 {
        0 => Some(OpCode::BR),
        1 => Some(OpCode::ADD),
        2 => Some(OpCode::LD),
        3 => Some(OpCode::ST),
        4 => Some(OpCode::JSR),
        5 => Some(OpCode::AND),
        6 => Some(OpCode::LDR),
        7 => Some(OpCode::STR),
        8 => Some(OpCode::RTI),
        9 => Some(OpCode::NOT),
        10 => Some(OpCode::LDI),
        11 => Some(OpCode::STI),
        12 => Some(OpCode::JMP),
        13 => Some(OpCode::RES),
        14 => Some(OpCode::LEA),
        15 => Some(OpCode::TRAP),
        _ => None,
    }
}

Notice that we’re matching it against the decimal representation of the instruction that’s coming as u16.

Now our execute_instruction makes a lot more sense:

pub fn execute_instruction(instr: u16, vm: &mut VM) {
    // Extract OpCode from the instruction
    let op_code = get_op_code(&instr);

    // Match OpCode and execute instruction
    match op_code {
        Some(OpCode::ADD) => add(instr, vm),
        Some(OpCode::AND) => and(instr, vm),
        Some(OpCode::NOT) => not(instr, vm),
        Some(OpCode::BR) => br(instr, vm),
        Some(OpCode::JMP) => jmp(instr, vm),
        Some(OpCode::JSR) => jsr(instr, vm),
        Some(OpCode::LD) => ld(instr, vm),
        Some(OpCode::LDI) => ldi(instr, vm),
        Some(OpCode::LDR) => ldr(instr, vm),
        Some(OpCode::LEA) => lea(instr, vm),
        Some(OpCode::ST) => st(instr, vm),
        Some(OpCode::STI) => sti(instr, vm),
        Some(OpCode::STR) => str(instr, vm),
        Some(OpCode::TRAP) => trap(instr, vm),
        _ => {}
    }
}

And if we print the op_code for each instruction coming from the hello world program, we get:

op_code: Some(LEA)

op_code: Some(TRAP)

op_code: Some(TRAP)

Now let’s continue the implementation and implement only the necessary to run the hello world program: Some(OpCode::LEA) => lea(instr, vm),

LEA stands for Load Effective Address. As we’ve seen before, the first 4 bits are the OpCode identification (1110)

The other two are the Direct Register (DR) and the Program Counter (PC) offset.

The first instruction is doing the following:

LEA R0, HELLO_STR

So, effectively, we’re loading an address into a register. In this case, we’re loading the string HELLO_STR into the register R0 by loading the address where the HELLO_STR lives.

From the original specs:

An address is computed by sign-extending bits [8:0] to 16 bits and adding this value to the incremented PC. This address is loaded into DR. The condition codes are set, based on whether the value loaded is negative, zero, or positive.

And then the footnote:

The LEA instruction does not read memory to obtain the information to load into DR. The address itself is loaded into DR.

Extracting information from a binary encoded value

So, for the LEA instruction, here’s what the code needs to do.

First, get the DR portion of the instruction:

let dr = (instruction >> 9) & 0x7;

This is a trick to get the destination register portion of the instruction.

Let’s break this down. The instruction that we’ve received at LEA R0, HELLO_STR is 0b1110_000_000000010.

From left to right: 1110 is the OpCode, 000 is the Direct Register (DR), and 000000010 is the PCOffset9.

Then when we shift it to the right 9 times, we get:

original instruction: 0b1110_000_000000010
instruction >> 9:     0b0000000001110_000

So we’ve shifted enough that the last three bits are the direct register in the instruction: 000. The bits shifted “out” of the value sorta “disappear” from the value; in other words, they don’t rotate back to the beginning of the value.

Now that we have our “target” bits aligned at the end of the value, we apply a mask & 0x7 (0b0000000000000111). This mask is a bitwise AND against the instruction that we shifted.

And the result is:

instruction >> 9 & 0x7: 0b0000000000000000

So it’s the register 0, as expected.

But why do we mask? There are usually three reasons to apply bitwise masking:

• Bitwise AND to extract a subset of the bits in the value
• Bitwise OR to set a subset of the bits in the value
• Bitwise XOR to toggle a subset of the bits in the value

In the AND case, for instance:

Mask:   0000   1111
Value:  0101   0101
-------------------
Result: 0000   0101

The goal of this mask is, from left to right, to clear the first 4 bits and keep the last 4 bits of the value. So the result will be 0b00000101.

So in our LEA R0, HELLO_STR case, (instruction >> 9) & 0x7 is trying to say: shift it so that the last three bits are the direct register we’re looking for, but since the result of that instruction >> 9 isn’t the final direct register (only the last 3 bits), mask it with 0b0000000000000111 so that we clear everything up to the last three bits, which is what we want. The resulting 16 bits value will be exactly the direct register! How cool is that?! I find the intricate simplicity behind this technique beautiful.

Sign extension

Now back to the LEA spec:

An address is computed by sign-extending bits [8:0] to 16 bits and adding this value to the incremented PC.

Let’s see how this sign-extending thing works. Simply put, because we’re operating on 16 bits, whenever a value has less than 16 bits, we have to extend it to be 16 bits.

The address we’re loading with LEA, bits 8 to 0 (left to right), clearly isn’t 16 bits – so we have to sign-extend it!

For positive numbers, it’s easy: add zeroes before the value. This doesn’t work for negative numbers, though. Borrowing an example from Justin:

For example, -1 in 5 bits is 1 1111. If we just extended it with 0’s, this is 0000 0000 0001 1111, which is equal to 31

So instead of just filling in zeroes, we must also fill in ones for negative numbers. So this is how it looks:

// ...

let pc_offset = sign_extend(instruction & 0x1ff, 9);

// ...

fn sign_extend(mut x: u16, bit_count: u8) -> u16 {
    if (x >> (bit_count - 1)) & 1 != 0 {
        x |= 0xFFFF << bit_count;
    }
    x
}

More bitwise tricks! Let’s break this down.

We start off with

sign_extend(instruction & 0x1ff, 9);

0x1ff is 111111111, nine consecutive ones. Another mask! By instruction & 0x1ff, we mean to keep only the last 9 bits of the instruction, which happens to be the PCOffset9, the address we’re loading into a register, and the value that we have to sign-extend.

So here we’re just extracting the PCOffset9 portion from the instruction. We didn’t need to shift things around because PCOffset9 is already right-aligned.

Now, because it’s a 9-bit value, we want to extend it to be 16. Enters sign_extend function. Let’s break it down. You can check the formal definition for sign extension here: https://en.wikipedia.org/wiki/Sign_extension.

The short version is, if it’s a signed positive number, padding with zeroes will work. Adding zeroes won’t preserve the value if it’s a signed negative number, but padding with ones will.

The sign_extend is doing that. It takes a bit_count which is the number of bits in the original value; in the case of the LEA command, we’re dealing with a 9-bit value that we want to convert into 16-bit. That’s why we call it like this: let pc_offset = sign_extend(instruction & 0x1ff, 9);

The if clause if (x >> (bit_count - 1)) & 1 != 0 is testing the sign of the value. We’re moving x to the right up until the sign bit (bit_count - 1) and applying a single bitmask & 1 to grab it. Then check if it’s different than zero; if it is, it’s signed as 1 (negative), meaning we have to pad with ones instead of zeroes. Otherwise, it’s zero, and we return as is, as it already is padded with zeroes.

And the part where we pad with ones is:

x |= 0xFFFF << bit_count;

This part is tricky if you’re new to bitwise operations. 0xFFFF << bit_count is happening, then its result is |="d against x. |= means bitwise OR, but with an assignment step, so x will be OR’d against the result of 0xFFFF << bit_count and the result of that OR will be assigned to x, replacing the original value.

0xFFFF is 1111111111111111 (16 bits). We shift it left bit_count times, i.e., the number of bits in the original value x (in this case,9). Meaning that, for the original value, we’re going to “leave it alone” and only set 1’s to the rest, all the way to 16 bits total. This way, we’re going to have 1111111_<original x value>, and we’re done! Super clever trick.

Effectively, the LEA command in our hello world example is loading the address of the beginning of the string onto the register 0.

The last part is updating the cond register based on the value stored in the register that we last operated on, in this case, R0. Which looks like

vm.registers.update_r_cond_register(dr);

// ...

pub fn update_r_cond_register(&mut self, r: u16) {
    if self.get(r) == 0 {
        self.update(9, ConditionFlag::ZRO as u16);
    } else if (self.get(r) >> 15) != 0 {
        // a 1 in the left-most bit indicates negative
        self.update(9, ConditionFlag::NEG as u16);
    } else {
        self.update(9, ConditionFlag::POS as u16);
    }
}

enum ConditionFlag {
    POS = 1 << 0, // Positive
    ZRO = 1 << 1, // Zero
    NEG = 1 << 2, // Negative
}

Then, in our hello world example, we run a TRAP OpCode. Trap is an OpCode for interacting with IO devices, such as halting, output/input of data, etc. In other words, it’s where we go to do a very known thing called system calls!

The trap spec is unique in comparison to the other OpCodes:

1111 is the OpCode identifier, but then we have this thing called a trap vector, which is 8 bits long. The trap vector holds the identifier to which system call it wants to execute. The spec is super clear about how it works:

trap allows interacting with I/O devices. First R7 is loaded with the incremented PC. (This enables a return to the instruction physically following the TRAP instruction in the original program after the service routine has completed execution.) Then the PC is loaded with the starting address of the system call specified by trap vector8. The starting address is contained in the memory location whose address is obtained by zero-extending trap vector8 to 16 bits.

And then below:

Memory locations x0000 through x00FF, 256 in all, are available to contain starting addresses for system calls specified by their corresponding trap vectors. This region of memory is called the Trap Vector Table.

And below that we have the list of all “trap service routines”:

So our trap implementation will look like something like this

pub fn trap(instruction: u16, vm: &mut VM) {
    println!("trap instruction: {:#018b}\n", instruction);

    match instruction & 0xFF {
        0x20 => {
            // Get character
        }
        0x21 => {
            // Write out character
        }
        0x22 => {
            // Puts
        }
        0x23 => {
            // In, Print a prompt on the screen and read a single character from the keyboard. The character is echoed onto the console monitor, and its ASCII code is copied into R0.The high eight bits of R0 are cleared.
        }
        0x24 => {
            // Putsp
        }
        0x25 => {
   // Halt
        }
        _ => {
            process::exit(1);
        }
    }
}

And just like the “higher level” OpCodes (LEA, ADD, etc.), we have to implement one by one.

In our hello world example, the first trap used is PUTS, to print a string. The instruction binary representation is: 1111000000100010. So, 00100010 (the last 8 bits) is the trap vector identifier. 00100010 is 0x22 in hex, which maps to PUTS!

Notice that we do a match instruction & 0xFF { to find 0x22. Just like before, we do the & 0xFF (called a mask) to capture only the 0xFF worth of bits from the instruction, 0xFF is 11111111, 8 bits, which is the length of the trap vector we want to capture.

Here’s how the PUTS implementation looks like

x22 => {
 // Puts
 let mut index = vm.registers.r0;
 
 let mut c = vm.read_memory(index);
 
 while c != 0x0000 {
     print!("{}", (c as u8) as char);
     index += 1;
     c = vm.read_memory(index);
 }
 
 io::stdout().flush().expect("failed to flush");
}

We’re printing char by char, and the address of the beginning of the string lives in the register 0. So we’re walking through it.

Let’s visualize this better. If we print instruction:address pairs at the beginning, when we’re loading the hello world program into the VM’s memory, we see this:

address: 12288 instruction: 57346
address: 12289 instruction: 61474
address: 12290 instruction: 61477
address: 12291 instruction: 72
address: 12292 instruction: 101
address: 12293 instruction: 108
address: 12294 instruction: 108
address: 12295 instruction: 111
address: 12296 instruction: 32
address: 12297 instruction: 87
address: 12298 instruction: 111
address: 12299 instruction: 114
address: 12300 instruction: 108
address: 12301 instruction: 100
address: 12302 instruction: 33
address: 12303 instruction: 0

Notice how at the beginning of the address 12291 the instructions become slightly different. This is where the characters of the "hello world" string start being loaded at. Fun detail: these “instructions” start to look like utf8 encoding!

Let’s further confirm that 12291 is the beginning of the string:

let mut index = vm.registers.r0;
println!("index: {:?}\n", index);

index: 12291

Alright! Once we understand that, the reading loop becomes trivial:

while c != 0x0000 {
    print!("{}", (c as u8) as char);
    index += 1;
    c = vm.read_memory(index);
}

We’re just walking through each address until we see a 0x0000. And the print! over there is printing it to the stdout, that’s how we see Hello, world! coming from the register 0!

Then, the next trap instruction we see is HALT, 1111000000100101. Which is the simplest one to implement:

0x25 => {
    println!("HALT detected");
    io::stdout().flush().expect("Failed to flush");
    process::exit(1);
}

And that’s it for the hello world example:

  Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/LC-3 examples/hello-world.obj`
OK
Hello World!HALT detected

We just managed to run a binary file for the LC-3 computer in our VM!

Now, our next goal is to implement the rest of the OpCodes, add some more keyboard input control, and then we’ll be good to run more complex programs. It might sound complicated, but it isn’t. Now that we understand the logic and flow behind this VM, its bitwise operation tricks, how we use registers, etc., it’s now a matter of reading the spec for each OpCode and implementing its logic, which is usually very straightforward.

Implementing the ADD OpCode

Here’s the ADD spec:

We have two operation modes for ADD, and we can pick which one we’re gonna use by setting the bit [5] of the instruction. Here’s how it’s officially described:

If bit [5] is 0, the second source operand is obtained from SR2. If bit [5] is 1, the second source operand is obtained by sign-extending the imm5 field to 16 bits. In both cases, the second source operand is added to the contents of SR1 and the result stored in DR. The condition codes are set, based on whether the result is negative, zero, or positive

In other words, ADD will be adding two numbers; the first comes from a register, the second will come either from a register (SR2) or by directly passing an immediate value to the instruction, like:

ADD R2, R3, R4 ; R2←R3+R4, R2 receives the sum of what's in R3 and R4
ADD R2, R3, #7 ; R2←R3+7, R2 receives the sum of what's in R3 and the num 7

In the immediate case, it needs to be in 16 bits because it’s a full value, so we sign-extend it. The trade-off is that the instruction only has room for a small number, up to 2^5=32.

Here’s how the code looks like

pub fn add(instruction: u16, vm: &mut VM) {
    let dr = (instruction >> 9) & 0x7;

    // First operand
    let sr1 = (instruction >> 6) & 0x7;

    // Check if we're in immediate mode or register mode.
    let imm_flag = (instruction >> 5) & 0x1;
    if imm_flag == 1 {
        let imm5 = sign_extend(instruction & 0x1F, 5);
        //val is declared as u32 to prevent from overflow.
        let val: u32 = imm5 as u32 + vm.registers.get(sr1) as u32;
        //val is declared as u16, so that type arithmatic kick in and number is rounded to get fit into u16.
        vm.registers.update(dr, val as u16);
    } else {
        /* first operand (SR2) */
        let sr2 = instruction & 0x7;
        let val: u32 = vm.registers.get(sr1) as u32 + vm.registers.get(sr2) as u32;
        vm.registers.update(dr, val as u16);
    }

    vm.registers.update_r_cond_register(dr);
}

Let’s break it down.

let dr = (instruction >> 9) & 0x7;

This gets the destination address using bitwise operation tricks.

instruction >> 9 will shift the binary from the instruction it 9 times to the right. That means the last bit will be the end of the DR portion of the instruction. And the bitwise-and (&) 0x7 will grab only the length of 111 out of the instruction, i.e., the last 3 bits, which is precisely the length of the DR.

let sr1 = (instruction >> 6) & 0x7;

Same thing as before, but now we’re moving only 6 times because the SR1 register is after the direct register (DR).

let imm_flag = (instruction >> 5) & 0x1;

Again, same thing, just the one bit at position 5 that represents the operation mode.

Then we proceed with the operation mode logic:

if imm_flag == 1 {
    let imm5 = sign_extend(instruction & 0x1F, 5);

    // This is declared as u32 to prevent from overflow.
    let val: u32 = imm5 as u32 + vm.registers.get(sr1) as u32;

    // Set the result of the sum to the target register
    vm.registers.update(dr, val as u16);
} else {
    // If not immediate mode, we need to extract the second register.
    let sr2 = instruction & 0x7;

    // Proceed as usual
    let val: u32 = vm.registers.get(sr1) as u32 + vm.registers.get(sr2) as u32;

    // Set the result of the sum to the target register
    vm.registers.update(dr, val as u16);
}

If it’s the immediate mode, grab the immediate value by extending it to be 16 bits. Then add and update the target register (R0).

If not immediate mode, grab the other register passed in the instruction and proceed as usual.

And that’s it! Then we have to update the condition register:

vm.registers.update_r_cond_register(dr);

We pass dr here because this is the register containing the result of the last operation. Remember that the cond register’s idea is to set positive/negative/zero based on the result of the last operation, which in this case live in dr.

Implementing the LDI OpCode

In simple terms, the LDI operation is loading the address of a piece of data onto a register. However, it’s a bit trickier than that because it adds another layer of indirection (that’s why it’s called load indirect).

It starts by adding the PC to a PCOffset9 in the instruction. This sum is an address to a memory location, and that address contains another value which is the address of the value to load. It sounds tricky, but the code probably makes it clear. It looks like this:

pub fn ldi(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let dr = (instruction >> 9) & 0x7;

    // Get the PC offset and sign extend it to be 16 bits
    let pc_offset = sign_extend(instruction & 0x1ff, 9);

    let first_read = vm.read_memory(vm.registers.pc + pc_offset);
   
    let resulting_address = vm.read_memory(first_read);
    
    vm.registers.update(dr, resulting_address);
    vm.registers.update_r_cond_register(dr);
}

Implementing the AND OpCode

Good old logical and, but applied to binary values. This one is similar to ADD; it has two operation modes: immediate and register. Since you probably already got the hang of it, here’s the code without much explanation:

pub fn and(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let dr = (instruction >> 9) & 0x7;

    // As seen in `add` fn, same tricks.
    let sr1 = (instruction >> 6) & 0x7;
    let imm_flag = (instruction >> 5) & 0x1;

    if imm_flag == 1 {
        let imm5 = sign_extend(instruction & 0x1F, 5);
        // Perform the bitwise and (`&`) and store its value in the DR.
        vm.registers.update(dr, vm.registers.get(sr1) & imm5);
    } else {
        let sr2 = instruction & 0x7;
        // Perform the bitwise and (`&`) and store its value in the DR.
        vm.registers
            .update(dr, vm.registers.get(sr1) & vm.registers.get(sr2));
    }

    vm.registers.update_r_cond_register(dr);
}

Nothing new here, same old tricks, but instead of +ing stuff, we’re just bitwise-and’ing it.

Implementing the NOT OpCode

Another dead simple OpCode, simple binary negation. Won’t bore you with the details:

pub fn not(instruction: u16, vm: &mut VM) {
    let dr = (instruction >> 9) & 0x7;
    let sr1 = (instruction >> 6) & 0x7;
    vm.registers.update(dr, !vm.registers.get(sr1));

    vm.registers.update_r_cond_register(dr);
}

Implementing the BR OpCode

Finally, we’re using the condition register! We use a branch (BR) operation to control the flow of a program. There are many ways to use it:

Let’s look at the instruction encoding:

This one has some tricks that are worth digging into.

We have three bits set here: n for negative, z for zero, and p for positive. If any of these conditions are true, we’ll move the flow of the program to wherever LABEL is, which is a position stored in the PC register.

In a more concrete example, if the BR instruction has 101 for nzp, we’re saying that if the last instruction resulted in a negative number of a positive number, but not zero, then we trigger the branching, by moving the PC to LABEL.

This brings us to how we implement the condition flags in our VM:

enum ConditionFlag {
    POS = 1 << 0, // Positive
    ZRO = 1 << 1, // Zero
    NEG = 1 << 2, // Negative
}

Simple enough, but why these values set for POS, ZRO, and NEG?

1 in binary representation is 0000000000000001, 1 << 2 means we’re shifting the bits twice to the left, so that 1 at the end will effectively move to left twice. It ends up being 0000000000000100, which in decimal representation is 4. Thus, 1 << 2 == 4.

So why are we storing 1, 2, 4 here? Glad you asked. In binary, with 3 bits only: 1 == 001 2 == 010 4 == 100

So we’re playing with the possible conditional flags settings! Because the condition instruction will be nzp (neg, zero, pos) and only one can be set at a time, it will either be:

• 001 (positive set nz1)
• 010 (zero set, n1p)
• 100 (negative set, 1zp)

And these three binary values are 1, 2, and 4 in decimal!

Now, our branching operation looks like this:

pub fn br(instruction: u16, vm: &mut VM) {
    // Grab the PCOffset of the instruction and sign extend it
    // You can read more sign extension inside the `sign_extend` fn.
    let pc_offset = sign_extend((instruction) & 0x1ff, 9);

    // Shift 9 and grab 3 bits (& 0x7 is doing that)
    // You can read more about this trick inside `lea` fn.
    let cond_flag = (instruction >> 9) & 0x7;

    if cond_flag & vm.registers.cond != 0 {
        let val: u32 = vm.registers.pc as u32 + pc_offset as u32;
        vm.registers.pc = val as u16;
    }

    // If the branch isn't taken (no condition met), PC isn't changed
    // and PC will just point to the next sequential instruction.
}

And we do the testing there as: f cond_flag & vm.registers.cond != 0 {

We’re taking the 001, or 010, or 100 stored in the condition register and &ing it to the 001, or 010, or 100 coming from the instruction; note that the one coming from the instruction can be 110, 111 or any combination.

For example, if the last operation turned positive, we have 001 in the condition register. If the condition to trigger the branching is 011 (either positive or zero), then 001 & 011 will be 001, which means the branching is enabled, meaning we will change the PC by adding the PCOffset to it and moving on. The next instruction executed will be decided by this new PCOffset; this is usually used for while/for loops and if-statements.

Implementing the JMP OpCode

JMP is another flow control operation, like BR, but without conditions. So we’re mutating the PC register no matter what.

The program unconditionally jumps to the location specified by the content of the base register in the instruction. Bits [8:6] identify the base register.

RET is listed as a separate instruction in the specification since it is a different keyword in assembly. However, it is a particular case of JMP. RET happens whenever R1 is 7.

This is how the encoding looks like:

In our implementation, RET and JMP are going to be handled by the same JMP function, as they’re basically the same, just different assembly keywords, and RET is always when the register is 111 (register 7) instead of an arbitrary register BaseR..

pub fn jmp(instruction: u16, vm: &mut VM) {
    // base_reg will either be an arbitrary register or the 
    // register 7 (`111`) which in this 
    // case it would be the `RET` operation.
    let base_reg = (instruction >> 6) & 0x7;
    vm.registers.pc = vm.registers.get(base_reg);
}

Implementing the JSR OpCode

Also known as JumptoSubRoutine. The spec itself for this one is incredibly clear:

First, the incremented PC is saved in R7. This is the linkage back to the calling routine. Then the PC is loaded with the address of the first instruction of the subroutine, causing an unconditional jump to that address. The address of the subroutine is obtained from the base register (if bit [11] is 0), or the address is computed by sign-extending bits [10:0] and adding this value to the incremented PC (if bit [11] is 1).

The code, then, is pretty straightforward:

pub fn jsr(instruction: u16, vm: &mut VM) {
    // Grab the base register
    let base_reg = (instruction >> 6) & 0x7;

    // 0x7ff == 11111111111 (11 ones, exactly the length of PCOffset11)
    // Grab it and extend it to 16 bits.
    let long_pc_offset = sign_extend(instruction & 0x7ff, 11);

    // Grab the flag bit at [11] and test it
    let long_flag = (instruction >> 11) & 1;

    // Save the incremented PC in R7
    vm.registers.r7 = vm.registers.pc;

    if long_flag != 0 {
        // JSR case, the address to jump is computed from PCOffset11
        let val: u32 = vm.registers.pc as u32 + long_pc_offset as u32;
        vm.registers.pc = val as u16;
    } else {
        // JSRR case, address to jump to lives in the base register
        vm.registers.pc = vm.registers.get(base_reg);
    }
}

Implementing the LD OpCode

The standard load operation. Incredibly simple one!

We’re loading onto a register DR the value stored in a place in memory computed using the PCOffset9.

pub fn ld(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let dr = (instruction >> 9) & 0x7;

    // Grab the PCOffset and sign extend it
    let pc_offset = sign_extend(instruction & 0x1ff, 9);

    let mem: u32 = pc_offset as u32 + vm.registers.pc as u32;

    // Read the value from the place where the memory above was computed
    let value = vm.read_memory(mem as u16);

    // Save that value to the direct register and update the condition register
    vm.registers.update(dr, value);
    vm.registers.update_r_cond_register(dr);
}

Implementing the LDR OpCode

It stands for Load Base+Offset; it’s a specialized version of the LD operation. The only difference is that instead of only using a base register, we add the base register to an Offset6 (sign-extended to 16 bits) before loading.

pub fn ldr(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let dr = (instruction >> 9) & 0x7;
    
    // Grab the base register 
    let base_reg = (instruction >> 6) & 0x7;

    // Grab the offset6 and sign extend it
    let offset = sign_extend(instruction & 0x3F, 6);

    // Compute the memory location to be loaded
    let val: u32 = vm.registers.get(base_reg) as u32 + offset as u32;

    // Read the value at that memory location
    let mem_value = vm.read_memory(val as u16).clone();

    // Update the register with the loaded value and update the condition register
    vm.registers.update(dr, mem_value);
    vm.registers.update_r_cond_register(dr);
}

Implementing the ST OpCode

Another simple operation: Store. It will store the contents in the register SR in the memory location computed by the current PC and the PCOffset9. As usual, the offset will be sign-extended to 16-bit.

The code for it is as easy as the description:

pub fn st(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let sr = (instruction >> 9) & 0x7;

    // Grab the PC offset and sign extend it
    let pc_offset = sign_extend(instruction & 0x1ff, 9);

    // Add the current PC to the PC offset
    // We're doing these conversions to avoid overflow
    let val: u32 = vm.registers.pc as u32 + pc_offset as u32;
    let val: u16 = val as u16;

    // Store the value in the register being passed in the instruction at
    // the address computed above
    vm.write_memory(val as usize, vm.registers.get(sr));
}

Implementing the STI OpCode

Store indirect, just like STORE, but with a layer of indirection. The encoding is very similar, though:

This part of the official spec covers it well IMO:

What is in memory at this address is the address of the location to which the data in SR is stored

It kind of sounds like pointers, right?!

The code is pretty much the same as ST (Store) with one extra step:

pub fn sti(instruction: u16, vm: &mut VM) {
    // Get the direct register encoded in the instruction (see `add` fn for more in-depth details)
    let sr = (instruction >> 9) & 0x7;

    // Grab the PC offset and sign extend it
    let pc_offset = sign_extend(instruction & 0x1ff, 9);
    
    // Add the current PC to the PC offset
    // We're doing these conversions to avoid overflow
    let val: u32 = vm.registers.pc as u32 + pc_offset as u32;
    let val: u16 = val as u16;

    // This is the difference between STI and ST
    let address = vm.read_memory(val) as usize;

    vm.write_memory(address, vm.registers.get(sr));
}

Implementing the STR OpCode

Store base+offset. Similar to Load Base+Offset, but store instead of load. The instruction has a base register and offset, and we add them together to get the place where we want to store the values in the register SR.

The code is also fairly simple:

pub fn str(instruction: u16, vm: &mut VM) {
    // Get the register encoded in the instruction (see `add` fn for more in-depth details)
    let dr = (instruction >> 9) & 0x7;

    // Grab the base register
    let base_reg = (instruction >> 6) & 0x7;

    // Grab the offset and sign extend it
    let offset = sign_extend(instruction & 0x3F, 6);

    // Get the value in the base_register and sum it to the offset encoded in the instruction
    // Note that we're doing some conversions here to prevent overflow.
    let val: u32 = vm.registers.get(base_reg) as u32 + offset as u32;
    let val: u16 = val as u16;
    vm.write_memory(val as usize, vm.registers.get(dr));
}

And that’s it! We’ve implemented all the operations from the spec, and we can now run more complex programs.

However, we will need to add more IO control stuff to our VM, like handling keyboard input and such. Let’s do that!

Memory-mapped registers

It’s common to create special registers to interact with peripherals, like a keyboard and mouse. These registers are only accessed by reading and writing from a reserved memory location.

In LC-3, we have two of those special memory-mapped registers:

pub enum MemoryMappedReg {
    // Keyboard status: The KBSR indicates whether a key has been pressed
    Kbsr = 0xFE00,

    // Keyboard data: The KBDR identifies which key was pressed
    Kbdr = 0xFE02,
}

One for keyboard status (i.e., is key pressed?) and another for keyboard data (i.e., which key pressed?). Why not just use getc to get this information? I like Justin Meiners’s explanation about it:

Although you can request keyboard input using GETC, this blocks execution until input is received. KBSR and KBDR allows you to poll the state of the device and continue execution, so the program can stay responsive while waiting for input.

Okay, so when do we read/write from/to these registers? Simple enough, whenever we’re reading the VM’s memory, we will perform this check. That means we’ll be extending our read_memory function from the beginning, which was:

pub fn read_memory(&mut self, address: u16) -> u16 {
    self.memory[address as usize]
}

And it will become this:

pub fn read_memory(&mut self, address: u16) -> u16 {
    if address == MemoryMappedReg::Kbsr as u16 {
        self.handle_keyboard();
    }
    self.memory[address as usize]
}

fn handle_keyboard(&mut self) {
    let mut buffer = [0; 1];
    std::io::stdin().read_exact(&mut buffer).unwrap();
    if buffer[0] != 0 {
        self.write_memory(MemoryMappedReg::Kbsr as usize, 1 << 15);
        self.write_memory(MemoryMappedReg::Kbdr as usize, buffer[0] as u16);
    } else {
        self.write_memory(MemoryMappedReg::Kbsr as usize, 0)
    }
}

So, if the address we’re reading is the address of the keyboard status register, we’ll enter this “keyboard handling mode,” where we’ll write the status and the keyboard data to these special registers.

With that done, we can now even run a game called Rogue in our LC-3 VM:

The compiled object for the Rogue program is inside examples/ in this project’s repository. To run it: cargo run -- examples/rogue.obj.

Where to go from here

We have a working Virtual Machine for the LC-3 computer, that’s awesome. We’re able to run any compiled program that’s supposed to run on LC-3 on our VM—This is super powerful!

However, we have one problem: we depend on compiled programs to do stuff with our VM. It would be nice to write LC-3 assembly in plaintext and compile it to the LC-3 architecture. This is something I’d like to work on at some point as a continuation of this project—that and a disassembler, i.e., from the compiled .obj files to plaintext assembly.

I might work on that if I have the time and do another write-up that would be a follow-up to this one.

Here is a simple pattern to apply graceful shutdowns when the program has some level of concurrency in it. If you want to browse the code, you can skip the explanation and get them here.

A simple scenario

Let’s consider a simple scenario where we have an HTTP endpoint that takes a job name, asynchronously starts 3 slow jobs related to this job (could be writing to a DB, S3, Kafka, and whatnot), and then responds a simple “job started” to the caller. The moment we return an answer to the caller doesn’t mean it has finished the 3 slow jobs — they’re running in the background, inside goroutines.

This is a “mock” code that represents this scenario:

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"
)


/////////////////////////////////////////////////////////////
/// 3 functions that take some time to run something fictitious
func slowJob1(name string) {
	fmt.Printf("starting job 1 for %s\n", name)
	time.Sleep(5 * time.Second)
	fmt.Printf("finished job 1 for %s\n", name)

}

func slowJob2(name string) {
	fmt.Printf("starting job 2 for %s\n", name)
	time.Sleep(4 * time.Second)
	fmt.Printf("finished job 2 for %s\n", name)
}

func slowJob3(name string) {
	fmt.Printf("starting job 3 for %s\n", name)
	time.Sleep(3 * time.Second)
	fmt.Printf("finished job 3 for %s\n", name)
}
/////////////////////////////////////////////////////////////

func handler(w http.ResponseWriter, r *http.Request) {
	jobName := r.URL.Path[1:]
	
	// Spawn three goroutines that will run in the background
	// even after this function returns
	go slowJob1(jobName)
	go slowJob2(jobName)
	go slowJob3(jobName)
	
	// Write back to the caller
	fmt.Fprintf(w, "job %s started", r.URL.Path[1:])
}

func main() {
	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Now let’s imagine that this endpoint gets called many times per second. That means we’re going to have many goroutines working in the background at any given time.

However, here’s an important contract we’ve implicitly made with our clients: once the job is started, it must be finished (well well well… what a surprise?). So, once we return “job $name started” to the client, they can be confident the service will execute it.

But, even assuming the server won’t ever crash and burn, in cases of scheduled maintenance or deployment rollouts, this service will be shut down at some point so that updated versions of it can be rolled out. So what happens with our service, right now, if we send a SIGTERM or SIGINT to its PID? Let’s see!

Oops. Seems like a CTRL-c (SIGINT) or a SIGTERM (coming from a docker stop or something) would force the service to exit before finishing the already started jobs; that’s not good.

Here’s where proper shutdown is crucial. In this scenario, when we receive a SIGTERM, what we want is:

1. Stop accepting new requests
2. Finish the jobs we already started
3. Exit

Important: note that if the HTTP requests were being handled synchronously, i.e., keeping the connection alive, a simple shutdown() in the HTTP server would suffice. It would wait for the handlers that are still working before shutting down the server. The tricky part here, however, is that the HTTP endpoint is acting as a trigger to more long-lived background-running jobs - a typical pattern. The HTTP connection between the client and server will be long gone before the jobs actually finish.

Signal handling and server shutdown

First, we need to capture these UNIX signals, such as SIGTERM and SIGINT. Golang gives you a nice abstraction that takes the signals you wanna listen to and a channel that it will use to notify you. You, then, read from this channel to get notified about the signals you’re listening to.

// Handle sigterm and await termChan signal
termChan := make(chan os.Signal)
signal.Notify(termChan, syscall.SIGTERM, syscall.SIGINT)

go func() {
	<-termChan // Blocks here until interrupted
	log.Print("SIGTERM received. Shutdown process initiated\n")
	httpServer.Shutdown(context.Background())
}()

$ go-complex-shutdown go run main.go
^C2020/10/21 17:42:52 SIGTERM received. Shutdown process initiated
2020/10/21 17:42:52 HTTP server shut down

Note the ^C, meaning I just slapped a CTRL-C in there, sending a SIGTERM to the process. Now we’re capturing it and shutting down the server. But we’re still not waiting for the goroutines to finish their job.

Gracefully shutting down the server

Now we’re capturing the SIGTERMs and shutting down the HTTP server. Let’s now wait for the running goroutines to finish before exiting the program.

To do that, we’re going to use the good old sync.WaitGroup. We want to pass a waitgroup into the handler itself. For that, we’ll have to create a custom handler struct that implements ServeHttp(w http.ResponseWriter, r *http.Request). Then we move the logic from the previous handler into this new ServeHTTP function

// Our custom handler that holds a wait group used to 
// block the shutdown while it's running the jobs.
type CustomHandler struct {
	wg *sync.WaitGroup
}

func NewCustomHandler(wg *sync.WaitGroup) *CustomHandler {
	// You can check for wg == nil if feeling paranoid
	return &CustomHandler{wg: wg}
}

func (h *CustomHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	jobName := vars["jobName"]

	fmt.Fprintf(w, "job %s started", jobName)
	
	// Here's where the magic happens. 
	// We'll use the *WaitGroup inside the handler struct to add 3 to it
	// and pass in the waitgroup to the slowJobs, they'll have to wg.Done()
	// once they return. 
    // Also keep in mind that this will happen for every HTTP 
    // request coming in.
	// So at any given time, we'll have `3 * n` in the "global" WaitGroup,
	// where `n` is the number of requests.  
	h.wg.Add(3)
	go slowJob1(jobName, h.wg)
	go slowJob2(jobName, h.wg)
	go slowJob3(jobName, h.wg)
}

And now our slowJob functions will look like this:

func slowJob1(name string, wg *sync.WaitGroup) {
	defer wg.Done()

	fmt.Printf("starting job 1 for %s\n", name)
	time.Sleep(5 * time.Second)
	fmt.Printf("finished job 1 for %s\n", name)
}

And that’s about it. Let’s see how it behaves once we interrupt the program:

Awesome, working just as intended — no jobs interrupted while shutting down and no new jobs started! The idea behind how we achieved this is fairly simple and you can easily replicate this pattern elsewhere. In my experience, this can be tricky to implement at later stages of the project. More things are going on, more concurrency, more layers. Introducing this capability, then, becomes quite a surgical task. Just imagine passing WaitGroup (or context objects) deeper into different layers. It’s not impossible, plus, if it makes our systems more reliable, predictable, and fault-tolerant, it’s usually worth the effort. But doing it early on is definitely a better approach.

Full code

Here’s the full code for what we’ve created so far:

package main

import (
	"context"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"

	"log"

	"github.com/gorilla/mux"
)

func slowJob1(name string, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("starting job 1 for %s\n", name)
	time.Sleep(5 * time.Second)
	fmt.Printf("finished job 1 for %s\n", name)
}

func slowJob2(name string, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("starting job 2 for %s\n", name)
	time.Sleep(4 * time.Second)
	fmt.Printf("finished job 2 for %s\n", name)
}

func slowJob3(name string, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("starting job 3 for %s\n", name)
	time.Sleep(3 * time.Second)
	fmt.Printf("finished job 3 for %s\n", name)
}

// Our custom handler that holds a wait group used to block the shutdown 
// while it's running the jobs.
type CustomHandler struct {
	wg *sync.WaitGroup
}

func NewCustomHandler(wg *sync.WaitGroup) *CustomHandler {
	// You can check for wg == nil if feeling paranoid
	return &CustomHandler{wg: wg}
}

func (h *CustomHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	jobName := vars["jobName"]

	fmt.Fprintf(w, "job %s started", jobName)

	h.wg.Add(3)
	go slowJob1(jobName, h.wg)
	go slowJob2(jobName, h.wg)
	go slowJob3(jobName, h.wg)
}

func main() {
	wg := &sync.WaitGroup{}
	customHandler := NewCustomHandler(wg)

	router := mux.NewRouter()
	router.Handle("/{jobName}", customHandler)

	httpServer := &http.Server{
		Addr:    ":8080",
		Handler: router,
	}

	// Handle sigterm and await termChan signal
	termChan := make(chan os.Signal)
	signal.Notify(termChan, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		<-termChan // Blocks here until interrupted
		log.Print("SIGTERM received. Shutdown process initiated\n")
		httpServer.Shutdown(context.Background())
	}()

	// Blocking
	if err := httpServer.ListenAndServe(); err != nil {
		if err.Error() != "http: Server closed" {
			log.Printf("HTTP server closed with: %v\n", err)
		}
		log.Printf("HTTP server shut down")
	}
	

	// This is where, once we're closing the program, we wait for all
	// jobs (they all have been added to this WaitGroup) to `wg.Done()`.
	log.Println("waiting for running jobs to finish")
	wg.Wait()
	log.Println("jobs finished. exiting")
}

A more complex scenario

Now, another typical pattern is to send the jobs straight to a shared job queue that’ll be consumed by a consumer process (likely in a separate goroutine), now it looks like this:

Let’s not get into the discussion of keeping all jobs in memory for now — a better approach would be writing these jobs to a Kafka topic — but the principle of graceful shutdown works roughly the same way for both. Still, with an in-memory job queue, we can easily demonstrate how to gracefully shut it down.

Once we get a SIGTERM, we want to:

1. Stop the HTTP handler from writing more jobs to the job queue
2. Stop the HTTP server from accepting new connections
3. Drain the job queue by running the leftover jobs in there
4. Actually shut down the program

Fortunately, Golang provides us with a nice construct for this: channel and controlling the flow to it is just a matter of closing it.

This is how our consumer looks like:

func consumer(jobQueue chan string) {
	wg := &sync.WaitGroup{}

	for job := range jobQueue {
		wg.Add(3)
		go slowJob1(job, wg)
		go slowJob2(job, wg)
		go slowJob3(job, wg)
	}

	log.Println("Waiting for running jobs to finish")
	wg.Wait()
	log.Println("Done, shutting down the consumer")
}

The slowJobN functions are the same from the previous scenario.

Now our handler simply writes to a channel that’s acting as our job queue:

func (h *CustomHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	jobName := vars["jobName"]

	h.jobQueue <- jobName

	fmt.Fprintf(w, "job %s started", jobName)
}

And our main gets a wee bit more complicated, now that the consumer should be the blocking part of this workflow:

func main() {
	jobQueue := make(chan string)

	customHandler := NewCustomHandler(jobQueue)

	router := mux.NewRouter()
	router.Handle("/{jobName}", customHandler)

	httpServer := &http.Server{
		Addr:    ":8080",
		Handler: router,
	}

	// Handle sigterm and await termChan signal
	termChan := make(chan os.Signal)
	signal.Notify(termChan, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		<-termChan // Blocks here until interrupted
		log.Println("SIGTERM received. Shutdown process initiated")

		log.Println("stopping the consumer")

		// This will force the consumer to stop its main loop
		close(jobQueue) 

		httpServer.Shutdown(context.Background())
	}()

	go func() {
		if err := httpServer.ListenAndServe(); err != nil {
			if err.Error() != "http: Server closed" {
				log.Printf("HTTP server closed with: %v\n", err)
			}
			log.Printf("HTTP server shut down")
		}
	}()

	// Now the consumer is the blocking part
	consumer(jobQueue)
}

And the trick is simply grabbing the SIGTERM and closing the jobQueue:

close(jobQueue) 

This means, from a producer (the handler) perspective: you can’t write to the channel anymore (can’t write to closed channels). And from a consumer perspective: it can read the data inside the channel, even if it was closed, but once we reach the last bit of data in there, the for-loop will break.

Once we’ve started all the slowJobNs, and after the for-loop breaks, we’ll be waiting on that wg.Wait(). Meaning that we’ve drained the jobQueue and waited for all of them to finish before exiting the program.

You can find the full code here: https://gist.github.com/digorithm/6ea1b0a129bea2ce4fac404143738cc5#file-jobqueue-go

Making this code better

Thanks to some comments, I’ve noticed a few issues with this last iteration, namely:

1. Calling httpServer.Shutdown() in a goroutine can cause some weird issues.
2. There’s a race condition between the goroutine that closes the channel (close(jobQueue)), and the handler writing to this channel (h.jobQueue <- jobName). That means the handler could write to a closed channel and the program would panic.
3. Context’s cancellation is a much better way to propagate shutdown signals.

So let’s do another iteration here and fix some loose ends.

For the main, let’s create a context.WithCancel and a channel that we’ll pass to the consumer and it will use to tell us when the jobs are finished, and it’s safe to exit the program:

ctx, cancel := context.WithCancel(context.Background())

// ...
// ... same as before
// ...

// doneChan will be the channel we'll be listening on
// to know all already started jobs have finished
// before we actually exit the program
doneChan := make(chan interface{})
go consumer(ctx, jobQueue, doneChan) // <- not blocking anymore

Now, to solve the first problem ("Calling httpServer.Shutdown() in a goroutine can cause some weird issues"), we won’t block on the consumer anymore. Instead, we’ll run the consumer on a separate goroutine and move our shutdown logic to be inside the main() itself:

// Wait for SIGTERM to be captured
<-termChan
log.Println("SIGTERM received. Shutdown process initiated")

// Shutdown the HTTP server
if err := httpServer.Shutdown(ctx); err != nil {
	log.Fatalf("Server Shutdown Failed:%+v", err)
}

// Cancel the context, this will make the consumer stop
cancel()

// Wait for the consumer's jobs to finish
log.Println("waiting consumer to finish its jobs")
<-doneChan
log.Println("done. returning.")

Now upon a SIGTERM we’ll cancel() the context, which will trigger the consumer to stop, then we’ll wait on doneChan for the running jobs to finish. Now our consumer looks like this:

func consumer(ctx context.Context, jobQueue chan string, 
									doneChan chan interface{}) {
	wg := &sync.WaitGroup{}

	for {
		select {
		// If the context was cancelled, a SIGTERM was captured
        // So we wait for the jobs to finish, 
        // write to the done channel and return
		case <-ctx.Done():
            // Note that the waiting time here is unbounded 
            // and can take a long time. If that's an issue you can:
			// (1) issue a SIGKILL after a certain time or
			// (2) use a context with timeout
			wg.Wait()
			fmt.Println("writing to done channel")
			doneChan <- struct{}{}
			log.Println("Done, shutting down the consumer")
			return
		case job := <-jobQueue:
			wg.Add(3)
			go slowJob1(job, wg)
			go slowJob2(job, wg)
			go slowJob3(job, wg)
		}
	}
}

Much cleaner and self-contained, isn’t it? Now once we cancel() the context, the case <-ctx.Done() will be selected, and we’ll not consume from the jobQueue anymore, wait for the running ones to finish, then let the main thread know the consumer is done. This solves the potential risk of writing to a closed channel (the channel isn’t being closed anymore). With a context with cancellation coming from the entrypoint, we can pass it further down in more complex scenarios.

Full code here: https://gist.github.com/digorithm/fc68573db5618bb6d57d090fd66528bd

Thanks to Kenny Louie and epsleq0 for reviewing this piece.

I joined this project from the very beginning. When I joined, a couple of technical decisions had been made. Initially, before I’d arrived on this project, the microservices would talk to each other directly, REST/gRPC style. Nothing new here.

Simple architecture where the services talk to each other directly

The great majority of these services communicated with each other through synchronous calls. In most scenarios, this would be 100% fine. In an ETL constantly streaming copious amounts of data that need fast processing though? Not so much.

One big problem with this architecture is that this application isn’t an “user-triggered” application and the events happens in a streaming fashion, i.e this is a streaming system, synchronous calls then quickly become a bottleneck. For instance: to process property X we’d need to wait for all other services to do their jobs before moving onto the property X + 1.

And because of that property (streaming system), it means we can go full asynchronous here; we want to stream data in a constant flow and each service will do its job independently.

Direct communication through an API gateway vs. streaming system using message-passing (fully async)

However, an unfortunate bad technical decision had been made before I arrived on this project: yes, they would go from sync requests to async requests, but they would do it through a service developed internally called Queuer, which, in hindsight, was nothing like a queue. Queuer would just take the request coming from one service, buffer it, and route it to another service (again, not a queue), which sounded more like what you’d get from a service mesh, but without all the reliability that you’d get from battle-tested robust tools. We had something like this:

A queueing system that acts exactly like an API gateway

We had many problems with this:

1. Maintaining a “queueing” system built in-house was terrible.
2. Queuer wasn’t service-agnostic; we had to encode domain knowledge about other services inside Queuer.
3. My favourite: Queuer had a buffer, which was simply a in-memory variable. It would buffer thousands of jobs there. If queuer died, well, the jobs were lost. Oh and forget about deliver-once guarantees and/or other goodies you can get by using a battle-tested queueing system such as RabbitMQ or Kafka.

About 3 months into the project I convinced people to move away from Queuer, it had become too much of a hassle to maintain and evolve it. Godspeed Queuer.

Also, around that time I was reading designing data-intensive data applications. It advocates – with very good arguments – for what’s called Message-Passing Dataflow. I’d seen this somewhere else with the name event-centric architecture or something like that; but I believe the philosophies between them are very similar. I like the simple way the author explains it, so here it is:

Composing stream operators into dataflow systems has a lot of similar characteristics to the microservices approach. However, the underlying communication mechanism is very different: one-directional, asynchronous message streams rather than synchronous request/response interactions.

I then decided to experiment with Message-Passing Dataflow; after all, we’re talking about a never-ending stream processing system. Therefore, I didn’t want much synchronous communication between the services and wanted something that’s closer to an actual streaming system.

To achieve this, we need a central component that would act as a communication bus, streaming messages between the services. And the services, instead of being just HTTP rest APIs, would be active consumers, consuming messages from this bus.

Services consume messages being passed to the channels they’re subscribed to

Not surprisingly, I decided to use Kafka as our communication bus. The services around Kafka are Kafka consumers and producers. All communication between the services happens through message passing.

Here are some observations after adopting this paradigm:

Async usually means that it will be faster

I noticed that this approach is faster that what we had before. A service does one thing and does it as fast as it can and don’t have to wait for the next steps. Then, it might produce a message to Kafka, which will be a job being performed by other services, and that’s it.

More robustness

This approach is much, much more robust. All services are completely stateless and they don’t hold jobs in memory. The service died? All good, the job can be easily recovered from Kafka, Kafka only moves its offsets when the offsets are commited, i.e when the service actually finishes the job.

You must think about how you handle the offsets

Even though it’s robust, consumers must be smart about it. Consuming offsets from topics (think of it as jobs in a queue) and failing to commit the new offset properly can be catastrophic. For instance, if we have a Kafka producer’s auto.commit enabled, the service reads a message from a topic, the auto commit mechanism commits after a few milliseconds, but then the process fail. That means we marked that offset as consumed, but we didn’t finished the job, so it’s… well, lost. Manually committing the offsets is much more reliable in high risk cases.

A different and refreshing computational model

It’s a very different computational model. Instead of proactively querying things from other services, we subscribe to channels of messages (called topics) and work our way through the messages in there. It’s closer to Alan Kay’s view of object-oriented programming (https://ovid.github.io/articles/alan-kay-and-oo-programming.html). And I’m not talking about that OOP created by the C++/Java community, but the core ideas of OOP thought by Kay and colleagues back in the early days of computing – which curiously was grounded in Biology. The idea of passing messages to objects in order to better scale systems (like biological cells do!). Kafka is nothing but a (very reliable) communication bus, and the objects (services, microservices, whatever) attach themselves to it and listen to messages, responding accordingly and sending other kinds of messages back to kafka which will be consumed by other services. Once you have well-defined messages and protocols, things flow beautifully.

Watch out for waste

Be careful with idle consumers, that means you gotta think about your partitions carefully. What’s worked for us is to set 1:1 partition-to-consumer ratio. For instance, we have an image processing service that fetches jobs from kafka. If we have more instances of that services than partitions, that means some of those instances will be idle, preventing us from achieving a higher throughput than we could.

Closing thoughts

I am, now, a big fan of this architectural style I can totally recommend if it fits your problems. Which in most cases it does. I believe we’ve been going with the flow that was set in the early 00s when it comes to architecting systems; we’ve been using the request-response paradigm without really questioning its efficacy.

To close, the author of designing data-intensive applications has some really cool insights about this paradigm:

It would be very natural to extend this programming model to also allow a server to push state-change events into this client-side event pipeline. Thus, state changes could flow through an end-to-end write path: from the interaction on one device that triggers a state change, via event logs and through several derived data systems and stream processors, all the way to the user interface of a person observing the state on another device. These state changes could be propagated with fairly low delay—say, under one second end to end.

Some applications, such as instant messaging and online games, already have such a “real-time” architecture (in the sense of interactions with low delay, not in the sense of “Response time guarantees”). But why don’t we build all applications this way?

The challenge is that the assumption of stateless clients and request/response interac‐ tions is very deeply ingrained in our databases, libraries, frameworks, and protocols. Many datastores support read and write operations where a request returns one response, but much fewer provide an ability to subscribe to changes—i.e., a request that returns a stream of responses over time.

In order to extend the write path all the way to the end user, we would need to funda‐ mentally rethink the way we build many of these systems: moving away from request/ response interaction and toward publish/subscribe dataflow. I think that the advantages of more responsive user interfaces and better offline support would make it worth the effort. If you are designing data systems, I hope that you will keep in mind the option of subscribing to changes, not just querying the current state.

My next steps? Experimenting with this architecture for applications where the events are user-triggered and the results are near real-time and user-facing.

The increasing complexity of configuring and tuning systems

If you have enough experience in the software industry, then it’s very likely that you’ve struggled with either a configuration problem or a tuning problem.

Configuration and tuning problems are pretty common and can lead to really bad outages. They often occur when:

1. Some parts of the system are poorly or wrongly configured, or

2. A configuration that worked before now doesn’t work because the context of the system has changed.

Think of a number of database replicas and their writing schemes. Or in Postgresql, think of the number of shared buffers, effective cache size, and the min and max wal size.

If wrongly configured from the start, it won’t work in the given context, plain and simple. What’s more interesting, though, is if it’s correctly configured, it might work at a given time. But as the context changes — system workload, system resources usage, overall system architecture — the system will behave poorly. Or, even worse, an outage might happen.

This will, inevitably, lead to manually-performed operations and the creation of heuristics. In other words, it will lead to:

Oh, we should set X to A, when workload is T, but it should be A+10 when workload is T+100 and we have system resources usage above 80%… I guess. Or maybe let’s just up a queue in front of this component, queues solve everything, right?

Now multiply this scenario by tens or hundreds of services. Think for a second about the cognitive burden resulting from these configurations.

This is not a new concern. In 2003, Ganek and Corbi discussed the need for autonomic computing to handle the complexity of managing software systems. They noted that managing complex systems became too costly, labor-intensive, and prone to error due to the pressure engineers felt while maintaining them. This increased the potential of system outages with a concurrent impact on business.

Even nowadays, most of the configurations and tuning of the systems are performed manually, often in run-time, which is known to be a very time-consuming and risky practice. Check out these two links (here and here) to read more about it.

The need for autonomic computing

Most decisions to configure and tune the system are made based on the context — there are many different variables such as workload, number of instances of some services, resources usage, and more. So why not delegate these tasks to something that excels at exactly that? Machine learning sounds like a feasible tool for the job.

After starting my Masters at the University of British Columbia, I kept working on this idea. It seemed interesting although quite weird, and, sometimes, unpractical and impossible to implement.

To my surprise, I realized I wasn’t alone. Some very interesting people were working on these ideas — so it might not be that weird, unpractical, and impossible.

Recently, Jeff Dean — a man that I admire a lot — gave a talk at NIPS 2017 talking about machine learning for systems, where he stated:

Learning should be used throughout our computing systems. Traditional low-level systems code (operating systems, compilers, storage systems) does not make extensive use of machine learning today. This should change!

Computer Systems are filled with heuristics: compilers, networking code, operating systems. Heuristics have to work well “in general case”. [They] generally don’t adapt to actual pattern of usage and don’t take into account available context

Learning in the core of all of our computer systems will make them better/more adaptive.

I was in complete awe when I read this. One of the engineers I admire the most was talking about the very same ideas I’ve been thinking about and working on.

This led me to think that it’s not only interesting but natural to think about enhancing software systems with machine learning. Throughout the whole software stack, we have many heuristics that, although they work well, could be improved by machine learning.

Is it challenging and potentially risky? Yes, most definitely. Especially given that interpretability, apparently, has become a secondary goal in the machine learning community. How can we interpret and explain the decisions made by neural nets?

However, with that said, these obstacles shouldn’t hinder scientific and technological progress. Yes, we should question old paradigms and try to improve things.

Towards machine learning-enhanced software systems

As Jeff Dean pointed out: we need to find practical ways to make systems data-aware. We need systems that collect metrics and metadata about themselves. To achieve this, we could learn a thing a two from the ideas in systems observability and instrumentation. We have been instrumenting systems for decades, and the data is already there.

We also need to find practical and clean ways to integrate machine learning components into software systems, making learning a first-class citizen in the system. This will lead to systems that learn how to improve themselves, beating heuristics and manually-performed operations. Think about this for a second. It does sound cool and feasible.

I would also add that we need practical and clean ways to propagate the decisions made by the learned models to the rest of the system. This would allow the system to have self-adaptive capabilities. Here, we could learn something from the control theory community.

The general idea is fairly simple: make a system learn about its behavior by training a model on its context. Then allow it to change its structures and configurations in order to optimize for a certain scenario. Now implement this idea in such a way that it could be possible to integrate it into many kinds of systems.

Summary

The most interesting questions I have in mind are:

1. Can self-adaptation by learned models lead to more stable, faster, safer software systems? Can it reduce the need for manually configuring and tuning systems, allowing engineers to focus on more important tasks?

2. Can this be easily integrated into software systems, requiring only small changes to the codebase?

3. Can this work with low overhead?

It is worth noting that this would not replace good engineers, but would rather free the engineers’ cognitive abilities to focus on what matters.

I genuinely believe that this will become a trend in the next few years. I myself am working on these ideas as part of my graduate studies, and I will be posting the results of my research, so stay tuned.

And I find that trying to understand this mathematical method by first understanding the concept through a metaphor (which, in this case, I think that the metaphor is our mathematical side, and not the biological one, the biological is just, you know, the real thing) is really valuable to gain an intuition on this topic.

If you were about to make a decision, given many variables, how would you make this decision? It may not look obvious, due the fact that, on normal occasions, we’re not thinking about our thinking process, but the truth is, we are collecting those variables that affect the future decision, and we’re giving weights to them. Weights? how? Simple, some variables are more crucial than other while making decision, isn’t it? the word ‘crucial’, in this case, means a huge weight on this variable, and it will strongly affect the final decision.

Suppose a scenario: we’re deciding whether we should go to beach or not. Let’s put a few binary variables on table:

1. Are we in the mood to go to the beach?
2. Is it raining?
3. Our other friends are going?
4. Do we have money?

So, depending on those variables, it will be more likely that we will go to the beach… or not. Let’s think about the variable ‘is it raining?', if it’s raining, we can say that it’s a deal breaker, so we can conclude that this variable has more weights than the other. The fact is that we already trained those weights inside our brain, long time ago.

That’s what an Artificial Neuron try to do, the AN try to learn the weights of things so it can make decisions. So, drawing back to the mathematical and computational aspect of it, the perceptron is a single unit that will receive external inputs*(variables)*, it will have a weight for each input, it will do some computation, send this result to a decision function, and, finally, output the final decision.

Those inputs are the variables we’re talking before, and like our process to make a decision, the Perceptron will weight each input to make a decision.

Now, how the Perceptron knows how the weights should be for each input? Well… it doesn’t. At least, not from the beginning of the learning process, just like you and me, when trying to learn something new.

That’s why this case is a case of supervised learning, what the Perceptron will do is: start with a random small weights, take one previously trained example (e.g: (1) yes to mood to beach, (2) not raining, (3) yes to friends going to beach and (4) yes to money, the output: 1 - yes, we shall go to the beach) do some computation taking in consideration the random weights we defined before, send it to a decision function, output something (1 - yes, 0 - no) and check if this output is equal to the expected (we’re using a trained example, remember?), of course that at the first try, it won’t be equal. So the Perceptron calculate the error rate and update its weights… after this, guess what? it repeat the process of trying to predict, but now, with the updated weights, after a few tries, the error rate tends to reduce and it starts predicting correctly. So it learned to predict a behavior, by training with previously trained examples.

So, a nice learning behavior would be something like this:

Which means, at every iteration (we call it epochs), the error rate tend to decrease and, as consequence, the Perceptron starts to predict correctly!

As you may have noted, I omitted a few details of the process so you could see the whole picture of the process: we take an example, we practice on it, we see what we missed, we try again, we learn. That’s the core process.

Now, to the details:

The computation

The first step that the Perceptron does it’s a computation that I referred as “some computation”, this is a simple computation, it’s just a simple Linear Combination of the feature vector (the variables) and the weight vector, which is just the sum of the products between feature/input and its respective weight:

$$ x_{1}w_{1}+x_{2}w_{2}+x_{3}w_{3}+ \cdots +x_{n}w_{n} $$

The Decision Function

So, after the Linear Combination, we send the result of it to a decision function, which will, somehow, based on some threshold (or without a threshold as we’re going to see), give us the output, that will be checked with the correct output, in case if it’s correct, fine, keep the weights like this, otherwise, it will calculate the error rate, adjust the weights and restart the process.

Binary output with threshold

This technique is very simple, after the linear combination, if the output is bigger than some threshold, it outputs 1 and we say that the neuron was activated, otherwise, output 0.

$$ \begin{equation} X= \begin{cases} 1, & \text{if}\ Linear,Combination>0 \
0, & \text{otherwise} \end{cases} \end{equation} $$

Using sigmoid function

Now, that a interesting one, it won’t use a threshold anymore, we’ll send the output of the Linear Combination to a sigmoid function

$$ S(\vec{w}\vec{x}) = \dfrac{1}{1+e^{-\vec{w}\vec{x}}} $$

which will, then, smooth the output, making it in the range of 0 and 1. Which is the most used in many Machine Learning Algorithms.

Adjusting the weights and learning

After the perceptron fails to predict correctly, it’s time to adjust the weight, using some rule, the classic perceptron has 2 rules:

Perceptron Learning Rule

This is very simple, when the perceptron has the incorrect output, it update its weights following this:

$$ w_{i} = w_{i} + \eta (t_{i} - o_{i})x_{i} $$

Which is simply multiplying a learning rate (usually 0.001) by the difference between the correct output and the wrong output and then multiplying it by its original input, after this, we add this value to the previous weight and then we have the value of the new weight. It’s fair simple, isn’t it? The problem arises when the data isn’t linearly separable, like this:

With this scenario, the result just won’t converge. So we adopt another rule!

Delta Rule

Now we must find a way to have a non-linear output, and what’s the best for this if not the classic Gradient Descent algorithm? GD will simply search through hypothesis spaces and try to minimize the cost function, I wrote about this here.

So, it’s just a technique to solve our previous problem, but still, the weights will be updated (now, using the GD) and then the process start over!

Code of a Perceptron

Now, here’s a code of a Perceptron that will behave like a simple Boolean function. I didn’t use the Delta Dule (Gradient Descent) to optimize the weights, as this problem (binary Boolean functions) is linearly separable.

  import math
  from numpy import random, array
  from random import choice
  import matplotlib.pyplot as plt

  class Perceptron():
      
      """
      this is the thereshold to activate the unit 
      """
      def activation(self,x):
          return 1/(1 + math.exp(-x))

      def linear_combination(self, X):
          total = 0
          for i in xrange(len(self.weights)):
              total += self.weights[i] * X[i]
          return total
      
      def unit_output(self, X):
          return self.activation(self.linear_combination(X))

      def train(self, training_data, epochs):
          
          self.learning_rate = 0.5
          self.errors = []

          # initializing weights
          self.weights = random.rand(len(training_data[1][0]))
          
          for i in xrange(epochs):
              X, y = choice(training_data)
              # > calculate output with current weight
              output = self.unit_output(X)
              # > calculate error rate (t - o)
              error_rate = y - output
              self.errors.append(error_rate)
              print 'the X: ', X
              print 'output: ',output
              print 'correct target: ', y
              # > apply learning rule which will update weights
              self.weights += self.learning_rate * error_rate * X

      def predict(self,X):
          y = self.unit_output(X)
          return y
                  

          training_data = [
                  (array([0,0,1]), 0),
                  (array([0,1,1]), 0),
                  (array([1,0,1]), 0),
                  (array([1,1,1]), 1),
                  
          ]

  p = Perceptron()

  p.train(training_data, 2000)
  print p.predict([1,0,1])
  plt.plot(p.errors)
  plt.ylabel('errors')
  plt.xlabel('epochs')
  plt.show()

Conclusion and next steps

With this we conclude what a simple single Perceptron is doing! Of course there are many improvements on it and we can connect many of these Artificial Neurons in many layers… that’s what is called Artificial Neural Network, I’ll be writing about it soon!

Then, how can we make the machine predicts things based on that learned data? Those are the question answered by one of the most classic Machine Learning Algorithms, the Gradient Descent Algorithm, from a Mathematical-Statistical side it’s called Univariate Linear Regression.

This is one of the tools of the Machine Learning toolbox, and what it tries to do is to model a relationship between a scalar dependent variable Y and a explanatory variable X.

In Layman’s term…

Let’s suppose you have a few points distributed in a Graph, so you already know that in a point A you have a well defined X and Y, which means, if you input X, your output will be Y, and in a point B you have a well defined X’ and Y’ as well. But, thing is, if a point emerge between A and B, and you only have the X… what will be the Y (the output)?

What this algorithm does is: It tries to predict this Y value, based on the previous data! Amazing, right?

At the end of the execution, you’ll have a full trend line that you can use to predict values! just like the image below.

The Theory Behind It

I’ll cover a few theories about this algorithm here, but, it won’t be complete, as this demand a great coverage of mathematical material that if I would write it all here, It would be a long, long, very long post. So I’m assuming that you’re already familiar with Calculus (Sums, Partial Derivatives), Statistics and Discrete Mathematics.

So, our goal here is to fit the best straight line in our initial data, right?

Thus, we need something to represent this straight line, which will be our hypothesis function:

Where this \( \Theta_{0} \) and \( \Theta_{1} \) are the parameters of the function, and finding the best parameters for this function is what is going to give us the correct straight line to plot on our data.

Here’s an example of what we’re trying to do, which is, fit the best straight line in the data:

So what we want to do is to find a \( \Theta_{0} \) and \( \Theta_{1} \) so our Hypothesis outputs can be very close to the real Y output.

Formally we want:

So we want minimize \( \Theta_{0} \) and \( \Theta_{1} \) so the difference between the Hypothesis and the real output is minimal.

But we want it for every point in our data, which is \(x(i)\) (which is the i-th x of our data), so we want the sum of this average, which is, formally:

And we’re going to call this function Cost Function, with the following notation:

So, our goals is to minimize this cost function:

This cost function is also called Square Error Function.

Now, the gradient descent algorithm

With our cost function built, we need to “keep” finding values for \( \Theta_{0} \) and \( \Theta_{1} \) so we can reach our ideal trend line. So, basically:

1. We start with some \( \Theta_{0} \) and \( \Theta_{1} \)

2. keep changing \( \Theta_{0} \) and \( \Theta_{1} \) to reduce our cost function \( J(\Theta_{0}, \Theta_{1}) \), until we find the minimum.

That’s quite simple and intuitive, right? There’s a lot of intuitive explanation and more visual examples of what this algorithm is doing in the machine learning course taught by Andrews Ng (From Stanford).

What this algorithm will be doing is: partially derive our cost function for \( \Theta_{0} \) and \( \Theta_{1} \) simultaneously, so we can find the minimum value for them, with every iteration updating our \( \Theta_{0} \) and \( \Theta_{1} \) with their new value!

Meh, talk is cheap show me the math!

Formally, the algorithm is:

Make sure that this will run for j = 0 and j = 1.

But, pay attention, this is the generic version, the \( \Theta_{j} \) represent both \( \Theta_{0} \) and \( \Theta_{1} \).

What the algorithm is saying is that we’ll be doing this procedure to \( \Theta_{0} \) and \( \Theta_{1} \) at the same time! So, we can put it in this way:

Also, we can expand the Cost Function that is being derived, doing this, it will be exactly what I’ll be putting into code soon, so, our final algorithm is:

Now it’s time to code all of it!

First things first, we’ll be using Google’s JMathPlot to plot graphs using Java and Swing to use its JFrame, we shall start with our class to represent our Initial Data, which will be the Training Set.

Now we’re going to take our first steps on writing the GradientDescent.java, we must be very careful here. Let’s start with the main settings and parameters of it:

Now let me explain a few details of this part:

Alpha is the Learning Rate, it’s a dangerous variable, it’s used to set the size of the step that the algorithm will take while trying to find the \( \Theta_{0} \) and \( \Theta_{1} \), that’s the learning rate of the algorithm. If Alpha is too low, the algorithm can be very slow, although very precise, if Alpha is higher, it will be taking larger steps, which can be faster, or dangerous, causing the algorithm to DIVERGE, which, trust me, you don’t want this! (Andrew Ng explain this part very well in its course)

The variable TrendLine is what we’ll use to plot the straight line which is our main goal.

The tol variable is our safe move in case of a dangerous convergence, which means, in case of convergence, it will stop the execution.

The other variables and objects in this part are very intuitive to understand, it’s auto explainable! (forgive if i’m wrong, just say something and I’ll put more detail on that).

About the Constructor, we’re saying that our initial guesses for \( \Theta_{0} \) and \( \Theta_{1} \) is 0. The rest is just data plotting.

next: our Hypothesis Function (that will be doing exactly as the model that I did show above)

Now, our two function to derive our \( \Theta \), again, it will be doing exactly the same as the mathematical model, there’s no magic!

Note that it does almost exactly the same as the mathematical model of the Gradient Descent demonstrated above, the difference is only a few details, such as the if and while to verify convergence or divergence (which is, if it reached the iteration’s limit)

Next we have the addTrendLine function, used to keep plotting our straight line as it will become more updated.

And now we have an extra, it’s a function to store and plot the convergence history of both \( \Theta_{0} \) and \( \Theta_{1} \), so we can see how it happened.

Finally, we have our Test Class, that will execute everything:

Now, executing the code, the output will be, at first, the initial data:

Executing the algorithm, it learns and generate its prediction based on its initial data:

Magical, right?

And then, we can see the convergence:

And you can see in the terminal the final values of \( \Theta_{0} \) and \( \Theta_{1} \) that minimized the Cost Function.

If it wasn’t Science, probably would be black magic. heh.

A few considerations

You can download the complete code here.

If you have any questions/suggestion, drop me an email so we can talk!

If you need further details of the mathematical model, I ultra advice to watch Andrew’s Ng Videos at Stanford@Coursera. His skills to teach it is something unbelievable awesome.

Thanks for reading!