Flow, Actors, and Simulation

• 4.1 The Problem Flow Solves
• 4.2 The Actor Model, FDB Edition
• 4.3 Futures and Promises
• 4.4 The Event Loop
• 4.5 Network = Just Another Future
• 4.6 The “Knobs” System
• 4.7 Simulation: How Bugs Get Caught Before You
  • How a simulation run works
  • Deterministic replay
  • Source pointers
• 4.8 Why You Care When You Use FDB from Go
• 4.9 Hands-On: Reading One Actor End-to-End
• Interview Questions

“Flow is the C++ language extension I always wanted but was too lazy to write. The FoundationDB team wrote it, and then they wrote a distributed database in it.” — paraphrased from a former Apple engineer

If you grep apple/foundationdb for ACTOR, you get ~7,500 hits. If you grep for Future<, you get ~25,000 more. Every line of FoundationDB — client library, Storage Server, Resolver, Proxy, Coordinator — is written in Flow, a C++14-compatible language extension for asynchronous, single-threaded, deterministic concurrency.

Flow is not a library. It is a source-to-source compiler that takes *.actor.cpp files, expands ACTOR / wait() / loop keywords into giant state machines, and emits ordinary *.g.cpp files that any C++14 compiler will accept. The Flow source is in flow/ and the compiler itself is in flow/actorcompiler/ — it is a 4,000-line C# program (yes, C# — leftover from when Apple’s tooling team owned the build).

You cannot read FoundationDB source without reading Flow. This chapter teaches the dialect.

4.1 The Problem Flow Solves

A database server is one of the world’s worst fits for traditional threading:

• A single Storage Server holds tens of thousands of simultaneous in-flight RPCs — reads, mutation pushes from TLogs, watch notifications, status pings.
• Each RPC may block on disk, network, or another RPC’s reply.
• The latency budget for a single read is ≈1 ms — most of which is network.
• Lock contention and context-switching costs are catastrophic at this scale.

The 2010-era options were:

• Threads + mutexes — too slow, deadlocks, race conditions.
• Callback chains (“callback hell”) — fast, but unreadable.
• Coroutines (Boost.Asio stackful) — readable, but slow and heavy.

The FDB team wanted callback performance with linear-code readability, plus something more: determinism, so that the simulation harness (§4.7) could reproduce any bug. None of the existing options gave determinism.

So they invented Flow.

4.2 The Actor Model, FDB Edition

In Flow, an Actor is a function that may suspend at any wait() call, yielding control back to the event loop (a single OS thread per process). While suspended, other actors run. When the awaited future completes, the actor resumes from where it left off.

ACTOR Future<Value> readWithRetry(Database db, Key key) {
    state Transaction tr(db);
    loop {
        try {
            Value v = wait(tr.get(key));
            return v;
        } catch (Error& e) {
            wait(tr.onError(e));
        }
    }
}

Three keywords are not standard C++:

• ACTOR — marks the function as Flow-compiled. The compiler will replace this with a class that holds the function’s local state and a step() method.
• state — marks a local variable as needing to survive across wait() calls. Plain locals are not preserved.
• wait() — suspends the actor until the given Future<T> is ready, then resumes with its value. Only legal inside ACTOR functions.
• loop — syntactic sugar for while (true) { ... }, distinct because the compiler optimizes the surrounding state machine.

What the compiler emits (simplified) for the above:

// machine-generated VersionedBTree.actor.g.cpp
class ReadWithRetryActor : public Actor<Value> {
    Database db;
    Key key;
    Transaction tr;             // ← state var, lives across waits
    int pc = 0;                  // program counter

    void step() {
        switch (pc) {
        case 0:                                       // initial entry
            tr = Transaction(db);
            pc = 1; [[fallthrough]];
        case 1: {                                     // loop top, before get
            Future<Value> f = tr.get(key);
            if (!f.isReady()) {
                f.addCallback([this]{ pc = 2; step(); });
                return;
            }
            pc = 2; [[fallthrough]];
        }
        case 2: {                                     // after wait, returned
            try {
                Value v = futureGet<Value>();
                sendResult(v);
                return;
            } catch (Error& e) {
                Future<Void> r = tr.onError(e);
                if (!r.isReady()) {
                    r.addCallback([this]{ pc = 1; step(); });
                    return;
                }
                pc = 1; goto case_1;                  // loop
            }
        }
        }
    }
};

That generated code is what your CPU actually executes. The ACTOR syntax is purely a compile-time convenience. You can read the actual generated .g.cpp files by building the project; CMake emits them into the build directory.

4.3 Futures and Promises

Flow’s Future<T> is not std::future<T>. It is a smart-pointer to a small SAV (“Single Assignment Variable”) object that holds either:

• the value T (once set), or
• a list of callbacks waiting for it, or
• an error.

Promise<T> is the writer end. The pattern:

Promise<int> p;
Future<int>  f = p.getFuture();

p.send(42);            // sets the value, fires all callbacks
// later:
int x = wait(f);       // (inside an ACTOR) returns 42 immediately if ready

The single-assignment rule (send exactly once) is enforced at runtime. There is no shared state synchronization, because the entire event loop runs on one thread. This eliminates a whole class of concurrency bugs and is the foundation of Flow’s determinism.

Source: flow/include/flow/flow.h, look for class Future, class Promise, class SAV.

4.4 The Event Loop

Net2 (flow/Net2.actor.cpp) is FDB’s event loop. Pseudo-code:

while (running) {
    Time now = monotonicTime();

    // 1. Run all ready callbacks
    while (!readyQueue.empty()) {
        readyQueue.pop()->fire();
    }

    // 2. Run any timers that have expired
    while (!timerHeap.empty() && timerHeap.top().t <= now) {
        timerHeap.pop().fire();
    }

    // 3. Poll the network (epoll/kqueue) and disk (io_uring/AIO)
    int timeoutMs = computeNextTimerDelay();
    pollEvents(timeoutMs);
    // ↑ pushes any newly-ready I/O completions onto readyQueue

    // 4. (optional) yield to the OS to maintain low priority for system tasks
    if (cpuHeavy) { sched_yield(); }
}

Three things to internalize:

1. One thread. Every wait() resumes on the same thread as the original ACTOR invocation. No data races, no mutexes inside Flow code.
2. Cooperative scheduling. An actor must wait() (or return) to yield. A CPU-heavy actor that doesn’t wait() will block every other actor in the process — including network I/O. Flow has utilities like yield(taskPriority) to voluntarily reschedule.
3. Priorities. Every actor runs at a TaskPriority (see flow/include/flow/Knobs.h). The event loop runs higher priorities first. This is how FDB keeps tail latency bounded: critical-path actors (commit pipeline) run at higher priority than background actors (data distribution, status reporting).

4.5 Network = Just Another Future

The single most elegant thing in Flow is that the network is presented exactly like any other asynchronous operation:

ACTOR Future<Void> echoServer(NetworkAddress addr) {
    state Reference<IListener> listener = INetworkConnections::net()->listen(addr);
    loop {
        Reference<IConnection> conn = wait(listener->accept());
        echoOne(conn);                          // fire-and-forget child actor
    }
}

ACTOR Future<Void> echoOne(Reference<IConnection> conn) {
    loop {
        wait(conn->onReadable());
        std::vector<uint8_t> buf(1024);
        int n = conn->read(buf.data(), buf.size());
        if (n == 0) return Void();
        wait(conn->onWritable());
        conn->write(buf.data(), n);
    }
}

IConnection is an interface. There are two implementations:

• Net2::Connection — backed by real Berkeley sockets via epoll/kqueue. Used in production.
• Sim2::Connection — backed by a simulated network. Used in tests. Same actor code runs unchanged.

This is the magic: you write one networking implementation, but you can run it on a real network or a simulator. The simulator can inject latency, drop packets, partition the cluster, even simulate clock skew — all without modifying a single line of the database code.

4.6 The “Knobs” System

Every tuning constant in FDB is a knob. They live in flow/include/flow/Knobs.h and per-module files like fdbserver/Knobs.h.

Examples relevant to performance tuning:

Knobs are tweakable at runtime by setting --knob_<NAME>=<VALUE> on the fdbserver command line. Changing knobs in production is strongly discouraged — most of them are co-tuned and undocumented in their interactions. But for performance experiments, knobs are the lever.

4.7 Simulation: How Bugs Get Caught Before You

The simulation harness — fdbserver -r simulation -f some_test.txt — is how the team finds bugs before customers do.

How a simulation run works

1. Boot: fdbserver creates a synthetic cluster of n processes (typically 5–15), each represented as a coroutine inside the same OS process.
2. Inject: a Sim2 instance replaces Net2. All INetworkConnections::net() calls now return simulated connections.
3. Time: monotonic time is virtual. delay(0.001) doesn’t sleep — it advances the virtual clock by 1 ms when there’s nothing else to do.
4. Workload: a test file (*.toml or *.txt in tests/) specifies which clients to run (Cycle, RandomTransactionWorkload, ClogTLog, etc.) and which failures to inject (Attrition, Clogging, MachineAttrition).
5. Fault injection: the simulator’s RNG (seeded from one 64-bit number) decides at every step whether to kill a process, partition the network, corrupt a disk page, slow a thread.
6. Validation: workloads verify their own invariants (Cycle checks that a ring of writes traverses correctly). If any actor calls ASSERT(...) and it fails, the simulator dumps the seed and exits.

Deterministic replay

Every random decision in the simulator goes through one PRNG, seeded by the test’s randomSeed. Two runs with the same seed produce identical event ordering. So when you see:

SimulatorTrace: failed assertion, seed = 1234567890123

…you re-run with --random-seed 1234567890123 and reproduce the bug deterministically — even on a different machine, even months later.

The discipline this imposes is fierce. Anything non-deterministic — system time, thread scheduling, hash table ordering — is banned from FDB code unless funneled through g_random. Code review at FDB famously rejects any std::unordered_map because its iteration order varies across allocator states. The codebase uses std::map or the Flow-provided ordered containers instead.

Source pointers

A worthwhile read for anyone learning Flow: open fdbserver/workloads/Cycle.actor.cpp. It is < 200 lines, uses every Flow primitive, and represents one of the canonical FDB correctness tests (the “ring cycle” — a sequence of transactions that should leave a ring of integers in a known configuration).

4.8 Why You Care When You Use FDB from Go

You write Go (or Java, Python, Ruby). You never write Flow. So why does this chapter matter to a layer developer?

Because the client library (libfdb_c.so) — which your Go bindings call through CGO — is itself written in Flow. Look at fdbclient/. Every Database, Transaction, Future object you use in Go is a thin handle over a Flow object running on a dedicated network thread inside your process.

This explains several things you might have noticed:

• Thread safety. fdb.Database is goroutine-safe even though its underlying state is mutable. Why? Because every operation eventually queues a callback onto the single Flow network thread.
• fdb.MustAPIVersion(710). Must be called exactly once per process. This boots the Flow event loop in a background thread. Subsequent fdb.OpenDatabase calls reuse it.
• Future.Get() blocking. Your goroutine Get()s a Future. Internally, the C client registers a callback on the Flow Future. When that fires (on the network thread), it signals a condition variable that wakes your goroutine. The hop is: Flow thread → CGO callback → Go scheduler → your goroutine resumes.
• The “transaction wait limit”. If your goroutine takes seconds between reads, you are not just risking MVCC expiry — you may also be holding network buffers and callback queues open on the Flow side.

Practical implication: keep transactions tight. Compute outside, transact inside. The Go binding is fast, but every CGO call costs ~200 ns and every Future has a Flow-side memory cost.

4.9 Hands-On: Reading One Actor End-to-End

Pick this real example — the function the Proxy calls to assign a commit version:

fdbserver/MasterServer.actor.cpp, search for ACTOR Future<Void> getVersion(. (In some 7.x branches this is in SequencerServer.actor.cpp.) The full function is ~60 lines and exercises nearly every Flow feature:

• state variables for the request batch.
• loop { ... } for accumulating requests within a 1-ms window.
• wait(delay(...)) to bound batch latency.
• Promise<...>::send(...) to fulfill each request after assigning versions.
• try / catch (Error& e) for the standard Flow retry pattern.

If you can read that function in one sitting and explain what it does, you can read 80% of the rest of FDB. The remaining 20% is Coordinator/Paxos and the storage engines — both covered in their own chapters here.

Interview Questions

Q: Why does FoundationDB run on a single thread per process?

To eliminate concurrent-access bugs in the database code and to make the simulation deterministic. Mutexes, atomics, and thread scheduling are sources of non-determinism that defeat replayable testing. A single thread makes every concurrent interaction explicit: it’s a wait() on a Future, not a hidden interleaving. Scalability across cores is achieved by running multiple Storage Server, Proxy, etc. processes on the same machine, not by threading within a process.

Q: What is the difference between wait(f) and f.get()?

wait(f) is legal only inside ACTOR functions. It suspends the current actor until f is ready, then resumes (without blocking the thread). f.get() is callable from anywhere but synchronously blocks the thread until f is ready — which inside Flow is a bug (it would freeze every other actor). f.get() is only used in test scaffolding and from non-Flow callers like the C client library’s Go binding bridge.

Q: Why are unordered_map and threads banned in FDB code?

Both are sources of non-determinism. unordered_map’s iteration order depends on hash-collision history. Threads introduce scheduling-dependent interleavings. Either breaks the property “run the simulator with the same seed twice and get identical execution,” which is the entire basis of FDB’s testing strategy.

Q: How does the simulation simulate disk I/O?

Sim2’s IAsyncFile implementation backs every file with a std::map of page bytes in RAM. Writes update the map; reads consult it. Latency is artificial: a write delays for a synthetic “disk latency” drawn from a configured distribution. Failures are also synthetic: at random times, chosen by the seeded PRNG, the simulator can drop pending writes (simulating a crash before flush) or corrupt a written page (simulating bit rot).

Q: A simulation run finds a bug at seed 0xABCD. The next day, on a new machine, you re-run with the same seed and get a different failure. What’s wrong?

Almost certainly someone introduced non-determinism — likely a new std::unordered_map, a direct gettimeofday(), or a thread-local. Standard FDB code review pattern: git bisect between the last good build and now, re-running the failing seed each time. The introduction of non-determinism is the bug; the original test failure is a symptom of whatever real bug the simulator was hunting.