📖 Local build — commit and timestamp are injected automatically on Cloudflare Pages.

Preface — Why This Book Exists

• How This Book Is Structured
• Who This Book Is For
• A Note on Style
• FoundationDB Layers — Go Implementations
  • Layout
  • Prerequisites
  • Bootstrap the cluster
  • Running a demo
  • Documentation (mdbook)

“In many of the more relaxed civilizations on the Outer Eastern Rim of the Galaxy, the Hitchhiker’s Guide has already supplanted the great Encyclopaedia Galactica as the standard repository of all knowledge and wisdom.” — Douglas Adams

This book aims to be that for FoundationDB: the practical, opinionated, source-level guide you wished existed when you first opened the docs.

There is no shortage of FoundationDB material on the internet. There is a tidy official documentation site, a handful of conference talks, a research paper (SIGMOD 2021), and a famous CMU Database Group talk by Markus Pilman. Each piece is good. None of them together is enough to build a layer.

What’s missing is the connective tissue:

• The history of how FDB got here — why the original storage engine was literally SQLite’s B-tree (KeyValueStoreSQLite), why that was replaced by Redwood, and what Sharded RocksDB is doing in the codebase as of 7.3.
• The source-level mechanics — what a commit actually does on the wire, how many fsync calls it costs, what the Resolver’s in-memory data structure looks like, how Flow’s actor model compiles down to C++.
• The numbers — concrete latency budgets, throughput ceilings, transaction volume curves, and what changes them. Not “FDB is fast” but “a 3-node cluster on c5.4xlarge sustains ≈55,000 cross-shard commits/sec at p99 < 8 ms, here is the workload, here is how to reproduce it.”
• The patterns — five hands-on labs in this repo that turn that theory into Go code you can run, modify, and break.

This book is what happens when you treat FoundationDB the way Brendan Gregg treats Linux performance, or the way Designing Data-Intensive Applications treats distributed systems: with curiosity, citations, and the willingness to follow a function call across a process boundary into another machine.

If you finish this book you should be able to:

1. Explain to a colleague exactly what happens during db.Transact(...) — every network hop, every disk write, every B-tree page touched.
2. Read the FoundationDB source tree (fdbserver/, fdbclient/, fdbrpc/, flow/) and know which files to open for a given question.
3. Design a new layer (graph, vector index, time series, queue) from scratch, choosing the right key encoding, conflict-range strategy, and atomic op primitives.
4. Reproduce the published benchmarks and reason about why your workload deviates from them.
5. Submit a meaningful PR to apple/foundationdb — fix a bug, add a test, improve a doc, optimize a hot path. The contributing chapter is a real path, not a token gesture.

How This Book Is Structured

The book has two parts. Part I — Background is a deep, source-level guide to FoundationDB itself. Read it in order; each chapter is a prerequisite for the next. Part II — Labs is five independent Go implementations you can run; pick any order, but each lab’s walk-through assumes you know Part I.

Who This Book Is For

• Backend engineers who use FDB-backed services (CloudKit, Snowflake metadata, Tigris, etc.) and want to understand the substrate.
• Database engineers designing a new storage system or evaluating FDB.
• Distributed systems students who have read DDIA and want a single open-source system to study end-to-end.
• Interview candidates preparing for senior systems roles — the “Interview Questions” sections at the end of each chapter are designed for exactly this.
• Aspiring FDB contributors — the final chapter is your runway.

A Note on Style

This book quotes the FoundationDB source tree liberally. All citations are to apple/foundationdb at the time of writing (release-7.3 branch unless noted). Where source files have moved between releases, I give the older path too. Code excerpts are short and used only for explanation; the project’s Apache 2.0 license permits this, and the original copyright stays with the FoundationDB authors.

Where I cite latency or throughput numbers, I cite the hardware and workload. Treat every number as falsifiable; the reproduction recipe is always nearby.

Now — don’t panic — and turn the page.

FoundationDB Layers — Go Implementations

Five self-contained Go implementations that show different ways to build a “layer” on top of FoundationDB. Modeled after real projects (mvsqlite, fdb-record-layer) but pared down for clarity.

Layout

Prerequisites

1. Docker — to run a local single-node FDB cluster
2. FoundationDB client library on the host (libfdb_c) — required by the Go bindings (CGO)
   • macOS: brew install foundationdb (or install the official .pkg from Apple’s release page)
3. Go 1.22+

Bootstrap the cluster

docker compose up -d
./scripts/bootstrap-fdb.sh

This creates ./fdb.cluster at the repo root. Every demo reads it via the relative path ../fdb.cluster.

To shut down: docker compose down (data persists in ./fdb-data).

To wipe everything: docker compose down -v && rm -rf fdb-data fdb-config fdb.cluster.

Running a demo

Each option is an independent Go module:

cd option-a-leveldb
go run ./demo

See each folder’s docs/ for an architecture walk-through.

Documentation (mdbook)

The docs are structured as an mdbook under book/. To read them locally:

# Install mdbook (once)
brew install mdbook

# Build the book (output → book/dist/)
mdbook build

# Or serve with live-reload
mdbook serve --port 3001

The book is organized into two parts:

• Background — The Hitchhiker’s Guide split into six chapters (storage stack, FDB internals, the layer concept, real-world systems, repo overview, further reading)
• Labs — One overview + architecture walk-through per option

The Hitchhiker’s Guide to FoundationDB and Storage Layers

• What This Guide Covers
• How to Read This
• Core Insight

“Don’t Panic.”

This guide assumes you have seen a key–value store before but have never thought hard about what happens underneath. By the end you should be able to design and build your own layer on top of FoundationDB — not just copy one.

What This Guide Covers

Ten chapters that take you from first principles to source-level fluency. Read in order; each chapter is a prerequisite for the next.

How to Read This

If you have 2 hours: Read chapters 1–3 (the storage stack, FDB in depth, and the storage-engine deep dive), then pick any one lab and read its Architecture Walk-through.

If you have a day: Read all ten chapters in order, then work through all five labs (A-leveldb → A-sqlite → B-leveldb → B-sqlite → C-record-layer). The labs increase in conceptual complexity.

If you’re preparing for a systems interview: Focus on chapters 1, 2, 5 (performance), and 6 (the layer concept), and the Interview Q&A sections at the end of each chapter.

If you want to contribute upstream: Read chapters 2, 3, and 4 carefully, then jump to Chapter 11 — Contributing for the runway.

If you just want to run the code: Jump straight to Chapter 9 — This Repository, then come back to the background chapters for depth.

Core Insight

Every database is a stack of layers, each one a client of the one below. FoundationDB occupies one specific layer: ordered, transactional, distributed key-value storage. Every layer above it — SQL, document model, LevelDB API, SQLite page store, record layer — is a pure encoding problem. Learn the encoding and you understand the database.

That is what this guide teaches.

The Storage Stack — Where Everything Lives

• The Canonical Stack
• B-Trees vs. Log-Structured Merge Trees
  • B-Trees
  • Log-Structured Merge Trees (LSM Trees)
  • The RUM Conjecture
• Write-Ahead Log (WAL) — Crash Recovery Mechanics
• Buffer Pool Management
• Where Each Lab Sits in the Stack
• Interview Questions

Every database you have ever used is a stack of layers, each hiding detail from the layer above. Knowing exactly where FoundationDB sits in that stack — and why that position is powerful — is the foundation of everything else in this guide.

The Canonical Stack

┌─────────────────────────────────────────────┐
│  Application  (SQL, gRPC, REST API)         │  ← you usually live here
├─────────────────────────────────────────────┤
│  Query Planner / Optimizer                  │  ← rewrites queries, picks indexes
├─────────────────────────────────────────────┤
│  Execution Engine  (joins, aggregates)      │  ← evaluates the query plan
├─────────────────────────────────────────────┤
│  Storage Engine  (rows, columns, indexes)   │  ← where FDB plays
├─────────────────────────────────────────────┤
│  Page / Buffer Cache                        │  ← in-memory hot pages
├─────────────────────────────────────────────┤
│  File / Block I/O  (VFS, pread/pwrite)      │  ← option-b-sqlite plugs in here
├─────────────────────────────────────────────┤
│  Kernel: page cache, scheduler, filesystem  │
├─────────────────────────────────────────────┤
│  Physical: SSD / NVMe / network storage     │
└─────────────────────────────────────────────┘

The interesting question is: which layer do you own, and what contract does it expose downward?

Most database literature focuses on the top (SQL, query plans). Most OS literature focuses on the bottom (filesystems, block I/O). The storage engine layer — the middle — is where the real trade-offs live: ordering, atomicity, durability, and concurrency.

FoundationDB is a storage engine that exposes a remarkably clean API to everything above it. This repository explores five different ways to build on top of that API.

B-Trees vs. Log-Structured Merge Trees

Every practical storage engine is built on one of two foundational data structures. Understanding both is essential for reasoning about FDB’s design and the designs of the engines it replaces.

B-Trees

A B-tree stores data as a balanced tree of fixed-size pages (usually 4–16 KB). Updates happen in-place: to change a value, you find its page, overwrite it, and write it back to disk.

                    ┌──────────────┐
                    │  [20] [50]   │  ← interior node (keys only)
                    └──────────────┘
                   /        |        \
        ┌──────────┐  ┌──────────┐  ┌──────────┐
        │ [5][12]  │  │[25][35]  │  │[55][80]  │  ← leaf nodes (key+value)
        └──────────┘  └──────────┘  └──────────┘

Properties:

• Read amplification: O(log N) page reads to find a key (very good)
• Write amplification: ~1–3× (one write per page touched, but journaling doubles it)
• Space amplification: ~33% overhead on average (partially filled pages)
• Random writes: each update may touch a different page → high random I/O on spinning disks

Why B-trees dominated for 40 years: HDDs were sequential-read fast, random-read slow. B-trees minimize the number of seeks to reach a key. SSDs changed this equation — random reads became cheap.

Used by: SQLite, PostgreSQL, MySQL InnoDB, Oracle, early MongoDB (MMAPv1)

Log-Structured Merge Trees (LSM Trees)

LevelDB, RocksDB, and Cassandra’s storage engine are all LSM trees. The core idea: never overwrite; always append.

Write path:
  incoming write
      ↓
  WAL (append-only log on disk — for durability)
      ↓
  MemTable (skip list in memory — for fast writes)
      ↓ (when MemTable is full, flush to disk)
  Level 0 SSTables (sorted, immutable files)
      ↓ (background compaction)
  Level 1, 2, ... SSTables

Anatomy of an SSTable (Sorted String Table):

┌─────────────────────────────────────────────┐
│ Block 0: apple→1, banana→2, cherry→3 ...    │
│ Block 1: date→4, elderberry→5, fig→6 ...    │
│ ...                                          │
│ Index block: [apple→offset0, date→offset1]  │
│ Bloom filter: membership query, ~1% FP rate │
│ Footer: offsets to index+filter blocks      │
└─────────────────────────────────────────────┘

Properties:

• Write amplification: high (10–30× in a 7-level LSM; data is rewritten during each level’s compaction)
• Read amplification: moderate (must check all levels for a key; bloom filters help)
• Space amplification: moderate (old versions linger until compaction)
• Sequential writes: almost entirely sequential → excellent on SSDs and HDDs alike

Why LSM trees dominate at scale today: write throughput is often the bottleneck at scale, and LSM trees turn random writes into sequential appends. RocksDB (Facebook) and Cassandra (Apache) both validate this at planetary scale.

The RUM Conjecture

The RUM conjecture (Idreos et al., 2016) formalizes the fundamental trade-off:

You cannot minimize Read cost, Update cost, and Memory/space cost simultaneously. Optimizing two always worsens the third.

FDB uses a custom B-tree variant internally on each Storage Server. The distributed nature adds replication overhead but also provides horizontal read scaling.

Write-Ahead Log (WAL) — Crash Recovery Mechanics

Every durable storage engine uses a WAL. Understanding WAL mechanics is essential for understanding what FDB guarantees and what xSync (in the SQLite VFS) must do.

The problem: writing a page to disk is not atomic. A 4 KB page write may partially complete before a power failure. Result: corrupted page.

WAL solution:

1. Before modifying Page X, write the INTENT to the WAL:
      WAL record: {LSN: 1042, page: X, before: <old bytes>, after: <new bytes>}
2. fsync() the WAL record to disk (durability guarantee)
3. Now apply the change to Page X in the buffer cache
4. Page X is written back to disk lazily (the WAL already protects us)

On crash recovery:

• Re-apply all WAL records since the last checkpoint → consistent state
• If a WAL record is incomplete (partially written) → discard it, prior state is clean

FDB’s relationship with WAL: FDB’s Transaction Log layer IS the WAL. When a commit is acknowledged, the write is persisted in the Transaction Log on f+1 machines (where f is the fault tolerance factor). Storage Servers apply these writes asynchronously. This is why FDB’s commit is durable even if Storage Servers crash before applying the write — the Transaction Log holds the record.

This also explains why our SQLite VFS (option-b-sqlite) makes xSync a no-op: by the time Transact() returns, FDB has already done the equivalent of fsync on the WAL.

Buffer Pool Management

In any storage engine, the buffer pool (or page cache) is the in-memory cache of disk pages. It is often the single most important performance variable.

Key metrics:

• Hit ratio: fraction of page reads served from cache (aim for >99%)
• Eviction policy: LRU (Least Recently Used), LFU (Least Frequently Used), CLOCK
• Write-back vs. write-through: most use write-back (dirty pages flushed lazily to amortize I/O)

Why this matters for FDB: FDB’s clients do not manage a buffer pool. Each Storage Server manages its own buffer pool internally. When you call rt.Get(key), the FDB client sends a network request; the Storage Server consults its buffer pool and returns the value. You have no visibility into whether this was a cache hit or a disk read.

For read-heavy workloads, adding more Storage Servers (and thus more total buffer pool capacity) linearly increases the read cache size.

Where Each Lab Sits in the Stack

┌──────────────────────────────────────────────────────────────┐
│              Application code  (demo/main.go)                │
├──────────────────────────────────────────────────────────────┤
│  option-a-leveldb  │  option-a-sqlite  │  option-c-record   │
│  (LevelDB API)     │  (SQL engine)     │  (Record + Index)  │
├──────────────────────────────────────────────────────────────┤
│  FoundationDB client library  (fdb.Transact / ReadTransact)  │
├──────────────────────────────────────────────────────────────┤
│           FDB cluster (coordinators, proxies, storage)       │
├──────────────────────────────────────────────────────────────┤
│  option-b-sqlite   │  option-b-leveldb                       │
│  (SQLite VFS)      │  (LevelDB storage.Storage)             │
├──────────────────────────────────────────────────────────────┤
│            FDB as the backing file store                      │
└──────────────────────────────────────────────────────────────┘

Options A and C sit above FDB — they use FDB as a storage substrate and build a new API on top. Options B sit below an existing engine — they replace the file system with FDB while keeping the engine’s logic intact.

This is a crucial architectural distinction. In the A/C case, FDB’s ACID semantics are exposed to the caller. In the B case, FDB’s semantics are hidden inside a file-like abstraction, and the engine above (LevelDB, SQLite) provides its own ACID layer — layered on top of FDB’s.

Interview Questions

Q: What is write amplification and why does it matter?

Write amplification is the ratio of bytes written to disk versus bytes written by the application. A write amplification of 10× means writing 1 MB of data causes 10 MB of disk writes. LSM trees have high write amplification (compaction rewrites data multiple times) but low latency writes. B-trees have low write amplification but high random-write I/O. At scale, write amplification directly determines storage cost and SSD wear.

Q: Why does an LSM tree use multiple levels?

Compaction merges SSTables from one level into the next. Without levels, you’d need to merge all SSTables at once — an O(total_data) operation for every write. Leveled compaction keeps each level to a fixed size ratio (~10×), so each compaction is bounded. This bounds read amplification too: there are at most one SSTable per key per level, so a lookup checks at most L SSTables.

Q: What is the difference between a WAL and an SSTable?

A WAL (Write-Ahead Log) is an append-only sequence of write operations — it records WHAT changed, not the final state. It’s used for crash recovery. An SSTable is an immutable, sorted snapshot of actual data — it stores key-value pairs in sorted order for efficient lookup. The WAL is ephemeral (truncated after checkpointing). SSTables are persistent (they ARE the data).

Q: What makes FDB’s storage engine different from a traditional B-tree?

FDB’s storage servers use a custom B-tree variant, but the key difference is architectural: FDB is distributed, sharded, and replicated. Each Storage Server holds a shard (a key range). Reads are served directly by Storage Servers (bypassing the Proxy). Writes go through the Transaction Log for durability before being applied to Storage Servers. This means durability and availability are decoupled — you can lose a Storage Server without losing committed data.

FoundationDB in Depth

• 2.1 What FDB Actually Is
• 2.2 The Ordered KV Model — Deep Dive
• 2.3 Transactions: The Full Picture
  • ACID Guarantees
  • The Commit Pipeline
  • Retry Semantics
  • Read-Only Transactions
• 2.4 MVCC — Multi-Version Concurrency Control
  • MVCC vs. Locking
• 2.5 Watches, Versionstamps, and Atomic Operations
  • Watches
  • Versionstamps
  • Atomic Operations
• 2.6 Cluster Architecture — Every Component Explained
  • Proxies (GRV Proxies + Commit Proxies)
  • Resolvers
  • Transaction Log
  • Storage Servers
  • Coordinators
• 2.7 Production Limits and Sizing
• 2.8 The Simulation Harness — Why FDB Is Trusted
  • How Simulation Works
  • Why Determinism Matters
  • Scale of Testing
  • What This Means for You
• Interview Questions

FoundationDB is one of the most carefully engineered distributed databases ever built. This chapter goes far beyond the API surface — it covers the commit pipeline, MVCC mechanics, cluster architecture, and the simulation harness that makes FDB trustworthy at Apple-scale production.

Where this chapter sits. This is the high-level architectural tour. The next three chapters drill into the parts that most warrant their own treatment: the on-disk storage engine (Chapter 3), the Flow language and simulation harness (Chapter 4), and the performance model in concrete numbers (Chapter 5). Read this chapter first, then those.

2.1 What FDB Actually Is

FoundationDB is a distributed, ordered, transactional key–value store. Each word carries precise meaning:

Distributed: data is automatically sharded across machines. You do not choose a partition key. FDB’s shard boundaries move dynamically as data grows or machines are added. The client library discovers shard locations by reading the cluster file and caching routing information.

Ordered: keys are stored in lexicographic byte order across the entire cluster. GetRange(begin, end) returns keys in sorted order regardless of which Storage Servers hold them. This is the single most powerful property for building layers, because co-location of related data is entirely under your control via key encoding.

Transactional: full ACID across arbitrary key ranges, on multiple machines, at unlimited scale. This is rare. DynamoDB offers single-item atomicity (later added limited cross-item transactions). Cassandra offers lightweight transactions (LWT) at one consistency level. Redis offers multi-key transactions (MULTI/EXEC) without conflict detection. FDB offers true serializable transactions with automatic conflict detection and retry — across any number of keys, on any number of machines.

Key–value: the value is an opaque byte string (max 100 KB). FDB does not care about structure inside the value. All richness is imposed by the layer above — this is by design. It keeps FDB’s guarantees simple and composable.

2.2 The Ordered KV Model — Deep Dive

The FDB key space is conceptually a single sorted array of byte arrays, spanning the entire cluster:

Key space (lexicographic byte order):
  ""  (smallest possible key)
  ...
  "\x00"
  "\x00\x00"
  ...
  "apple"
  "apple\x00price"       ← prefix scan works naturally
  "apple\x00weight"
  "banana"
  ...
  "\xff\xff\xff..."  (largest possible key)

Range semantics — half-open intervals:

All FDB ranges are [begin, end) — inclusive begin, exclusive end. This is universal in FDB’s API:

fdb.KeyRange{Begin: fdb.Key("apple\x00"), End: fdb.Key("apple\x01")}
// Returns: apple\x00price, apple\x00weight, apple\x00... everything
// Does NOT return: apple\x01, banana, ...

The trick of incrementing the last byte (\x00 → \x01) creates a tight upper bound for prefix scans. This pattern appears in every layer in this repo.

Why ordering is the enabler for layers:

Without ordering, you could only do point lookups (O(1) Get). Ordering enables:

• Prefix scans: all keys for user 42 (range scan on user:42:*)
• Range queries: all records with age BETWEEN 25 AND 40 (range scan on encoded age values)
• Sorted iteration: return all records in PK order (scan the records subspace)
• Pagination: “next page” = resume range scan from last seen key

Every SQL feature that involves ORDER BY, BETWEEN, prefix LIKE, or primary key range is implementable purely through ordered range scans.

2.3 Transactions: The Full Picture

ACID Guarantees

FDB transactions implement Strict Serializability (Herlihy & Wing, 1990): the strongest isolation level. It is equivalent to:

• Serializability (transactions appear to execute one at a time)
• Real-time ordering (if T1 commits before T2 starts, T1 appears before T2)

Most databases offer weaker levels by default. PostgreSQL defaults to “Read Committed” (not serializable). MySQL InnoDB defaults to “Repeatable Read”. FDB always runs at strict serializability — there is no weaker option.

The Commit Pipeline

This is the key to understanding FDB’s performance characteristics:

Client                  Proxy               Resolver            Transaction Log       Storage Servers
  │                       │                    │                      │                     │
  │  1. Begin Txn         │                    │                      │                     │
  │  (local, no network)  │                    │                      │                     │
  │                       │                    │                      │                     │
  │  2. Reads → GetReadVersion                 │                      │                     │
  │ ─────────────────────►│                    │                      │                     │
  │ ◄─────────────────────│ readVersion=v100   │                      │                     │
  │                       │                    │                      │                     │
  │  3. rt.Get(key)       │                    │                      │                     │
  │ ─────────────────────────────────────────────────────────────────────────────────────►  │
  │ ◄─────────────────────────────────────────────────────────────────────────────────────  │
  │  (reads served directly by Storage Servers — Proxy NOT in critical read path)           │
  │                       │                    │                      │                     │
  │  4. tr.Set(key,val)   │                    │                      │                     │
  │  (buffered locally, no network yet)        │                      │                     │
  │                       │                    │                      │                     │
  │  5. Commit: send {readVersion, readSet, writeSet}                  │                     │
  │ ─────────────────────►│                    │                      │                     │
  │                       │  6. Assign commitVersion=v101             │                     │
  │                       │ ──────────────────►│                      │                     │
  │                       │  7. Check conflicts│                      │                     │
  │                       │  (did anyone write a key in readSet       │                     │
  │                       │   between v100 and v101?)                 │                     │
  │                       │ ◄──────────────────│                      │                     │
  │                       │  8. If no conflict: write to TLog         │                     │
  │                       │ ─────────────────────────────────────────►│                     │
  │                       │  9. TLog confirms durability (f+1 copies) │                     │
  │                       │ ◄─────────────────────────────────────────│                     │
  │  10. Commit ack       │                    │                      │                     │
  │ ◄─────────────────────│                    │                      │                     │
  │                       │  (Storage Servers apply writes asynchronously from TLog)        │

Key insight from the pipeline:

1. Reads bypass the Proxy — they go directly to Storage Servers. This means read throughput scales linearly with Storage Server count.
2. Writes buffer locally — no network traffic until commit. A transaction that writes 1,000 keys generates exactly one network round-trip.
3. Conflict checking is done by the Resolver — a separate stateless process that checks whether any key in the transaction’s read set was written by another transaction between readVersion and commitVersion. No lock managers, no 2PL.
4. Durability before acknowledgment — FDB doesn’t ack a commit until the write is in the Transaction Log on f+1 machines. Storage Servers may lag behind.

Retry Semantics

The db.Transact(func) loop handles retries automatically. The function is called again if:

• A conflict is detected (error code 1020)
• The transaction is too old (error code 1007 — MVCC window expired)
• A transient network error occurs

The retry rule: the function passed to Transact must be idempotent for side effects outside FDB. FDB operations are always safe to retry (each retry gets a fresh transaction). But if your function sends an email, charges a credit card, or writes to another database, a retry will do that twice.

// WRONG: email sent on retry
db.Transact(func(tr fdb.Transaction) (interface{}, error) {
    tr.Set(key, value)
    sendWelcomeEmail(user)   // ← sent multiple times on conflict!
    return nil, nil
})

// RIGHT: defer side effects to after commit
_, err := db.Transact(func(tr fdb.Transaction) (interface{}, error) {
    tr.Set(key, value)
    return nil, nil
})
if err == nil {
    sendWelcomeEmail(user)  // ← exactly once
}

Read-Only Transactions

db.ReadTransact(func) opens a read-only transaction. It:

• Does not track a write set
• Does not need a commit round-trip
• Cannot conflict (reads never conflict with other reads)
• Uses a recent committed version as the read version

Read-only transactions are cheaper and should be preferred for any operation that only reads data.

2.4 MVCC — Multi-Version Concurrency Control

FDB stores multiple versions of each key, identified by a monotonically increasing version number (a 64-bit integer incremented globally for every commit):

Version timeline:
  v100: Set("color", "red")     ← committed at v100
  v101: Set("count", "5")       ← committed at v101
  v102: Set("color", "blue")    ← committed at v102
  v103: Set("count", "6")       ← committed at v103

A transaction reading at version v101 sees:

• "color" = "red" (latest version ≤ v101)
• "count" = "5" (latest version ≤ v101)

A transaction reading at version v103 sees:

• "color" = "blue" (latest version ≤ v103)
• "count" = "6" (latest version ≤ v103)

The 5-second window:

FDB garbage-collects old versions after approximately 5 seconds. This is configurable but the default. A transaction that starts at v100 and then tries to read 6 seconds later will get error 1007 (transaction_too_old). This is why long-running transactions are not supported in FDB.

This is a fundamental design choice. FDB optimizes for high-throughput short transactions. If you need to read data while performing long-running computation (e.g., a streaming job), the correct pattern is:

1. Read a batch of data, process it
2. Commit the batch
3. Read the next batch in a new transaction

MVCC vs. Locking

MVCC is strictly better for read-heavy workloads and any workload that mixes reads and writes.

2.5 Watches, Versionstamps, and Atomic Operations

These three primitives are rarely covered in introductory FDB material but are essential for building production systems.

Watches

tr.Watch(key) returns a FutureNil that fires when the key’s value changes. The watch is registered at commit time and fires asynchronously.

Production use cases:

• Distributed lock re-acquisition: hold a lock key; watch it to know when released
• Cache invalidation: watch a “cache bust” key; invalidate local cache when it fires
• Event notification: watch a “latest event” key; re-read when it changes
• Leader election: watch the leader key; trigger re-election when it’s cleared

// Polling-free pub/sub pattern
func watchForChanges(db fdb.Database, key fdb.Key) {
    for {
        var watch fdb.FutureNil
        db.Transact(func(tr fdb.Transaction) (interface{}, error) {
            watch = tr.Watch(key)
            return nil, nil
        })
        watch.Get()  // blocks until key changes
        // process the change...
    }
}

Limits: FDB recommends no more than 10,000 active watches per client. Each watch consumes resources on the Storage Server holding that key.

Versionstamps

A versionstamp is a 10-byte identifier that is globally unique and monotonically increasing. It consists of the commit version (8 bytes) plus a user-assigned offset (2 bytes) for ordering within one transaction.

tr.SetVersionstampedKey(keyTemplate, value): writes a key where one 10-byte slot in keyTemplate is replaced with the commit versionstamp.

// Building a totally ordered event log — no coordination required
keyTemplate := append([]byte("events:"), make([]byte, 10)...)  // 10-byte placeholder
keyTemplate = append(keyTemplate, []byte(":metadata")...)

tr.SetVersionstampedKey(fdb.Key(keyTemplate), eventPayload)
// After commit: key is "events:<10-byte versionstamp>:metadata"
// Every write gets a globally unique, strictly ordered key

Why this is powerful:

• A sequence of commits from different clients, on different machines, all get globally ordered keys
• No sequence number coordinator required
• Reading all events in order = a single GetRange("events:", "events\x01") scan

This is how fdb-record-layer implements change feeds and how any FDB-based event sourcing system should be built.

Atomic Operations

Atomic operations modify a key without reading it in the same transaction. They commute — two concurrent Add(key, 1) calls on the same key will both succeed, with a final result of 2, even if they had the same read version.

Critical property: atomic ops do not generate a read conflict for their key. Two transactions can both do tr.Add("counter", 1) without conflicting. This is fundamentally different from read-modify-write (which would conflict).

Under the hood: atomic ops are sent to the Storage Server as special mutation records that the server applies during version materialization. The server has the current value and applies the operation when making the value visible.

2.6 Cluster Architecture — Every Component Explained

                    ┌─────────────────────────────────┐
                    │         Client Process           │
                    │   fdb.OpenDatabase(clusterFile)  │
                    └────────────┬────────────────────┘
                                 │ reads cluster file once;
                                 │ caches routing table
                    ┌────────────▼────────────────────┐
                    │       Coordinators (3–5)         │
                    │  Paxos-based consensus           │
                    │  Store: cluster configuration    │
                    │  Elect: active Proxies, TLogs,   │
                    │         Data Distributor,        │
                    │         Ratekeeper                │
                    └────────────┬────────────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              ▼                  ▼                   ▼
    ┌─────────────────┐  ┌──────────────┐  ┌──────────────────┐
    │    Proxies      │  │   Resolvers  │  │  Transaction Log  │
    │ (GRV + commit)  │  │ (conflicts)  │  │  (WAL, durable)   │
    └────────┬────────┘  └──────────────┘  └──────────────────┘
             │ after conflict check + TLog commit
             │ Storage Servers apply writes asynchronously
    ┌────────▼──────────────────────────────────────┐
    │             Storage Servers (many)             │
    │  Each holds a shard (contiguous key range)    │
    │  Serves reads directly; applies mutations     │
    │  from TLog                                    │
    └────────────────────────────────────────────────┘

Proxies (GRV Proxies + Commit Proxies)

FDB 7.x splits proxy responsibilities:

• GRV Proxies (Get Read Version): assign read versions to transactions. This is a global operation — all reads in the cluster must see a consistent version ordering. GRV proxies serialize these assignments.
• Commit Proxies: receive commit requests, forward to Resolvers for conflict checking, then write to the Transaction Log.

Proxies are stateless (they hold no data). They can be added or removed without data migration.

Resolvers

Resolvers track which keys were written in each recent commit. When a Commit Proxy submits a transaction’s read set, the Resolver checks: “was any key in this read set written by a committed transaction between this transaction’s read version and now?” If yes → conflict, retry. If no → proceed.

Resolvers are in-memory only (they hold recent write history, not persistent data). Their state window matches the MVCC window (~5 seconds of commits).

Transaction Log

The TLog is FDB’s Write-Ahead Log — but distributed. Each commit is written to f+1 TLog machines before being acknowledged to the client. TLog uses its own durable storage (either fdatasync-safe files or SSDs).

After Storage Servers confirm they have applied all mutations up to a version (the “durable version”), TLog can truncate its older entries.

Storage Servers

Each Storage Server holds a subset of the key space (a shard). When a client wants to read a key, it looks up which Storage Server holds that key’s shard (from its cached routing table), and sends the read directly to that server.

Storage Servers use a custom storage engine (called “KeyValueStore” internally) built on a variant of SQLite or a custom B-tree, depending on configuration. Each mutation from the TLog is applied to this local store.

Drill-down: the storage engine deserves its own chapter — see Chapter 3 — Storage Engines for the full SQLite → Redwood → Sharded RocksDB story, with source pointers and page-layout diagrams.

Data Distribution: The Data Distributor process monitors shard sizes and access patterns. If a shard grows too large (>250MB by default), it splits it and migrates half to another Storage Server. This is transparent to clients.

Coordinators

Coordinators run a Paxos-like consensus protocol to maintain the “cluster controller” — the process that manages the lifecycle of all other FDB roles. If the cluster controller fails, coordinators elect a new one. Coordinators themselves are stateless beyond their Paxos log.

The cluster file (fdb.cluster) contains just the coordinator addresses: fdbdemo:fdbdemo@127.0.0.1:4500. This single file is how clients bootstrap their connection to any FDB cluster.

2.7 Production Limits and Sizing

These are not academic — they affect how you design your layer:

Production rule of thumb: keep transactions small (< 1 MB), short (< 1 second), and focused (few key ranges). This maximizes throughput and minimizes conflicts.

2.8 The Simulation Harness — Why FDB Is Trusted

FDB’s simulation framework is the most sophisticated correctness testing system in any open-source database. Understanding it explains why Apple, Snowflake, and others trust FDB with mission-critical data.

How Simulation Works

The entire FDB cluster — network stack, disk I/O, clocks, process scheduling — is implemented twice:

1. The real implementation (for production)
2. A deterministic simulation (for testing)

In simulation mode:

• All network calls are function calls (no actual sockets)
• Disk I/O is simulated in memory with configurable failure injection
• Clocks are virtual (controlled by the simulator)
• Processes are coroutines (no real threads)
• Random seeds control all non-determinism

The simulator injects:

• Machine failures: simulate killing any process at any time
• Network partitions: drop messages between specific nodes
• Disk failures: simulate corrupted writes, partial writes
• Clock skew: advance clocks non-uniformly across nodes
• Slow operations: simulate disk latency spikes

Why Determinism Matters

When a bug is found in simulation:

1. Record the random seed that triggered it
2. Reproduce the exact same sequence of events with the same seed
3. Add print statements, breakpoints, rerun — identical behavior every time

This is impossible with real distributed systems, where timing non-determinism makes bugs nearly unreproducible.

Scale of Testing

Before each FDB release, the simulation runs for thousands of machine-years worth of simulated time. A team member once stated: “We’ve found more bugs through simulation than through all other methods combined.”

Drill-down: the simulation harness, the Flow language it’s built on, and the source-level mechanics that make determinism possible are covered in Chapter 4 — Flow, Actors, and Simulation.

This testing discipline is why FDB can make strong correctness guarantees. It’s also why the FDB team is one of few database teams in the world that will confidently claim their ACID implementation is correct under arbitrary failure injection.

What This Means for You

When you use FDB as your storage substrate, you inherit these correctness guarantees. The five layers in this repository can fail in arbitrary ways — the processes can crash, the network can partition — and FDB will maintain consistency. The only failure that can cause data loss is the simultaneous loss of f+1 machines during a transaction commit window (where f is the configured fault tolerance level, typically 1 for a 3-machine cluster or 2 for a 5-machine cluster).

Interview Questions

Q: What is strict serializability and how does it differ from serializability?

Serializability means transactions appear to execute in some sequential order. Strict serializability (also called linearizability + serializability) additionally requires that the sequential order respects real-time ordering: if transaction T1 commits before transaction T2 begins, T1 must appear before T2 in the serial order. This matters for distributed systems where different clients observe commits at different times. FDB provides strict serializability. Most databases (PostgreSQL serializable, MySQL serializable) provide only serializability.

Q: How does FDB avoid deadlocks?

FDB doesn’t use locks. Instead, it uses optimistic concurrency control: reads are conflict-tracked, writes are buffered, and conflict detection happens at commit time. If a conflict is detected, the transaction retries with a new read version — it never “waits” for another transaction to release a lock. Since there’s no waiting, there can’t be a cycle of transactions waiting on each other.

Q: What happens to a running FDB transaction if the Proxy crashes?

The client’s Transact loop handles this. The proxy failure will manifest as a network error on the commit request. The FDB client library translates this to a retriable error, and Transact calls the function again with a new transaction and a new proxy (the client discovers the new proxy from the Coordinator). The retry is transparent to application code.

Q: Why does FDB limit transaction size to 10 MB?

FDB’s conflict resolution requires the Resolver to hold recent write history in memory. Large transactions would require holding large amounts of data in the Resolver’s conflict window. Additionally, the Transaction Log must write the entire transaction atomically before acknowledging. Large transactions create tail latency. The 10 MB limit keeps both of these bounded. For bulk loading, the correct approach is to batch writes in 1–5 MB transactions.

Q: What is a “read version” in FDB?

A read version is a 64-bit integer that represents a point in time in FDB’s commit history. Every committed transaction increments the global version counter. A transaction reading at version V sees all commits with version ≤ V and no commits with version > V. Read versions are assigned by GRV Proxies and represent a consistent snapshot of the entire cluster at that version.

Q: Can two FDB transactions that don’t share any keys conflict?

No. Conflict detection in FDB is purely key-based. Two transactions with disjoint key sets can never conflict, regardless of how close together they commit. This is why carefully designed key spaces (using subspaces to isolate different logical entities) dramatically reduce conflict rates in high-throughput applications.

Storage Engines: From SQLite to Redwood to Sharded RocksDB

• 3.1 The Storage Server Interface
• 3.2 Era One: KeyValueStoreSQLite (2013 – 2021)
  • What “ssd-1” actually does on a write
  • Why two flavors, ssd-1 and ssd-2?
  • Why it had to be replaced
• 3.3 Era Two: Redwood (2020 – present)
  • Why “versioned”?
  • Page layout
  • Copy-on-write writes
  • Concurrency
  • What to read in the source
  • How to know if you’re running Redwood
• 3.4 Era Three: Sharded RocksDB (2022 – present)
  • What “sharded” means here
  • What you give up
  • Where the gains are
  • When to pick which
• 3.5 The Memory Engine — Not Just for Testing
• 3.6 The “Outer MVCC” — How All Three Engines Look the Same to FDB
• 3.7 Putting It Together: A Worked Example
• 3.8 What This Means for the Labs
• Interview Questions

“FoundationDB’s storage engine is the part most outsiders get wrong. They picture a thin shim over RocksDB. The reality is three engines, fifteen years of evolution, and a bytecode interpreter that runs inside a B+tree.”

The previous chapter covered FoundationDB’s distributed plane: proxies, resolvers, transaction logs, and the cluster controller. This chapter zooms into the box labeled “Storage Server” and asks what is actually written to disk.

The answer has changed three times since FoundationDB became public in 2013:

If your fdbserver was compiled in the last year and you typed configure ssd, you got Redwood. If you typed configure ssd-rocksdb-v1 you got Sharded RocksDB. If you typed configure memory you got the in-memory engine. The original SQLite-based engine is still in the tree (ssd-1/ssd-2) and still used for catalog data; it is the engine the rest of the world used in production for the better part of a decade.

Why three engines? Because the abstract interface that a Storage Server needs is very narrow, but the physics underneath it are very picky.

3.1 The Storage Server Interface

A Storage Server is just an IKeyValueStore. The interface is defined in fdbserver/IKeyValueStore.h:

class IKeyValueStore : public IClosable {
public:
    virtual KeyValueStoreType getType() const = 0;

    virtual void  set(KeyValueRef keyValue, const Arena* arena = nullptr) = 0;
    virtual void  clear(KeyRangeRef range,  const Arena* arena = nullptr) = 0;
    virtual Future<Void> commit(bool sequential = false) = 0;

    virtual Future<Optional<Value>>      readValue(KeyRef key, ...) = 0;
    virtual Future<Standalone<RangeResultRef>>
        readRange(KeyRangeRef keys, int rowLimit, int byteLimit, ...) = 0;

    virtual StorageBytes getStorageBytes() const = 0;
};

Five operations: set, clear, commit, readValue, readRange. That’s it. Every storage engine in FDB implements those five and a handful of introspection methods. Everything you have ever heard about FDB — MVCC, sharding, replication, MVCC GC — lives above this interface.

That narrowness is what made it possible to swap SQLite for Redwood for RocksDB without changing the Resolver, the Transaction Log, the Data Distributor, the client library, or any layer.

The Storage Server itself (the role, as opposed to the engine) is implemented in fdbserver/storageserver.actor.cpp. That file is ~10,000 lines of Flow actors that:

1. Subscribe to the Transaction Log for mutations.
2. Buffer them in an in-memory MVCC structure called the VersionedMap (a persistent — i.e., immutable-snapshot — radix tree, see fdbserver/VersionedMap.h).
3. Periodically write a “durable version” of the VersionedMap into the underlying IKeyValueStore.
4. Serve reads by composing the durable engine’s data with anything newer in the VersionedMap.

So when you read a key from FDB, you are reading from two layers: a recent in-memory MVCC overlay, plus the durable on-disk B-tree (or LSM). The MVCC overlay is what gives you the 5-second window of historical reads; the durable engine is what gives you petabytes of capacity.

3.2 Era One: KeyValueStoreSQLite (2013 – 2021)

When FoundationDB Inc. was founded in 2009, the team made a deliberate decision: don’t write a new B-tree. B-trees are notoriously hard to get right — crash recovery, concurrency, page-level corruption — and SQLite’s B-tree had already been deployed on a couple billion devices and tested by Richard Hipp’s TH3 test suite, which famously exercises 100% MC/DC branch coverage.

So they took SQLite’s B-tree (btree.c, pager.c, os_unix.c), forked it, stripped out everything above it (the SQL parser, virtual machine, VFS layer), and wired the resulting key-value primitive into Flow as KeyValueStoreSQLite. The fork still lives in the repo at fdbserver/sqlite/ — the file names will look very familiar to anyone who has read SQLite source.

What “ssd-1” actually does on a write

Storage Server receives mutation: set("apple", "1") at version v100
   │
   ▼
KeyValueStoreSQLite::set()                            ← in-memory, just a queue
   │
   ▼  (commit called by storageserver.actor.cpp every "MUTATION_BATCH_INTERVAL"
   │   — typically 5 ms, configured by KNOB STORAGE_COMMIT_INTERVAL)
   │
KeyValueStoreSQLite::commit()
   │
   ├──► sqlite3BtreeBeginTrans(WRITE)
   ├──► for each mutation:  sqlite3BtreeInsert() / sqlite3BtreeDelete()
   ├──► sqlite3BtreeCommitPhaseOne()    ← writes WAL frame
   ├──► sqlite3BtreeCommitPhaseTwo()    ← fsync, then page move
   ▼
returns Future<Void> that fires when on disk

The SQLite engine runs in rollback-journal mode, not WAL mode (counter- intuitively — SQLite’s “WAL” is unrelated to FDB’s distributed Transaction Log). The rollback journal works like this:

1. Before modifying page X, copy X’s current contents to a side file (*.fdb-rj).
2. Modify page X in memory.
3. fsync the rollback journal.
4. Overwrite page X in the main file.
5. On crash recovery, if a rollback journal exists, restore each page from it.

This is the same algorithm as PostgreSQL’s older “physical log” approach. It gives crash atomicity at the page level. The cost: every page modification incurs two writes — one to the journal, one to the main file — plus an fsync.

Why two flavors, ssd-1 and ssd-2?

ssd-1 was the original. ssd-2 is the same engine with a btree corruption check and a tweaked page size (introduced in FDB 6.1). In practice you should not be choosing between them; both are legacy.

Why it had to be replaced

Three problems compounded as FDB went to multi-terabyte Storage Servers:

1. Page locking. SQLite serializes all access to its pager. Reads and writes contend on a single mutex. With Flow actors generating thousands of concurrent reads, the mutex became a bottleneck around 2016.
2. No MVCC. SQLite is a single-writer engine. Every commit has to wait for the previous one to finish. FDB worked around this by batching mutations, but the batch wall-time directly became commit tail latency.
3. Write amplification. With 4 KB pages and the rollback journal, write amplification was ≈ 6× even before SSD wear leveling.

By 2018 a Storage Server saturating a single NVMe at sustained write throughput was a known hardware-limited operation. The team decided to write a purpose-built engine that knew it was running inside an MVCC system.

3.3 Era Two: Redwood (2020 – present)

Redwood is a versioned, copy-on-write, prefix-compressed B+tree with delta encoding inside each page. It is implemented in a single ~10,000-line file: fdbserver/VersionedBTree.actor.cpp. The lead designer is Steve Atherton; his FDB Summit 2018 talk (“Redwood Storage Engine”) is required viewing.

Why “versioned”?

Recall from §3.1 that the Storage Server maintains an in-memory VersionedMap and periodically flushes a “durable version” to disk. With SQLite, “flush” meant replace — the on-disk state always represented one version, and older versions only existed in memory.

Redwood is fundamentally different: the B+tree itself is multi-versioned. Each page has a version. When you read at version V, the tree exposes the view that was current at version V — even if newer versions exist on disk.

Concretely:

Page #42  versions:  v100  →  [apple=1, banana=2]
                     v150  →  [apple=1, banana=2, cherry=3]
                     v200  →  [apple=9, banana=2, cherry=3]

Read at v160 returns:  [apple=1, banana=2, cherry=3]   ← uses v150
Read at v250 returns:  [apple=9, banana=2, cherry=3]   ← uses v200

This collapses three things into one structure:

1. The in-memory MVCC overlay (no longer needed for the 5-s window).
2. The durable on-disk tree.
3. The “checkpoint” / snapshot mechanism (a checkpoint is just a frozen version).

The win: a single commit writes new versions of touched pages once, with no journal copy and no rewrite. Write amplification drops from ≈6× to ≈1.5×.

Page layout

Each Redwood page is laid out (see BTreePage in the source, around line 2200 of VersionedBTree.actor.cpp):

┌──────────────────────────────────────────────────────────────┐
│ Header: page type (leaf|internal), height, version, flags    │ 16 B
├──────────────────────────────────────────────────────────────┤
│ DeltaTree: prefix-compressed sorted entries                  │
│                                                              │
│   anchor key: "user:1024:profile"                            │
│   delta 1   : +3 "_v1"           value=...                   │
│   delta 2   : +3 "_v2"           value=...                   │
│   delta 3   : skip 4 "_pic"      value=...                   │
│   ...                                                        │
│                                                              │
│   Each delta:  { prefix_len, suffix_bytes, value_or_ptr }    │
└──────────────────────────────────────────────────────────────┘

The DeltaTree2 structure (fdbserver/DeltaTree.h) is the cleverest part: it stores keys as deltas from the previous key in sorted order, then arranges those deltas in a balanced binary search tree inside the page so you can binary-search without decompressing the whole page. A typical FDB workload has 60–80% prefix overlap inside a page (keys like user:1024:posts:001, user:1024:posts:002, …); DeltaTree2 routinely compresses pages 3–5×.

Copy-on-write writes

Writes never modify a page in place. Instead:

1. The mutation logically applies to page #42.
2. A new physical page (say #9817) is allocated containing the new content.
3. Page #42’s parent in the B+tree is rewritten to point to #9817 (this recurses to the root — O(log N) new pages per commit).
4. The old page #42 is not freed immediately. It is kept alive until the oldest reader’s version is past it.
5. A background actor periodically scans the page allocator’s free queue and reclaims pages whose oldest_reader_version constraint is satisfied.

The free-page allocator is itself a B+tree (a “queue of free pages”), making allocation transactional. See FIFOQueue in fdbserver/FIFOQueue.h.

Concurrency

Redwood uses lock-free reads: a reader takes a snapshot of the root pointer for its version, then walks down. Because pages are copy-on-write, no other writer can mutate the pages the reader is walking — they will allocate new pages instead. This is the same trick used by LMDB and CouchDB.

There is exactly one writer thread per Redwood instance. With one writer, write serialization is implicit; no page locks needed.

What to read in the source

The DWALPager (“Direct-Write-Ahead-Log Pager”) is Redwood’s WAL. Like the distributed Transaction Log, it lets the engine acknowledge commits before the B+tree itself has been fully updated, then flushes lazily.

How to know if you’re running Redwood

fdbcli> status details

Look at each Storage Server line:

10.0.0.5:4500 (...)     class=storage      storage_engine=ssd-redwood-1

In Go-binding clients there is no API for this — it is configured at the cluster level via fdbcli> configure ssd-redwood-1-experimental (the suffix was dropped in 7.3, but the keyword survives for backward compat).

3.4 Era Three: Sharded RocksDB (2022 – present)

The third engine is a deliberate move toward leveraging a battle-tested LSM implementation. The motivation is not technical envy; it is operational reality:

• RocksDB has decades of investment in compaction tuning, write throttling, rate limiters, and observability hooks.
• Many FDB consumers (Snowflake, Apple) already run RocksDB elsewhere in their stack and have on-call expertise.
• LSMs match write-heavy workloads better than B+trees, and a meaningful fraction of FDB deployments are write-dominated (event logs, time series).

What “sharded” means here

This is not LevelDB-style. A single Storage Server in modern FDB hosts many shards (key ranges, typically ~250 MB each). In ssd-rocksdb-v1 each shard is its own RocksDB instance:

Storage Server process
  ├── shard "user:0000..." → /data/shards/abc123/  (one RocksDB tree)
  ├── shard "user:0001..." → /data/shards/def456/  (another RocksDB tree)
  ├── shard "user:0002..." → /data/shards/ghi789/  (another RocksDB tree)
  └── ...

Why per-shard instances? Because when Data Distribution moves a shard, the Storage Server can physically move the RocksDB directory instead of copying key by key. The destination server just opens the directory in place. This makes data movement nearly instantaneous, and bypasses the compactor entirely.

This trick was prototyped in Tigris’s FDB fork and brought upstream by the Snowflake team. The implementation lives in fdbserver/KeyValueStoreShardedRocksDB.actor.cpp.

What you give up

• Read amplification. LSM lookups must consult multiple SSTs. For point lookups bloom filters help; for range scans they don’t. Redwood beats RocksDB on read-heavy workloads by a non-trivial margin.
• MVCC native to the engine. The MVCC overlay must come back; Sharded RocksDB cannot serve historical reads natively, so the VersionedMap in storageserver.actor.cpp is doing real work again.
• Snapshot semantics. Cheap with copy-on-write, harder with LSMs.

Where the gains are

• Bulk ingestion. SST file ingestion (IngestExternalFile) lets backup restore at near-disk-bandwidth rates.
• Mature tuning knobs. Rate-limited compaction, prioritized flushes, block cache observability — all features of the engine itself.

When to pick which

3.5 The Memory Engine — Not Just for Testing

KeyValueStoreMemory (source) is sometimes dismissed as a test fixture. It is actually used in production for workloads where the entire dataset fits comfortably in RAM and ultra-low latency matters — Snowflake uses it for some metadata services.

Its design is straightforward:

• An in-memory radix tree holds all keys.
• A disk WAL records every mutation, fsynced before commit ack.
• On restart, the WAL is replayed to reconstruct the tree.
• Periodic compaction rewrites the WAL as a snapshot to bound restart time.

The latency win is real: a set is O(log N) pointer manipulations plus an append to an open log file. A 64-core Storage Server running the memory engine can sustain ~500,000 set/sec on a single NVMe, vs ~120,000 for Redwood on the same hardware. The cost is obvious — capacity is bounded by RAM.

3.6 The “Outer MVCC” — How All Three Engines Look the Same to FDB

Regardless of engine, the rest of FDB only ever sees:

struct StorageServerState {
    VersionedMap<KeyRef, ValueOrClearToRef> versionedData;   // in-memory MVCC overlay
    IKeyValueStore*                          storage;         // SQLite | Redwood | RocksDB
    Version                                  storageVersion;  // last durable version
    Version                                  oldestVersion;   // GC horizon (~5 s old)
    Version                                  latestVersion;   // most recent applied mutation
};

A read at version v:

1. Walk versionedData for the key at version v. If found, return.
2. Otherwise, read from storage (which holds the snapshot as of storageVersion).
3. If v < oldestVersion, return transaction_too_old (error 1007).

A mutation:

1. Update versionedData at the new version.
2. Update latestVersion.
3. Periodically (every ~5 ms or 5 MB, configurable via STORAGE_COMMIT_INTERVAL and STORAGE_COMMIT_BYTES knobs), flush the versionedData entries with version ≤ some cutoff into storage, then advance storageVersion.

The flush itself is the engine-specific part — Redwood writes copy-on-write pages, Sharded RocksDB writes to a memtable, SQLite writes to its journal.

This separation explains why FDB can “upgrade” your storage engine via a rolling restart: stop one Storage Server, change its config to use the new engine, start it back up, and the Data Distributor will gradually move shards to it. No data migration script — just shard relocation, which FDB does anyway.

3.7 Putting It Together: A Worked Example

A client commits a transaction at commitVersion = v100 that sets apple = 1 and clears banana. The Storage Server holding both keys is running Redwood. Here is the byte-by-byte journey:

Notice: one fsync per commit batch, not per mutation. With a 5 ms batch window and ~5,000 mutations per batch at peak load, this gives Redwood its characteristic ≈250,000 mutations/sec/disk throughput on commodity NVMe.

3.8 What This Means for the Labs

The five labs in this repo treat FDB as a black-box ordered KV store. But the performance of each lab is shaped by which engine is underneath:

• Option B labs (SQLite/LevelDB on FDB). They store pages (4–32 KB) as values. Redwood’s prefix compression buys you nothing here — pages don’t share prefixes. RocksDB sharded engine, with native block compression, is usually faster for this workload.
• Option A labs (KV API / SQL above FDB). They store many small keys with shared prefixes (table/pk/..., index/col/val/pk/...). Redwood’s DeltaTree2 will compress these aggressively. Redwood is the better default.
• Option C (Record Layer). Same prefix-heavy story as Option A. Redwood wins.

Try this experiment with the labs:

# Default cluster: Redwood
docker compose up -d
./scripts/bootstrap-fdb.sh
cd option-c-record-layer && go run ./demo

# Re-configure to memory engine and re-run
docker exec -it $(docker ps -qf name=fdb) fdbcli --exec "configure memory"
cd option-c-record-layer && go run ./demo
# Notice the latency drop on commits, but capacity is now RAM-bounded

Interview Questions

Q: Why did FoundationDB build Redwood instead of using RocksDB or LMDB?

Three reasons. First, RocksDB at the time of Redwood’s design (2017–2019) did not have efficient native MVCC; FDB already maintained an in-memory MVCC overlay and wanted to push that down into the engine. Second, FDB’s workload has heavy prefix overlap (long encoded keys like \x15\x01user\x00\x15\x01...) and RocksDB had no prefix compression inside a block. Third, FDB’s copy-on-write semantics are a poor match for LSM compaction overhead. Redwood was designed to fit FDB’s existing MVCC model rather than fight it.

Q: How does Redwood reclaim free pages without breaking historical reads?

Each page has a version when freed. The free-page queue stores tuples (pageID, versionFreed). The pager only re-allocates a page when versionFreed < oldestReaderVersion across all live transactions. Because FDB’s MVCC window is ~5 seconds, free pages typically take 5–10 s to become re-allocatable. The free queue is itself a B+tree, so the bookkeeping is transactional.

Q: What is the difference between FDB’s Transaction Log and an engine’s WAL?

FDB’s Transaction Log is the distributed WAL — f+1 copies of every commit written across machines before client ack. The engine’s WAL (e.g., Redwood’s DWALPager, SQLite’s rollback journal, RocksDB’s MANIFEST/WAL) is the local WAL for that one Storage Server, used to recover that server’s on-disk state if it crashes before its in-memory pages were flushed. The durability guarantee comes from the Transaction Log; the engine’s WAL is purely for local crash recovery.

Q: Why does the Memory engine still need a disk WAL?

So that commits survive a process crash. The in-memory radix tree is rebuilt from the WAL on startup. Without the WAL, a restart would lose data even if FDB’s distributed Transaction Log retained the commits — the engine couldn’t re-derive its state quickly. The WAL turns recovery into an O(WAL size) sequential read instead of an O(dataset) cross-cluster re-replication.

Q: A Storage Server is being moved to a new machine. Compare what happens under SQLite vs. Sharded RocksDB.

Under SQLite: the source Storage Server streams every key in the shard’s range to the destination over the network, which then writes each key into its own SQLite B-tree. Cost: O(shard size) bytes over network, O(shard size) random writes on the destination disk. Under Sharded RocksDB: the source copies the SST files of that shard’s RocksDB directory to the destination, which opens the directory directly. Cost: O(shard size) bytes over network, but writes on the destination are sequential SST file writes — no key-by-key insertion overhead. The latter is typically 5–10× faster wall-time.

Flow, Actors, and Simulation

• 4.1 The Problem Flow Solves
• 4.2 The Actor Model, FDB Edition
• 4.3 Futures and Promises
• 4.4 The Event Loop
• 4.5 Network = Just Another Future
• 4.6 The “Knobs” System
• 4.7 Simulation: How Bugs Get Caught Before You
  • How a simulation run works
  • Deterministic replay
  • Source pointers
• 4.8 Why You Care When You Use FDB from Go
• 4.9 Hands-On: Reading One Actor End-to-End
• Interview Questions

“Flow is the C++ language extension I always wanted but was too lazy to write. The FoundationDB team wrote it, and then they wrote a distributed database in it.” — paraphrased from a former Apple engineer

If you grep apple/foundationdb for ACTOR, you get ~7,500 hits. If you grep for Future<, you get ~25,000 more. Every line of FoundationDB — client library, Storage Server, Resolver, Proxy, Coordinator — is written in Flow, a C++14-compatible language extension for asynchronous, single-threaded, deterministic concurrency.

Flow is not a library. It is a source-to-source compiler that takes *.actor.cpp files, expands ACTOR / wait() / loop keywords into giant state machines, and emits ordinary *.g.cpp files that any C++14 compiler will accept. The Flow source is in flow/ and the compiler itself is in flow/actorcompiler/ — it is a 4,000-line C# program (yes, C# — leftover from when Apple’s tooling team owned the build).

You cannot read FoundationDB source without reading Flow. This chapter teaches the dialect.

4.1 The Problem Flow Solves

A database server is one of the world’s worst fits for traditional threading:

• A single Storage Server holds tens of thousands of simultaneous in-flight RPCs — reads, mutation pushes from TLogs, watch notifications, status pings.
• Each RPC may block on disk, network, or another RPC’s reply.
• The latency budget for a single read is ≈1 ms — most of which is network.
• Lock contention and context-switching costs are catastrophic at this scale.

The 2010-era options were:

• Threads + mutexes — too slow, deadlocks, race conditions.
• Callback chains (“callback hell”) — fast, but unreadable.
• Coroutines (Boost.Asio stackful) — readable, but slow and heavy.

The FDB team wanted callback performance with linear-code readability, plus something more: determinism, so that the simulation harness (§4.7) could reproduce any bug. None of the existing options gave determinism.

So they invented Flow.

4.2 The Actor Model, FDB Edition

In Flow, an Actor is a function that may suspend at any wait() call, yielding control back to the event loop (a single OS thread per process). While suspended, other actors run. When the awaited future completes, the actor resumes from where it left off.

ACTOR Future<Value> readWithRetry(Database db, Key key) {
    state Transaction tr(db);
    loop {
        try {
            Value v = wait(tr.get(key));
            return v;
        } catch (Error& e) {
            wait(tr.onError(e));
        }
    }
}

Three keywords are not standard C++:

• ACTOR — marks the function as Flow-compiled. The compiler will replace this with a class that holds the function’s local state and a step() method.
• state — marks a local variable as needing to survive across wait() calls. Plain locals are not preserved.
• wait() — suspends the actor until the given Future<T> is ready, then resumes with its value. Only legal inside ACTOR functions.
• loop — syntactic sugar for while (true) { ... }, distinct because the compiler optimizes the surrounding state machine.

What the compiler emits (simplified) for the above:

// machine-generated VersionedBTree.actor.g.cpp
class ReadWithRetryActor : public Actor<Value> {
    Database db;
    Key key;
    Transaction tr;             // ← state var, lives across waits
    int pc = 0;                  // program counter

    void step() {
        switch (pc) {
        case 0:                                       // initial entry
            tr = Transaction(db);
            pc = 1; [[fallthrough]];
        case 1: {                                     // loop top, before get
            Future<Value> f = tr.get(key);
            if (!f.isReady()) {
                f.addCallback([this]{ pc = 2; step(); });
                return;
            }
            pc = 2; [[fallthrough]];
        }
        case 2: {                                     // after wait, returned
            try {
                Value v = futureGet<Value>();
                sendResult(v);
                return;
            } catch (Error& e) {
                Future<Void> r = tr.onError(e);
                if (!r.isReady()) {
                    r.addCallback([this]{ pc = 1; step(); });
                    return;
                }
                pc = 1; goto case_1;                  // loop
            }
        }
        }
    }
};

That generated code is what your CPU actually executes. The ACTOR syntax is purely a compile-time convenience. You can read the actual generated .g.cpp files by building the project; CMake emits them into the build directory.

4.3 Futures and Promises

Flow’s Future<T> is not std::future<T>. It is a smart-pointer to a small SAV (“Single Assignment Variable”) object that holds either:

• the value T (once set), or
• a list of callbacks waiting for it, or
• an error.

Promise<T> is the writer end. The pattern:

Promise<int> p;
Future<int>  f = p.getFuture();

p.send(42);            // sets the value, fires all callbacks
// later:
int x = wait(f);       // (inside an ACTOR) returns 42 immediately if ready

The single-assignment rule (send exactly once) is enforced at runtime. There is no shared state synchronization, because the entire event loop runs on one thread. This eliminates a whole class of concurrency bugs and is the foundation of Flow’s determinism.

Source: flow/include/flow/flow.h, look for class Future, class Promise, class SAV.

4.4 The Event Loop

Net2 (flow/Net2.actor.cpp) is FDB’s event loop. Pseudo-code:

while (running) {
    Time now = monotonicTime();

    // 1. Run all ready callbacks
    while (!readyQueue.empty()) {
        readyQueue.pop()->fire();
    }

    // 2. Run any timers that have expired
    while (!timerHeap.empty() && timerHeap.top().t <= now) {
        timerHeap.pop().fire();
    }

    // 3. Poll the network (epoll/kqueue) and disk (io_uring/AIO)
    int timeoutMs = computeNextTimerDelay();
    pollEvents(timeoutMs);
    // ↑ pushes any newly-ready I/O completions onto readyQueue

    // 4. (optional) yield to the OS to maintain low priority for system tasks
    if (cpuHeavy) { sched_yield(); }
}

Three things to internalize:

1. One thread. Every wait() resumes on the same thread as the original ACTOR invocation. No data races, no mutexes inside Flow code.
2. Cooperative scheduling. An actor must wait() (or return) to yield. A CPU-heavy actor that doesn’t wait() will block every other actor in the process — including network I/O. Flow has utilities like yield(taskPriority) to voluntarily reschedule.
3. Priorities. Every actor runs at a TaskPriority (see flow/include/flow/Knobs.h). The event loop runs higher priorities first. This is how FDB keeps tail latency bounded: critical-path actors (commit pipeline) run at higher priority than background actors (data distribution, status reporting).

4.5 Network = Just Another Future

The single most elegant thing in Flow is that the network is presented exactly like any other asynchronous operation:

ACTOR Future<Void> echoServer(NetworkAddress addr) {
    state Reference<IListener> listener = INetworkConnections::net()->listen(addr);
    loop {
        Reference<IConnection> conn = wait(listener->accept());
        echoOne(conn);                          // fire-and-forget child actor
    }
}

ACTOR Future<Void> echoOne(Reference<IConnection> conn) {
    loop {
        wait(conn->onReadable());
        std::vector<uint8_t> buf(1024);
        int n = conn->read(buf.data(), buf.size());
        if (n == 0) return Void();
        wait(conn->onWritable());
        conn->write(buf.data(), n);
    }
}

IConnection is an interface. There are two implementations:

• Net2::Connection — backed by real Berkeley sockets via epoll/kqueue. Used in production.
• Sim2::Connection — backed by a simulated network. Used in tests. Same actor code runs unchanged.

This is the magic: you write one networking implementation, but you can run it on a real network or a simulator. The simulator can inject latency, drop packets, partition the cluster, even simulate clock skew — all without modifying a single line of the database code.

4.6 The “Knobs” System

Every tuning constant in FDB is a knob. They live in flow/include/flow/Knobs.h and per-module files like fdbserver/Knobs.h.

Examples relevant to performance tuning:

Knobs are tweakable at runtime by setting --knob_<NAME>=<VALUE> on the fdbserver command line. Changing knobs in production is strongly discouraged — most of them are co-tuned and undocumented in their interactions. But for performance experiments, knobs are the lever.

4.7 Simulation: How Bugs Get Caught Before You

The simulation harness — fdbserver -r simulation -f some_test.txt — is how the team finds bugs before customers do.

How a simulation run works

1. Boot: fdbserver creates a synthetic cluster of n processes (typically 5–15), each represented as a coroutine inside the same OS process.
2. Inject: a Sim2 instance replaces Net2. All INetworkConnections::net() calls now return simulated connections.
3. Time: monotonic time is virtual. delay(0.001) doesn’t sleep — it advances the virtual clock by 1 ms when there’s nothing else to do.
4. Workload: a test file (*.toml or *.txt in tests/) specifies which clients to run (Cycle, RandomTransactionWorkload, ClogTLog, etc.) and which failures to inject (Attrition, Clogging, MachineAttrition).
5. Fault injection: the simulator’s RNG (seeded from one 64-bit number) decides at every step whether to kill a process, partition the network, corrupt a disk page, slow a thread.
6. Validation: workloads verify their own invariants (Cycle checks that a ring of writes traverses correctly). If any actor calls ASSERT(...) and it fails, the simulator dumps the seed and exits.

Deterministic replay

Every random decision in the simulator goes through one PRNG, seeded by the test’s randomSeed. Two runs with the same seed produce identical event ordering. So when you see:

SimulatorTrace: failed assertion, seed = 1234567890123

…you re-run with --random-seed 1234567890123 and reproduce the bug deterministically — even on a different machine, even months later.

The discipline this imposes is fierce. Anything non-deterministic — system time, thread scheduling, hash table ordering — is banned from FDB code unless funneled through g_random. Code review at FDB famously rejects any std::unordered_map because its iteration order varies across allocator states. The codebase uses std::map or the Flow-provided ordered containers instead.

Source pointers

A worthwhile read for anyone learning Flow: open fdbserver/workloads/Cycle.actor.cpp. It is < 200 lines, uses every Flow primitive, and represents one of the canonical FDB correctness tests (the “ring cycle” — a sequence of transactions that should leave a ring of integers in a known configuration).

4.8 Why You Care When You Use FDB from Go

You write Go (or Java, Python, Ruby). You never write Flow. So why does this chapter matter to a layer developer?

Because the client library (libfdb_c.so) — which your Go bindings call through CGO — is itself written in Flow. Look at fdbclient/. Every Database, Transaction, Future object you use in Go is a thin handle over a Flow object running on a dedicated network thread inside your process.

This explains several things you might have noticed:

• Thread safety. fdb.Database is goroutine-safe even though its underlying state is mutable. Why? Because every operation eventually queues a callback onto the single Flow network thread.
• fdb.MustAPIVersion(710). Must be called exactly once per process. This boots the Flow event loop in a background thread. Subsequent fdb.OpenDatabase calls reuse it.
• Future.Get() blocking. Your goroutine Get()s a Future. Internally, the C client registers a callback on the Flow Future. When that fires (on the network thread), it signals a condition variable that wakes your goroutine. The hop is: Flow thread → CGO callback → Go scheduler → your goroutine resumes.
• The “transaction wait limit”. If your goroutine takes seconds between reads, you are not just risking MVCC expiry — you may also be holding network buffers and callback queues open on the Flow side.

Practical implication: keep transactions tight. Compute outside, transact inside. The Go binding is fast, but every CGO call costs ~200 ns and every Future has a Flow-side memory cost.

4.9 Hands-On: Reading One Actor End-to-End

Pick this real example — the function the Proxy calls to assign a commit version:

fdbserver/MasterServer.actor.cpp, search for ACTOR Future<Void> getVersion(. (In some 7.x branches this is in SequencerServer.actor.cpp.) The full function is ~60 lines and exercises nearly every Flow feature:

• state variables for the request batch.
• loop { ... } for accumulating requests within a 1-ms window.
• wait(delay(...)) to bound batch latency.
• Promise<...>::send(...) to fulfill each request after assigning versions.
• try / catch (Error& e) for the standard Flow retry pattern.

If you can read that function in one sitting and explain what it does, you can read 80% of the rest of FDB. The remaining 20% is Coordinator/Paxos and the storage engines — both covered in their own chapters here.

Interview Questions

Q: Why does FoundationDB run on a single thread per process?

To eliminate concurrent-access bugs in the database code and to make the simulation deterministic. Mutexes, atomics, and thread scheduling are sources of non-determinism that defeat replayable testing. A single thread makes every concurrent interaction explicit: it’s a wait() on a Future, not a hidden interleaving. Scalability across cores is achieved by running multiple Storage Server, Proxy, etc. processes on the same machine, not by threading within a process.

Q: What is the difference between wait(f) and f.get()?

wait(f) is legal only inside ACTOR functions. It suspends the current actor until f is ready, then resumes (without blocking the thread). f.get() is callable from anywhere but synchronously blocks the thread until f is ready — which inside Flow is a bug (it would freeze every other actor). f.get() is only used in test scaffolding and from non-Flow callers like the C client library’s Go binding bridge.

Q: Why are unordered_map and threads banned in FDB code?

Both are sources of non-determinism. unordered_map’s iteration order depends on hash-collision history. Threads introduce scheduling-dependent interleavings. Either breaks the property “run the simulator with the same seed twice and get identical execution,” which is the entire basis of FDB’s testing strategy.

Q: How does the simulation simulate disk I/O?

Sim2’s IAsyncFile implementation backs every file with a std::map of page bytes in RAM. Writes update the map; reads consult it. Latency is artificial: a write delays for a synthetic “disk latency” drawn from a configured distribution. Failures are also synthetic: at random times, chosen by the seeded PRNG, the simulator can drop pending writes (simulating a crash before flush) or corrupt a written page (simulating bit rot).

Q: A simulation run finds a bug at seed 0xABCD. The next day, on a new machine, you re-run with the same seed and get a different failure. What’s wrong?

Almost certainly someone introduced non-determinism — likely a new std::unordered_map, a direct gettimeofday(), or a thread-local. Standard FDB code review pattern: git bisect between the last good build and now, re-running the failing seed each time. The introduction of non-determinism is the bug; the original test failure is a symptom of whatever real bug the simulator was hunting.

Performance — Latency, Throughput, Concurrency in Numbers

• 5.1 Latency — Where Every Millisecond Goes
  • Latency by operation, real measurements
• 5.2 Throughput — What a Cluster Can Sustain
  • Single-node (this repo’s setup)
  • Production-shape cluster (3 × c5.9xlarge, single AZ, double replication)
  • Scaling rules of thumb
• 5.3 Concurrency — How Many Transactions Can You Have In Flight?
  • Conflict rate as a function of concurrency
  • How to reason about concurrency in your layer
• 5.4 Transaction Volume — Sizing the Cluster
• 5.5 I/O — Where the Bytes Go
• 5.6 Reproducing the Numbers in This Repo
• 5.7 The Performance Mental Model
• Interview Questions

“FDB is fast” is not engineering. “A 3-node FDB cluster on c5.4xlarge with Redwood sustains 55,000 cross-shard commits/sec at p99 = 7.8 ms” is engineering.

This chapter gives you concrete numbers, the workloads that produced them, and the model that lets you predict performance for your own workload before you measure it.

All numbers below are from published benchmarks (FoundationDB Summit 2021, Snowflake’s 2021 Engineering Blog, the official 7.0 release notes, and our own reproduction on the labs in this repo). Treat them as starting points, not contracts.

5.1 Latency — Where Every Millisecond Goes

Anatomy of a single read in a healthy local-datacenter cluster:

Client process              GRV Proxy          Storage Server
     │                          │                     │
     │── GetReadVersion ───────►│                     │           0.3 ms (RTT/2)
     │                          │ batch with peers    │           0.1 ms
     │                          │ assign version v    │           0.0 ms
     │◄── version=v ────────────│                     │           0.3 ms
     │── readValue(key)@v ─────────────────────────►  │           0.3 ms
     │                                                │  routing  0.0 ms (cached)
     │                                                │  MVCC     0.05 ms
     │                                                │  storage  0.1–10 ms ← block cache vs disk
     │◄── value ─────────────────────────────────────  │          0.3 ms
     │
                                                       Total:  ≈ 1.5 ms (cache hit)
                                                       Total:  ≈ 6 ms   (disk read)

Anatomy of a single commit (small transaction, single shard):

Client → Commit Proxy            0.3 ms
Commit Proxy → Resolvers         0.3 ms          ↘
Resolver conflict check          0.05 ms          ├ done in parallel
Resolvers → Commit Proxy         0.3 ms          ↙
Commit Proxy → TLogs             0.3 ms
TLog fsync + ack                 1–4 ms     ← dominant cost; SSD/NVMe dependent
TLogs → Commit Proxy             0.3 ms
Commit Proxy → Client            0.3 ms

Total: ≈ 3–6 ms (p50)   ≈ 8–15 ms (p99 on commodity NVMe)

The TLog fsync is the dominant cost in a commit. This is why FDB clusters that need low write latency are run on machines with battery-backed write caches or Intel Optane, where fsync is sub-millisecond.

Latency by operation, real measurements

Measured on the cluster bootstrap’d by this repo’s docker-compose.yml (single-machine, single process, Redwood engine, on a 2023 MacBook Pro M2):

Multi-node production clusters add 1–3 ms per round trip (cross-rack) or 30–100 ms (cross-region). FDB is firmly a single-region product — cross- region replication is supported but adds the round-trip latency to every commit.

5.2 Throughput — What a Cluster Can Sustain

Single-node (this repo’s setup)

Production-shape cluster (3 × c5.9xlarge, single AZ, double replication)

Published by the FDB Summit 2021 talk “FoundationDB at Snowflake”:

Scaling rules of thumb

• Read throughput scales linearly with Storage Server count, until you saturate either the network or the GRV Proxy.
• Commit throughput scales sub-linearly with TLog count. Each TLog handles ~50–80 K commits/s. Adding TLogs helps until the Commit Proxy becomes the bottleneck (~250 K commits/s per Commit Proxy).
• Adding Commit Proxies shards commit traffic by key range and scales almost linearly until the Resolver becomes saturated. A Resolver handles ~500 K conflict-resolution ops/s.

If your workload exceeds these single-process limits, the answer is add more of the bottleneck process, not a bigger machine. FDB scales out, not up.

5.3 Concurrency — How Many Transactions Can You Have In Flight?

The fundamental concurrency parameter is the MVCC window — the number of versions kept readable. Default ≈ 5 seconds × ~1 M versions/sec = 5 M.

This sets:

• Maximum concurrent transactions. Each transaction holds a read version. With ≈ 1 M versions/sec issued by GRV Proxies, holding 5 M versions means you can have about 5 M in-flight transactions — though in practice the per-Resolver memory limit binds first.
• Maximum transaction duration. Same 5 seconds, end-to-end.
• Resolver memory. Holds recent write-key sets. With 250 K commits/s × 5 s = 1.25 M commits, each with ~10 keys, that’s 12.5 M keys × ~64 B per key = 800 MB. Hence the 1 GB RESOLVER_STATE_MEMORY_LIMIT knob.

Conflict rate as a function of concurrency

The classic OCC formula: if N transactions run concurrently, each touching k keys uniformly at random from a key space of size M, the per-transaction conflict probability is approximately:

$$ P_{\text{conflict}} \approx 1 - \left(1 - \frac{k}{M}\right)^{N \cdot k} $$

For k = 10, M = 1,000,000, the conflict rate by N:

This is why hot key contention dominates real-world FDB performance. A hot key effectively shrinks M to 1 for those transactions, and the conflict rate goes to ~100%. The fix is almost always sharding by client ID or time, transforming (counter) into (counter, clientID % 64) and summing on read.

How to reason about concurrency in your layer

Two questions to answer for every layer you build:

1. What is the largest natural key prefix shared by transactions? If two transactions both write user:42:*, they conflict on the implicit “user 42” read of the schema row. Split the schema reads from the data writes.
2. What is your atomic-op opportunity rate? Anywhere you do “read-modify-write” of a counter, replace it with Add(). This removes the read from the read-conflict set entirely — atomic ops never conflict.

5.4 Transaction Volume — Sizing the Cluster

Capacity planning starts with the bottleneck. For each FDB component:

Worked example. You need to sustain 100 K commits/s × 50 keys per commit (= 5 M sets/s), 500 K reads/s, with 10 TB total storage.

• Commit Proxies: 100 K / 250 K = 1; round up to 2 for HA. ✓
• TLogs: 100 K / 50 K = 2; round up to 4 for replication. ✓
• Resolvers: 100 K / 500 K = 1. ✓
• Storage Servers: 5 M sets/s ÷ (≈100 K writes/s per server) = 50. Cross-check: 10 TB ÷ (≈200 GB/server target) = 50. ✓ ✓
• Coordinators: 3 (or 5) for Paxos. Standard.

Total: 50 storage servers + 2 commit proxies + 1 GRV proxy + 1 resolver + 4 TLogs + 3 coordinators = 61 roles. Pack them onto ~20 machines (multiple roles per machine is fine, except don’t co-locate TLog and Storage Server unless you have separate disks for each — they will compete for fsync).

This calculation is exactly what the fdbcli> configure command lets you declare: configure new double ssd commit_proxies=2 grv_proxies=1 resolvers=1 logs=4.

5.5 I/O — Where the Bytes Go

For a single commit of B bytes of writes on a triple-replicated cluster with one Resolver:

This is much better than RocksDB’s 10–30× and slightly worse than naive in-place B-tree (which would be ≈ 4× but loses durability + concurrency). On a write- heavy workload pushing 100 MB/s of application writes, the cluster’s aggregate disk write rate is ≈ 600 MB/s — plan SSD endurance accordingly.

For reads with cold cache, every read is one B+tree path: 3–5 page reads of 4–32 KB each = 12–160 KB of disk reads per logical key, served from the Storage Server’s page cache when hot.

5.6 Reproducing the Numbers in This Repo

Use the lab clusters to measure your local equivalent:

# Spin up FDB
docker compose up -d
./scripts/bootstrap-fdb.sh

# Quick latency probe using fdbcli
docker exec -it $(docker ps -qf name=fdb) bash -lc \
  "echo -e 'option on TIMEOUT 5000\nget x' | fdbcli"

# Throughput: use option-a-leveldb's batch demo (writes 100k keys)
cd option-a-leveldb && time go run ./demo

To get production-shape numbers, run the official benchmark:

# Inside the FDB container:
fdbcli> configure new double ssd
mako --mode build --cluster_file /etc/foundationdb/fdb.cluster \
     --rows 1000000 --commits 50 --keys_per_xact 10
mako --mode run --cluster_file /etc/foundationdb/fdb.cluster \
     --rows 1000000 --threads 32 --duration 60 --tps 100000

mako is FoundationDB’s official load generator (in bindings/c/test/mako/). Its output gives per-operation latency histograms, conflict rate, retry rate, and aggregate throughput — the same toolchain the FDB team uses for releases.

5.7 The Performance Mental Model

Internalize these and you can ballpark almost any FDB workload:

1. Reads are cheap. ~1 ms p50, scale linearly with Storage Servers. If reads are your bottleneck, add Storage Servers.
2. Commits cost one fsync. ~3 ms p50, scale linearly with TLogs. If commits are your bottleneck, add TLogs (and check that you’re not hitting a Resolver / Commit Proxy ceiling first).
3. Conflicts are quadratic in concurrency. Halve hot-key access, halve conflict rate^2 (because both the writer and the reader’s odds drop).
4. Write amplification is ~5×. Plan SSD endurance and bandwidth from that, not from application MB/s.
5. The 5-second MVCC window bounds transaction duration. Long-running work must be chunked into bounded transactions.
6. There is no cross-region magic. Cross-region adds RTT to every commit. Use a single region with multi-DC zoning for HA, not for low latency.

Interview Questions

Q: A production FDB cluster has p50 commit latency of 4 ms but p99 of 80 ms. What are the likely causes?

The p99 outliers almost always trace to (a) TLog fsync stalls on a contended disk — check disk service times and whether TLog and Storage Server share disks; (b) conflict-retry cascades, where a contested hot key forces many transactions into multiple retries — check the cluster’s conflict_rate and retry_rate metrics; or (c) shard movement triggering brief read or commit queueing — check Data Distributor activity. The fact that p50 is good rules out a sustained capacity problem; it’s an episodic stall.

Q: How would you measure read amplification of a specific layer?

Run the layer with FDB’s per-transaction stats enabled (tr.GetEstimatedRangeSize(...) and tr.GetApproximateSize() give you read byte and write byte counts). Sum reads-per-user-request and compare to the size of the user-visible result. For an index-heavy schema the ratio is typically 2–4× (one read for the index, one for the row, plus schema lookups). If it’s > 10×, you have a missing index or schema-cache opportunity.

Q: A new shard split happens every few seconds during a bulk load. Why, and how do you mitigate?

Default shard target size is ~250 MB. A bulk load that writes contiguously fills shards quickly, and the Data Distributor splits them. Each split copies data to a new Storage Server, briefly degrading throughput. Mitigation: randomize the prefix of bulk-loaded keys (e.g., prefix with a hash) so writes spread across many existing shards instead of growing one shard linearly. Alternatively, pre-split: write sentinel keys to force the Data Distributor to pre-create shards at known boundaries before the bulk load begins.

Q: Your application sees consistent 500 K reads/s but commits cap at 40 K/s. Cluster has 1 Commit Proxy, 1 Resolver, 2 TLogs, 12 Storage Servers. What do you change?

40 K commits/s is roughly the per-TLog ceiling × 1 (the slower TLog is the bottleneck of replicated commits). Add 2 more TLogs (logs=4). If commits still don’t scale, the Commit Proxy is next (250 K/s ceiling, but it spends RPC bandwidth fanning out to TLogs, so the practical ceiling is lower); add a second Commit Proxy. Reads are fine because they bypass the commit path entirely and hit Storage Servers directly.

Q: You need to support a 30-second analytical scan over 100 GB of data. FDB’s MVCC window is 5 seconds. What’s the pattern?

Don’t. Break the scan into ~1-second chunks, each its own transaction, each resuming from the last key of the previous chunk (tr.GetRange(lastKey+\x00, end)). Accept that you no longer have a consistent snapshot across chunks — if that matters, snapshot the data to a side store (object storage) first and analyze the snapshot. For production-grade analytics over FDB data, the canonical pattern is to stream changes (via versionstamped change-feed keys) into a downstream columnar store like ClickHouse or BigQuery.

The Layer Concept

• 3.1 Encoding Is Everything
  • Property 1: Unambiguous Decoding
  • Property 2: Co-location
  • Property 3: Minimal Write Amplification
  • Property 4: Sort-Preserving Encoding
• 3.2 The Subspace Pattern
• 3.3 The Tuple Layer — Industry Standard Encoding
• 3.4 Atomicity as a Design Axiom
  • Case 1: Split Insert (SQL Layer)
  • Case 2: Split Index Update (Record Layer)
  • Case 3: Split File Rename (LevelDB Storage Layer)
• 3.5 Key Space Design Patterns and Anti-Patterns
  • Pattern: Sequential Integer Keys
  • Pattern: Reverse Timestamp Keys
  • Pattern: Composite Keys for Multi-Dimensional Queries
  • Anti-Pattern: Large Values
  • Anti-Pattern: Hot Key Contention
• 3.6 The FDB Blob Layer Pattern
• 3.7 Schema Evolution — How Layers Survive Change
• Interview Questions

The layer concept is the central architectural idea of FoundationDB. FDB’s API surface is tiny: Get, Set, Clear, ClearRange, GetRange, and transactions. All the richness of a “database” — tables, indexes, files, records, schemas — must be built from these six primitives. This chapter explains exactly how.

Want the production-grade example? The next chapter, The Record Layer — A Deep Dive, pulls Apple’s open-source fdb-record-layer apart end-to-end: tuple layer, directory layer, FDBRecordStore, the planner, and a byte-by-byte walk-through of one saveRecord() call.

3.1 Encoding Is Everything

FDB stores byte strings. There is no concept of integer columns, string columns, or JSON fields at the FDB level. You must encode your data into bytes and decode it back out. The art of building a layer is entirely the art of encoding — turning rich data structures into sequences of (key, value) pairs such that:

1. Unambiguous decoding: given a raw key-value pair, you can reconstruct the original data
2. Co-location: related data is adjacent in key order, enabling efficient range scans
3. Minimal write amplification: updating one piece of data doesn’t force rewrites of unrelated data
4. Sort-preserving encoding: values you want to range-query sort correctly under byte comparison

All four properties must hold simultaneously. Let’s examine each.

Property 1: Unambiguous Decoding

Suppose you store two fields, name and city, in a single key:

key: "user:" + name + ":" + city

If name = "Alice:Paris" and city = "London", the key becomes "user:Alice:Paris:London", which looks identical to name = "Alice", city = "Paris:London". Ambiguous. Solution: choose a separator that cannot appear in field values, or escape it.

FDB’s Tuple layer solves this by escaping \x00 inside strings as \x00\xFF, then using bare \x00 as a separator. This makes encoding unambiguous for arbitrary byte strings.

Property 2: Co-location

Imagine storing a user’s profile fields as separate keys:

"user:alice:name"    → "Alice"
"user:alice:email"   → "alice@example.com"
"user:alice:city"    → "Paris"
"user:bob:name"      → "Bob"

Reading Alice’s full profile = one GetRange("user:alice:", "user:alice\x01") call. All three keys are adjacent in FDB’s sorted key space. One round-trip, no matter how many fields she has.

Contrast with storing all users in one key: "users" → msgpack([alice, bob, ...]). Reading one user requires reading the entire blob. Updating one user requires reading the entire blob, deserializing, modifying, re-serializing, and writing back. Terrible co-location.

Co-location means: keys for the same logical entity should share a prefix, so they sort together and can be read or deleted in one range operation.

Property 3: Minimal Write Amplification

Consider a user with 50 indexed fields. If a user moves from Paris to Tokyo, only two keys change: the old city index entry (cleared) and the new city index entry (set). The user’s record and all other index entries are untouched.

Designing your encoding so that “update one field” touches only the keys for that field is the goal. If your encoding stores the entire record in one value, every field update touches the entire value — wasteful when records are large.

Property 4: Sort-Preserving Encoding

If you want to range-query age BETWEEN 25 AND 40, the encoded age values must sort in the same order as the numeric values.

String encoding fails for integers:

"9"  > "30"  (byte comparison: '9' = 0x39 > '3' = 0x33)
→ sorted order: "1", "10", "100", "2", "20", "3", "30", "9" — wrong

Signed big-endian fails across sign boundary:

-1 → 0xFFFFFFFFFFFFFFFF  (sorts AFTER all positive numbers)

The correct encoding (used in every layer in this repo):

// encoding.go (option-c-record-layer)
func encodeInt64(x int64) []byte {
    b := make([]byte, 8)
    // Flip the sign bit. Maps signed range to unsigned range preserving order.
    // -2^63 → 0x0000...0000 (smallest unsigned)
    // -1    → 0x7FFF...FFFF
    //  0    → 0x8000...0000
    //  1    → 0x8000...0001
    //  2^63-1 → 0xFFFF...FFFF (largest unsigned)
    binary.BigEndian.PutUint64(b, uint64(x)^(1<<63))
    return b
}

This encoding is the industry standard for sort-preserving signed integer encoding. It appears in Apache HBase, Apache Cassandra, FDB’s own Tuple layer, and Google Bigtable.

Why XOR with 1<<63 is the same as adding 2^63: For any signed 64-bit integer x, uint64(x) XOR (1<<63) == uint64(x) + 2^63 when computed in 64-bit unsigned arithmetic. XOR is just the bit-manipulation shorthand for the offset encoding.

3.2 The Subspace Pattern

The single most important pattern in any FDB layer is the subspace: a byte prefix that namespaces all keys for a logical entity.

// encoding.go (option-a-leveldb)
type Subspace struct {
    prefix []byte
}

func (s Subspace) Pack(userKey []byte) fdb.Key {
    out := make([]byte, 0, len(s.prefix)+1+len(userKey))
    out = append(out, s.prefix...)
    out = append(out, 0x00)        // separator byte
    out = append(out, userKey...)
    return fdb.Key(out)
}

func (s Subspace) Range() fdb.KeyRange {
    begin := append([]byte{}, s.prefix...)
    begin = append(begin, 0x00)
    end := append([]byte{}, s.prefix...)
    end = append(end, 0x01)  // one byte past the separator
    return fdb.KeyRange{Begin: fdb.Key(begin), End: fdb.Key(end)}
}

Why the separator byte?

Without a separator, subspaces "foo" and "foobar" collide: the key "foobar\x00somekey" has "foo" as a prefix but was packed by the "foobar" subspace. The separator \x00 prevents this: "foo\x00" is never a prefix of "foobar\x00" because their 4th bytes differ (\x00 vs b).

The range trick — \x00 → \x01:

Incrementing the separator byte from \x00 to \x01 creates a tight upper bound. Every key in the "demo\x00" subspace sorts before "demo\x01". So GetRange("demo\x00", "demo\x01") returns exactly the keys for the “demo” namespace — no more, no less.

Subspace composition (nested subspaces):

You can build hierarchical key spaces by composing prefixes:

users subspace:    "app\x00users\x00"
orders subspace:   "app\x00orders\x00"
indexes subspace:  "app\x00idx\x00"
  city index:      "app\x00idx\x00city\x00"
    value "Paris": "app\x00idx\x00city\x00Paris\x00"

Each prefix cleanly separates a logical domain. ClearRange("app\x00users\x00", "app\x00users\x01") atomically deletes all users. ClearRange("app\x00idx\x00city\x00Paris\x00", "app\x00idx\x00city\x00Paris\x01") atomically deletes all city=“Paris” index entries.

All five layers in this repo use this pattern:

3.3 The Tuple Layer — Industry Standard Encoding

The official FDB client libraries (Python, Java, Go, Ruby) provide a Tuple encoding that standardizes the above patterns. It encodes lists of typed values into byte strings that sort correctly under FDB’s lexicographic comparison.

Tuple type codes and sort behavior:

Why the layers in this repo don’t use the Tuple layer:

Using the official Tuple layer would be correct and production-appropriate. The layers here use simpler hand-rolled encodings so that each file is readable without understanding the Tuple specification. When reading the source, you can see append(out, 0x00) instead of tuple.Pack(tuple.Tuple{schema, pk}) — the intent is obvious.

For production use, adopt the Tuple layer. It handles edge cases (null bytes in strings, arbitrarily large integers, nested structures) that the hand-rolled encodings in this repo skip.

3.4 Atomicity as a Design Axiom

The most important design discipline in FDB layer development: every logical operation that must be atomic must live inside one transaction.

This sounds obvious but is violated constantly by developers who treat FDB as a fancy cache. Let’s examine the consequences of each atomicity violation:

Case 1: Split Insert (SQL Layer)

// WRONG — two transactions for one logical insert
rowid := allocateRowid()           // Transaction 1: increment counter
writeRow(schema, rowid, values)    // Transaction 2: write row data

// If crash between T1 and T2:
// → counter incremented, row never written
// → counter is now permanently incorrect (rowid was "consumed")
// → future inserts skip this rowid

The correct approach (used in option-a-sqlite):

// RIGHT — one transaction
db.Transact(func(tr fdb.Transaction) (interface{}, error) {
    // Read and increment rowid counter
    rowid := tr.Get(rowidKey).Get() + 1
    tr.Set(rowidKey, encode(rowid))
    // Write the row
    tr.Set(rowKey(schema, rowid), encode(values))
    return nil, nil
})

Case 2: Split Index Update (Record Layer)

// WRONG — separate transactions for record and index
tr1.Set(recordKey, newRecord)        // T1: update record
tr2.Clear(oldIndexKey)               // T2: remove old index
tr2.Set(newIndexKey, nil)            // T2: add new index

// If crash between T1 and T2:
// → record shows new value
// → index still points to old value
// → LookupByIndex returns stale data forever
// → This is a phantom index entry — silent corruption

The correct approach (used in option-c-record-layer):

// RIGHT — all in one transaction
s.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
    // Read old record to know what indexes to remove
    prevBytes, _ := tr.Get(recordKey(s.ns, schema, pk)).Get()
    if prevBytes != nil {
        var prev Record
        msgpackUnmarshal(prevBytes, &prev)
        for _, f := range sc.Indexes {
            if v, ok := prev[f]; ok {
                tr.Clear(indexKey(s.ns, schema, f, v, pk))
            }
        }
    }
    // Write new record and new indexes
    tr.Set(recordKey(s.ns, schema, pk), encoded)
    for _, f := range sc.Indexes {
        if v, ok := rec[f]; ok {
            tr.Set(indexKey(s.ns, schema, f, v, pk), nil)
        }
    }
    return nil, nil
})

Case 3: Split File Rename (LevelDB Storage Layer)

// WRONG — separate transactions for copy and delete
copyFileData(newFd, oldFd)    // T1: copy all chunks
deleteFile(oldFd)              // T2: clear old chunks

// If crash between T1 and T2:
// → both files exist simultaneously
// → LevelDB manifest references old name
// → on restart, LevelDB sees duplicate SSTables → corruption

The correct approach (used in option-b-leveldb):

// RIGHT — one transaction for atomic rename
s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
    kvs, _ := tr.GetRange(s.dataRange(oldfd), ...).GetSliceWithError()
    tr.ClearRange(s.dataRange(oldfd))   // clear old
    tr.Clear(s.metaKey(oldfd))
    for _, kv := range kvs {            // write new
        tr.Set(translateKey(kv.Key, oldfd, newfd), kv.Value)
    }
    tr.Set(s.metaKey(newfd), metaBytes)
    return nil, nil
})

The pattern: read old state → clear old keys → write new keys — in one transaction. Crash at any point: transaction not committed, neither state is visible. Crash after commit: new state is fully visible. No intermediate state is ever observable.

3.5 Key Space Design Patterns and Anti-Patterns

Pattern: Sequential Integer Keys

key: ns + 0x00 + encode_uint64(rowid)

Values: row data. Row id is monotonically increasing. New rows always insert at the end of the range (append-like behavior in the key space). Range scans return rows in insertion order.

Production consideration: In a write-heavy workload, all inserts land on the same key space boundary, creating a hotspot on the FDB shard holding the highest rowid. FDB will split this shard, but the split lags the write rate. For very high insert rates, use a randomized prefix (hash of rowid) to distribute writes across shards.

Pattern: Reverse Timestamp Keys

key: ns + 0x00 + encode_uint64(MaxUint64 - timestamp)

Most recent entries sort first. A GetRange with no bounds returns entries in reverse chronological order. Useful for “recent events”, “activity feeds”, “audit logs”.

Pattern: Composite Keys for Multi-Dimensional Queries

key: ns + encode(field1) + 0x00 + encode(field2) + 0x00 + pk

Example: city + 0x00 + age + 0x00 + pk enables range queries on city AND age:

• All users in Paris sorted by age: GetRange(prefix + "Paris\x00", prefix + "Paris\x01")
• All users in Paris aged 25–40: GetRange(prefix + "Paris\x00" + encode(25), prefix + "Paris\x00" + encode(40) + "\xff")

This is the compound index pattern. It is exactly how MySQL’s compound indexes work.

Anti-Pattern: Large Values

Storing all of a user’s data in one 50 KB value:

key: ns + pk
value: msgpack({name, email, all_orders: [...], all_addresses: [...], ...})

Problems:

• Reading one field requires reading and deserializing the entire value (O(size))
• Updating one field rewrites the entire value (O(size) write amplification)
• The value may approach FDB’s 100 KB limit
• Hot users (many orders) create unbalanced value sizes

Better: normalize into separate keys per logical entity:

ns + 0x00 + pk              → profile (name, email)
ns + 0x01 + pk + orderid    → individual orders
ns + 0x02 + pk + addrid     → individual addresses

Anti-Pattern: Hot Key Contention

key: "global_counter"   ← every write touches this

All transactions that increment a counter conflict on this key. At high throughput, this becomes a serialization bottleneck.

Solutions:

1. FDB atomic ops: tr.Add("global_counter", 1) — atomic, no conflict, no retry needed
2. Sharded counters: split into N shard keys; read by summing all shards
3. Approximate counting: use a HyperLogLog or count-min sketch in a single key

3.6 The FDB Blob Layer Pattern

FDB’s value size limit (100 KB) is not a fundamental limitation — it’s a performance tuning decision. The solution is chunking, used in option-b-leveldb:

key: ns + 0x01 + fileType + fileNum + chunkNum  →  up to 65,536 bytes

To read a file: range-scan all chunks (sorted by chunkNum due to big-endian encoding), concatenate values.

This pattern generalizes to any large object:

• Files: split by byte range (64 KB chunks)
• Images: split by byte range or by tile
• Blobs: split by arbitrary chunk size
• Large JSON documents: split by logical sub-object

FDB’s official Blob Layer (in the FDB foundationdb-python-layers repository) implements this exact pattern. Amazon S3 and Google Cloud Storage’s consistency guarantees are actually powered by similar chunk-based storage backed by distributed KV stores.

3.7 Schema Evolution — How Layers Survive Change

A layer’s encoding is its schema. Changing the encoding without migrating old data is a common source of production incidents. Strategies:

1. Prefix-versioned keys:

ns + version_byte + ...rest

New code writes version = 0x02 keys; old code writes version = 0x01 keys. A background migration reads all 0x01 keys, converts, writes 0x02 keys. Read code handles both versions.

2. Additive-only changes: If your value is msgpack/Protobuf, adding new optional fields is backward compatible. Old code ignores unknown fields. New code sets defaults for missing fields. This is safe and zero-downtime.

3. Dual writes during migration: Write to old AND new key format simultaneously during a migration window. Verify correctness. Backfill old data. Cut over to new format. Remove old code path. Zero-downtime migration.

4. Never reuse key prefixes: Deleted keys leave no trace in FDB (there’s no tombstone value). But if you reuse a deleted subspace’s prefix for a new purpose, you risk stale data colliding with new data. Always use new prefixes for new logical entities.

Interview Questions

Q: Why is key ordering so important in FDB — can’t you just use a hash map?

A hash map gives O(1) point lookups but cannot efficiently answer range queries (“all users aged 25–40”), prefix queries (“all keys starting with ‘user:42:’”), or sorted iteration. Ordered storage enables all three with a single range scan. The entire record layer concept — where secondary indexes are just cleverly prefixed keys — relies on ordering. Without ordering, every query would require a full scan.

Q: What is a “subspace” in FDB and why is it important?

A subspace is a key prefix that namespaces a set of keys. It provides namespace isolation (keys in subspace A cannot collide with keys in subspace B), atomic deletion (one ClearRange deletes all keys in a subspace), and efficient range queries (one GetRange over the subspace prefix reads all related data). Every production FDB layer uses subspaces as the fundamental organizational unit.

Q: How do you handle null bytes inside keys in FDB?

FDB keys can contain any bytes, but if you use \x00 as a separator, null bytes inside field values create ambiguity. The official solution is the Tuple encoding: escape \x00 inside string values as \x00\xFF, and use bare \x00 as the separator. The decoder sees \x00\xFF and knows it’s an escaped null; bare \x00 ends the field. The layers in this repo use simple encodings that assume field values don’t contain null bytes — acceptable for a teaching implementation, not for production with arbitrary user data.

Q: Why is the PutRecord function in the record layer more expensive than a simple Set?

PutRecord must read the old record before writing the new one, so it can remove stale index entries. This adds one Get call to every write — a full round-trip to a Storage Server. The cost is necessary: without reading the old record, you can’t know which old index entries to remove. Production systems amortize this by batching multiple record updates in one transaction, or by using techniques like “shadow documents” (keeping old field values in the record itself to avoid the read).

Q: What is the difference between Transact and ReadTransact in FDB?

ReadTransact opens a read-only transaction: no conflict tracking, no commit round-trip, cheaper. It’s for operations that only read data. Transact opens a read-write transaction: tracks the read set for conflict detection, buffers writes, commits at the end. Use ReadTransact whenever you’re not writing anything — it’s faster and imposes less load on the cluster.

The FoundationDB Record Layer — A Deep Dive

• 7.1 Why a Record Layer?
• 7.2 The Layered Architecture
• 7.3 Tuple Layer — Order-Preserving Encoding
  • Worked example
• 7.4 Subspaces and the Directory Layer
• 7.5 The Record Layer Core — FDBRecordStore
  • The schema is protobuf
  • Opening a store
  • saveRecord byte-by-byte
  • Reading a record
  • Queries and the planner
• 7.6 Index Types in Detail
  • Online index builds
• 7.7 Concurrency, Conflicts, and the 5-Second Rule
• 7.8 How This Maps to the Toy Go Implementation
  • Running the lab
• 7.9 When Should You Use the Record Layer?
• 7.10 Reading Order for the Source
• 7.11 Interview Questions

The previous chapter taught you that a layer is just an encoding of a data model onto FDB’s ordered byte-string KV interface. This chapter takes the most production-grade example of that idea — Apple’s open-source foundationdb/fdb-record-layer — and pulls it apart end-to-end.

By the time you finish you should be able to:

1. Explain, on a whiteboard, how a Record Layer saveRecord() call becomes a set of FDB key-value mutations.
2. Read the upstream Java source and recognize the major subsystems.
3. Decide when to use the Record Layer instead of writing your own layer.
4. Map every concept back to the toy Go reimplementation in this repo (option-c-record-layer) so you can experiment in 100 lines of Go before touching the 200k-line Java codebase.

Reference revisions. Code references in this chapter point at the main branch of FoundationDB/fdb-record-layer and FDB release-7.3. Specific class names and file paths are stable across recent versions.

7.1 Why a Record Layer?

FDB itself gives you six operations: get, set, clear, clearRange, getRange, and a transaction wrapper. That’s a beautifully small surface, but if you build an application directly on it you will, within a week, have written:

• A primary-key encoder.
• A secondary-index maintainer.
• A schema-evolution story for when you add a field.
• A query planner that picks between scanning by primary key and scanning by index.
• Pagination with cursors that survive transaction boundaries.
• Type-safe Java/Kotlin/Swift bindings instead of raw byte arrays.

Every team that builds on FDB writes the same eight thousand lines of plumbing. Apple wrote them once, hardened them with CloudKit (production since 2015, billions of users), open-sourced them in 2018, and called them the Record Layer.

So a Record Layer is a layer on top of FDB that gives you:

The price is a Java dependency and a sizable conceptual surface. The rest of this chapter is that surface.

7.2 The Layered Architecture

The Record Layer is itself a stack of three layers above FDB:

┌──────────────────────────────────────────────────┐
│  Your application                                │
├──────────────────────────────────────────────────┤
│  RecordStore  (schema + index + query API)       │  ← fdb-record-layer-core
├──────────────────────────────────────────────────┤
│  Subspace / Tuple / Directory layer              │  ← fdb-extensions, Java tuple layer
├──────────────────────────────────────────────────┤
│  FDB transactions  (get/set/clear/getRange)      │  ← libfdb_java → libfdb_c
├──────────────────────────────────────────────────┤
│  FoundationDB cluster                            │
└──────────────────────────────────────────────────┘

Each of the three layers above FDB has a distinct responsibility:

1. Tuple layer — order-preserving binary encoding of typed tuples. Source: com.apple.foundationdb.tuple.Tuple in the Java bindings.
2. Subspace / Directory layer — namespace prefixes that turn one giant keyspace into a hierarchy of sub-keyspaces. Source: com.apple.foundationdb.subspace.Subspace, com.apple.foundationdb.directory.DirectoryLayer.
3. Record Layer core — protobuf-typed records, indexes, queries. Source: com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore.

We climb each in turn.

7.3 Tuple Layer — Order-Preserving Encoding

The tuple layer’s job is to convert a typed sequence like ("users", 42L, "alice") into a byte string whose lexicographic byte order matches the natural sort order of the original tuple. This is the hinge that everything else swings on, because FDB sorts keys by raw bytes.

The wire format uses a one-byte type tag per element, then the value:

The integer encoding is the cleverest bit. To make a 64-bit signed integer sort-correctly under byte comparison without using a fixed 8-byte width (which would waste space), the tuple layer uses 13 different type codes (0x0B through 0x1D) representing lengths from -8 bytes (very negative) to +8 bytes (very large positive), and the value bytes are written in big-endian with a complement for negatives. Decoding is just the inverse; encoding 42 takes 2 bytes total.

Read the source: com.apple.foundationdb.tuple.TupleUtil in bindings/java/src/main/com/apple/foundationdb/tuple/TupleUtil.java. The encoding rules are codified there and copied — bit for bit — in every official FDB binding so that a Go process and a Java process can interop.

Worked example

byte[] k = Tuple.from("users", 42L, "alice").pack();
// k = 02 75 73 65 72 73 00       ← "users"\0
//     15 2A                       ← int tag 0x15 = 1 byte positive, value 0x2A = 42
//     02 61 6C 69 63 65 00        ← "alice"\0

Now Tuple.from("users", 42L, "bob").pack() is byte-greater than the above because "bob" > "alice" lexicographically, and Tuple.from("users", 43L).pack() is byte-greater again because the integer prefix differs. Co-location and ordering come for free.

The Go toy version in this repo, encoding.go, sidesteps the full tuple encoding by using fixed 0x00 separators and assuming string fields don’t contain nulls. That’s fine for a demo, but the production tuple layer handles arbitrary bytes correctly.

7.4 Subspaces and the Directory Layer

A Subspace is just a tuple prefix plus convenience methods:

Subspace users = new Subspace(Tuple.from("app", "users"));
byte[] key  = users.pack(Tuple.from(42L, "alice"));
// equivalent to Tuple.from("app","users",42L,"alice").pack()
Range range = users.range();
// → from "app/users/\x00" to "app/users/\xFF"

This is com.apple.foundationdb.subspace.Subspace — a few hundred lines.

The Directory Layer is the next step up. Hard-coding "app/users" as a prefix in production is wasteful (it’s repeated millions of times in keys). The directory layer maps human-readable paths to short integer prefixes:

DirectoryLayer dir = DirectoryLayer.getDefault();
DirectorySubspace usersDir = dir.createOrOpen(tr, List.of("app", "users")).join();
// usersDir.getKey() might be \x15\x0C — a 2-byte allocated prefix

Underneath, the directory layer stores its mapping in a well-known subspace (\xFE) and allocates new short prefixes using a small high-contention allocator (HighContentionAllocator) that uses random partitioning to avoid hot-spotting. The allocator is itself an instructive chunk of source: com.apple.foundationdb.directory.HighContentionAllocator.

This matters because every Record Layer store lives in a directory. A multi-tenant Record-Layer-based service will typically have a KeySpacePath of the form /applications/{appId}/environments/{env}/recordStores/{storeId}, each piece resolved through the directory layer into a tiny integer prefix. Adding a tenant is one transaction; deleting one is one Transaction.clear(Range).

7.5 The Record Layer Core — FDBRecordStore

This is where it gets interesting. The central class is com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore. A record store is bound to:

• a Subspace (its root prefix, usually from the directory layer),
• a RecordMetaData (the schemas of the records it can hold),
• an open FDB transaction (record stores are short-lived, one-per-transaction objects).

The schema is protobuf

Schemas are .proto files. The Record Layer ships a small Protobuf extension package that adds annotations:

import "record_metadata_options.proto";

message User {
  required int64  id    = 1 [(field).primary_key = true];
  required string name  = 2;
  optional string email = 3 [(field).index = { type: "value" }];
  optional int64  signup_ts = 4;
}

message RecordTypeUnion {
  optional User _User = 1;
}

The RecordTypeUnion message is a oneof-style container so a single record store can hold heterogeneous record types in a shared key range.

RecordMetaDataBuilder reads the .proto, extracts the primary_key annotation, finds the field.index annotations, and builds an in-memory RecordMetaData describing every type and index. Source: com.apple.foundationdb.record.metadata.RecordMetaDataBuilder.

Opening a store

try (FDBDatabase db = FDBDatabaseFactory.instance().getDatabase()) {
  db.run(ctx -> {
    KeySpacePath path = users.add("env", "prod").add("store", "main");
    FDBRecordStore store = FDBRecordStore.newBuilder()
        .setContext(ctx)
        .setKeySpacePath(path)
        .setMetaDataProvider(metaData)
        .createOrOpen();
    // ... use store ...
    return null;
  });
}

The createOrOpen path runs a small store header read at offset <prefix>/0/info to verify schema compatibility. The header records:

• the format version of the on-disk layout,
• the user-supplied metadata version,
• whether the store is in the middle of an online index build.

If any of these don’t match what the caller expects, createOrOpen will either upgrade the header in place or throw, depending on the StoreExistenceCheck you passed. This is the schema-evolution machinery — there is no ALTER TABLE, just an in-place metadata version bump plus (optionally) a background index build.

saveRecord byte-by-byte

store.saveRecord(User.newBuilder()
    .setId(42)
    .setName("Alice")
    .setEmail("alice@example.com")
    .setSignupTs(System.currentTimeMillis())
    .build());

This single line expands into roughly the following FDB mutations inside one transaction. Assume the store prefix has resolved to S (a few bytes), and the union type tag for User is 1:

(Sub-key prefixes 0 = store header, 1 = records, 2 = secondary indexes, 3 = record versions, 4 = record-type index, 5 = aggregate indexes — see FDBRecordStoreKeyspace.)

Two things are worth pausing on:

1. Every byte is written in one FDB transaction. Records and all their indexes commit atomically. There is no “index out of sync with table” failure mode.
2. Record splitting (rows 2–3) is how the Record Layer handles records larger than FDB’s per-value limit (100 kB). The split key uses a fixed subkey suffix so range-reads can reassemble.

Reading a record

FDBStoredRecord<Message> rec = store.loadRecord(Tuple.from(42L));
User user = User.newBuilder().mergeFrom(rec.getRecord()).build();

loadRecord does a getRange over S / 1 / 1 / 42 / * (to pick up split parts), concatenates the bytes, decodes the protobuf, and wraps it in an FDBStoredRecord carrying the primary key and the optional versionstamp.

Queries and the planner

RecordQuery query = RecordQuery.newBuilder()
    .setRecordType("User")
    .setFilter(Query.field("email").equalsValue("alice@example.com"))
    .build();
RecordCursor<FDBQueriedRecord<Message>> cursor = store.executeQuery(query);

RecordQueryPlanner (in com.apple.foundationdb.record.query.plan.RecordQueryPlanner) takes the RecordQuery, inspects the metadata’s indexes, and emits a tree of RecordQueryPlan nodes. For the example above the plan is one RecordQueryIndexPlan over the email value index, followed by a RecordQueryLoadByKeysPlan to fetch each matched record.

Execution is streaming: RecordCursor<T> returns a record plus a continuation — an opaque byte string that resumes the scan from where it stopped. Continuations are the answer to “how do I paginate across transaction boundaries”, because FDB transactions are limited to 5 seconds and 10 MB of writes. A long scan is just many transactions, each fed the previous one’s continuation.

Read: RecordCursorIterator, KeyValueCursor, RecordQueryPlanner. The planner is ~3000 lines; it’s a rules-based planner, not (yet) cost-based, but the rule set is sophisticated — it understands compound indexes, covering indexes, sorted unions, and intersection.

7.6 Index Types in Detail

The Record Layer ships eight production index types. Each is implemented as a subclass of IndexMaintainer, registered through an IndexMaintainerFactory. The maintainer is called by saveRecord / deleteRecord with the old and new records and updates its key space accordingly. Source: com.apple.foundationdb.record.provider.foundationdb.indexes.

The brilliant trick for count/sum/min/max is that they use FDB’s atomic mutations (ADD, MAX, MIN) so concurrent writers never conflict on the aggregate key. Update 10 000 records in parallel; the count is always exact and no transaction has to retry.

rank is implemented as a skip-list-like layered index, also visible in the source as RankedSet (com.apple.foundationdb.async.rankset). It’s a standalone library you can use without records.

Online index builds

You can add an index to a deployed store without taking it down. OnlineIndexer (com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer) scans the records in chunks, indexing each in its own transaction, while new writes go through the freshly registered maintainer. The index is marked WRITE_ONLY until the back-fill completes, then promoted to READABLE. The promotion itself is a single transaction. This is exactly the trick MySQL ALTER TABLE … ALGORITHM=INPLACE does, only without the quirks, because the underlying transactional KV makes it straightforward.

7.7 Concurrency, Conflicts, and the 5-Second Rule

Record Layer transactions inherit FDB’s two hard limits:

• 5 seconds of wall-clock time between getReadVersion() and commit.
• 10 MB of writes per transaction.

So large workloads are decomposed into many small transactions, glued together by continuations. OnlineIndexer.IndexingPolicy is one example of this pattern; MultiTransactionalRunner is another.

Two patterns appear over and over in production code:

// Pattern A: chunked write
db.runAsync(ctx -> {
    FDBRecordStore store = openStore(ctx);
    return AsyncUtil.whileTrue(() -> store
        .scanRecords(continuation)
        .map(rec -> doSomething(rec))
        .onHasNext()
    ).thenApply(v -> store.getNextContinuation());
});

// Pattern B: read-modify-write with retry
db.run(ctx -> {
    FDBRecordStore store = openStore(ctx);
    User u = store.loadRecord(Tuple.from(id)).getRecord();
    User updated = u.toBuilder().setEmail(newEmail).build();
    store.saveRecord(updated);
    return null;
});  // db.run() retries on `not_committed` errors automatically

The cost-per-byte of those not_committed retries is precisely Chapter 5 §5.3’s conflict probability formula in action. If two transactions touch the same primary key and one commits, the other reads stale data and rolls back. If they touch only different primary keys, they commit without coordination.

7.8 How This Maps to the Toy Go Implementation

The option-c-record-layer lab in this repo is a deliberately minimal re-implementation of the same ideas — in Go, in roughly 400 lines, on top of the official Go FDB bindings. Use it as a Rosetta stone:

Look at option-c-record-layer/recordlayer/store.go. The PutRecord method does exactly steps 2–4 of the byte-by-byte table in §7.5 above:

1. Read the old record (to find its old index entries).
2. Clear those index entries.
3. Write the new record bytes.
4. Write the new index entries.

…all inside one fdb.Transact() call. That’s the whole shape; the production layer differs in features (rank, text, versioned indexes, splitting, planner, cursors, online builds) rather than in mechanism.

If you can read the Go file, you have the mental model for the Java one.

Running the lab

cd option-c-record-layer
go run ./demo
# Inserts a few users with a value index on `city`,
# then LookupByIndex("user", "city", "Paris") and prints the results.

Set a breakpoint inside PutRecord and watch the four FDB calls happen in order. That is the Record Layer in miniature.

7.9 When Should You Use the Record Layer?

Use it when you would otherwise build your own ORM-like layer on FDB: multi-tenant SaaS, document-style storage with secondary indexes, anything where a schema and indexes evolve over time. The Record Layer encapsulates five years of CloudKit production lessons; reinventing it is rarely the best use of your time.

Skip it when:

• You have a single, well-known access pattern and need maximum control over key layout (e.g., the LevelDB-API and SQLite-VFS labs in this repo).
• You don’t want a JVM dependency. (The Record Layer is Java-only today; community ports to other languages do not exist as of this writing.)
• You are doing low-level systems work where the Tuple + Directory + Subspace primitives are enough.

For comparison: Apple’s CloudKit is built on the Record Layer. Snowflake’s metadata store is built directly on the Java bindings, no Record Layer. Both choices are reasonable; they reflect different trade-offs about schema flexibility vs. layout control.

7.10 Reading Order for the Source

If you decide to learn the upstream source, read in this order. Each step is roughly one focused session.

1. bindings/java/src/main/com/apple/foundationdb/tuple/TupleUtil.java — the tuple encoding. Everything is downstream of this.
2. com.apple.foundationdb.subspace.Subspace and com.apple.foundationdb.directory.DirectoryLayer — namespacing.
3. com.apple.foundationdb.record.metadata.RecordMetaData and RecordMetaDataBuilder — how a .proto becomes runtime metadata.
4. com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore — saveRecord, loadRecord, deleteRecord. The core. Skim, don’t memorize.
5. com.apple.foundationdb.record.provider.foundationdb.indexes.ValueIndexMaintainer — the simplest index maintainer; read top to bottom.
6. com.apple.foundationdb.record.query.plan.RecordQueryPlanner — the planner. Read just plan(RecordQuery) first, then drill into one rule.
7. com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer — chunked index build with continuations. Beautiful, real example of the 5-second-rule pattern.
8. The integration tests under fdb-record-layer-core/src/test — they are the best documentation. FDBRecordStoreTestBase has the canonical setup.

After that, jump to the Contributing chapter for the PR workflow, which is essentially the same as FDB itself.

7.11 Interview Questions

Each of these is answered by something earlier in this chapter. If you can answer all five, you understand the Record Layer.

1. Why does the tuple layer use 13 different type codes for integers instead of always using 8 bytes?
2. How does the Record Layer keep a secondary index consistent with the record it points at, even under concurrent writes?
3. What is a continuation, and why does it exist? (Hint: 5-second transaction limit.)
4. A user wants to add a new index to a 1 TB record store without downtime. Walk through what OnlineIndexer does.
5. You have two record stores in different KeySpacePaths under the same FDB cluster. Can a single transaction span both? Why or why not? (Hint: yes — they share an FDB cluster — but you must be careful about the 5 s / 10 MB limits and conflict ranges.)

How Real Systems Use FDB

• 4.1 Apple’s FoundationDB Record Layer (iCloud / CloudKit)
• 4.2 FoundationDB Document Layer (MongoDB Wire Protocol)
• 4.3 mvsqlite — SQLite on FDB with Multi-Process MVCC
• 4.4 LevelDB’s goleveldb Storage Interface Pattern
• 4.5 Snowflake’s Use of FDB
• 4.6 Apple CloudKit — FDB at Hundreds of Millions of Devices Scale
• Interview Questions

One of the most useful ways to understand an architecture is to see it in production at scale. FoundationDB is unusual in that several of the systems built on it are open-source or at least publicly described. Each of the five labs in this repo maps directly to a pattern used in a real, deployed production system.

4.1 Apple’s FoundationDB Record Layer (iCloud / CloudKit)

The fdb-record-layer is a Java library open-sourced by Apple in 2018. It is the storage engine behind Apple’s CloudKit metadata services — the cross-device sync that backs Photos, Contacts, Notes, iMessage, and nearly every other iCloud-connected app.

What it provides:

• Protobuf-typed records (define your schema in .proto files)
• Declarative index definitions: value indexes, rank indexes, aggregate indexes, text indexes
• A query planner that compiles predicates into FDB range scans
• Schema evolution (add/remove/change fields without downtime)
• Continuations (long queries can be resumed across transactions)
• Change feeds via versionstamped keys

How the key layout maps to our option-c-record-layer:

fdb-record-layer uses a similar two-subspace structure:

[storePrefix] [META_DATA_VERSION] → version metadata
[storePrefix] [RECORD_TYPE_KEY]   → schema info
[storePrefix] [RECORD] [pk bytes] → serialized Protobuf record
[storePrefix] [INDEX] [indexName] [indexValue] [pk bytes] → index entry

Our record layer uses a simpler version:

ns + 0x00 + schema + 0x00 + pk    → msgpack record
ns + 0x01 + schema + 0x00 + field + 0x00 + value + 0x00 + pk → index entry (empty value)

The production scale: CloudKit syncs data for hundreds of millions of Apple devices. Every Photos upload, every note edit, every contact sync creates record layer transactions. FDB handles this via horizontal partitioning: different users’ data lives in different FDB shards, so per-user transaction load is low. The global throughput is enormous, but each individual transaction is small.

Key insight — continuations: fdb-record-layer’s most important production feature is the “continuation” — a cursor that can be serialized, stored, and resumed across transaction boundaries. Since FDB limits transaction duration to 5 seconds, long queries must be broken into chunks. A continuation saves the last-seen key so the next transaction can resume exactly where the previous one left off. Our labs don’t implement this, but every production layer needs it for full-table scans, bulk exports, and maintenance jobs.

Schema evolution in production: Adding a new field to a record type is safe (existing records just don’t have it; the field defaults to absent). Removing a field requires first removing all code that writes it, then removing the field from the schema. Renaming a field is the dangerous operation — it looks like “add new field + remove old field” but requires migrating all existing records. fdb-record-layer provides a dedicated schema migration API for this case.

4.2 FoundationDB Document Layer (MongoDB Wire Protocol)

The Document Layer was an open-source FDB layer that spoke the MongoDB wire protocol. A MongoDB client (driver, shell, or Compass) could point at the Document Layer unchanged, and all data was stored in FDB with ACID guarantees.

What it did:

• Parsed MongoDB wire protocol (BSON over TCP)
• Translated BSON documents into FDB key-value pairs
• Translated MongoDB queries ({city: "Paris", age: {$gte: 25}}) into FDB range scans and filters
• Provided MongoDB’s consistency guarantees (which are weaker than FDB’s — the Document Layer could offer stronger consistency than stock MongoDB by virtue of FDB’s serializable transactions)

Key encoding for a BSON document:

[collectionPrefix] [docId]               → BSON document value
[indexPrefix] [fieldName] [fieldValue]  [docId] → index entry

This is structurally identical to option-c-record-layer with schema = collectionName.

Why it was deprecated: The Document Layer was maintained by FoundationDB’s internal team at Apple. When Apple open-sourced FDB in 2018 and then in 2019 changed some library licensing, maintaining a full MongoDB-compatible query engine was outside scope. It was deprecated in 2019 but the codebase remains instructive as a reference implementation.

Lesson: A full relational/document query engine can be built atop FDB in ~50,000 lines of C++. The query planner complexity is in compiling predicate trees to efficient index scan strategies. The storage layer is just option-c-record-layer.

4.3 mvsqlite — SQLite on FDB with Multi-Process MVCC

mvsqlite is a Rust project by Heyang Zhou (losfair). It implements a SQLite VFS backed by FDB such that multiple processes can open the same “database” concurrently with full MVCC isolation.

How it maps to option-b-sqlite:

Our pagestore stores pages at:

ns + 0x01 + pageNum (big-endian) → 4096-byte page content

mvsqlite stores pages at:

[namespace] [pageNum_big_endian] → page content at that version

What mvsqlite adds beyond our implementation:

1. Page-level MVCC: Instead of storing just the current page, mvsqlite stores multiple historical versions. Old page versions are kept for active readers. This is implemented by appending a version prefix:

   [namespace] [pageNum] [snapshotVersion] → page at that version
   

   A reader at snapshot V only reads pages at version <= V. The GC process cleans up versions older than the oldest active reader.

2. Multi-process write serialization: mvsqlite serializes concurrent writers using an FDB transaction on a “write lock” key. The winner gets exclusive write access; losers retry. SQLite’s own xLock/xUnlock calls are translated to FDB key operations.

3. Write-ahead buffering in FDB: Instead of a local WAL file, mvsqlite buffers pending writes in FDB under a separate subspace. Commit flushes the buffer to the page subspace atomically.

4. Namespace mapping: SQLite filenames map to FDB key prefixes. ATTACH DATABASE 'db2.mvsqlite' AS db2 opens a second namespace in the same FDB cluster.

The production use case: A web application running in a multi-process or multi-instance environment (e.g., multiple Gunicorn workers, multiple Docker containers) can share a single mvsqlite database. Each worker reads from a consistent snapshot without locking other workers. Writes are serialized. This is impossible with standard SQLite (which only allows one writer and assumes a shared filesystem).

4.4 LevelDB’s goleveldb Storage Interface Pattern

The goleveldb storage.Storage interface (used in option-b-leveldb) is a seam designed for exactly this purpose: replacing the local filesystem with a remote storage backend while keeping all of LevelDB’s logic intact.

In practice, option-b-leveldb is analogous to how production systems host embedded databases on shared storage:

Real analog — etcd on block storage: etcd uses bbolt (a B-tree engine) for its data. When running in Kubernetes, etcd’s data directory is a persistent volume (a network-attached block device). The “storage.Storage” for etcd is the filesystem API on top of that network block device.

Real analog — TiKV’s RocksDB on FDB: TiKV, the storage layer behind TiDB, uses RocksDB. There are research prototypes that replace RocksDB’s Env (RocksDB’s equivalent of storage.Storage) with a remote storage backend backed by a distributed KV store.

Real analog — FoundationDB’s own Blob Layer: FDB’s official Blob Layer (in the Python layers repository) uses the exact chunking pattern from option-b-leveldb:

[prefix] [blobId] [chunkNum] → up to 10,000 bytes

The chunk size is tunable. Read performance is O(n_chunks) round-trips, optimized by using GetRange to fetch all chunks in one round-trip.

What makes option-b-leveldb interesting architecturally: LevelDB produces 7 types of files (log, sstable, manifest, current, lock, temp, info). Each has different I/O patterns: log files are append-only, sstables are write-once read-many, manifests are written atomically. Our storage.Storage implementation stores all file types as FDB key ranges, but a production implementation would optimize differently per file type (e.g., log files could use FDB’s atomic append via versionstamps rather than read-modify-write chunking).

4.5 Snowflake’s Use of FDB

Snowflake uses FoundationDB as its metadata store — the catalog that tracks table schemas, partition file locations, transaction history, and access controls. The actual table data is stored in cloud object storage (S3/Azure Blob/GCS), but the metadata that makes queries possible lives in FDB.

Why FDB for metadata?

• Cloud object storage is strongly consistent for object operations but cannot do atomic multi-object updates. Creating a table, writing partition files, and committing a transaction all need to appear atomically or not at all. FDB provides this.
• Metadata is small but highly contended. FDB’s optimistic concurrency and high throughput handle schema-level operations (DDL, partition registration) without bottlenecks.
• FDB’s key ordering makes catalog operations efficient: scan all partitions for table T, scan all columns in schema S, scan all transactions in time range [t1, t2].

The encoding (likely, based on public talks):

[catalog] [tenant] [schemaName] [tableName] [partitionId] → partition metadata
[txnLog]  [startVersion]                                  → transaction record
[schema]  [tenant] [schemaName] [tableName]               → column definitions

This is structurally option-a-sqlite (the SQL catalog pattern) extended to a distributed multi-tenant context.

The lesson: FDB is not used because Snowflake needed a storage engine for query data — S3 handles that. FDB is used because catalog metadata needs ACID guarantees that no other component in a cloud-native stack provides cheaply. If you’re building a distributed system that needs a consistent metadata store, FDB is frequently the correct answer.

4.6 Apple CloudKit — FDB at Hundreds of Millions of Devices Scale

CloudKit is Apple’s cross-device sync infrastructure. It stores photos metadata, notes content hashes, contact sync tokens, and app-defined data for third-party iOS apps. As of 2023, it services hundreds of millions of Apple devices.

The FDB usage in CloudKit:

• User records (contacts, notes, photo metadata) are stored as fdb-record-layer records
• Sync tokens are versionstamped keys that encode the exact FDB version at which a sync point was taken
• Conflict resolution uses FDB’s read-your-writes consistency to detect and merge concurrent edits

Why versionstamps are central to sync:

// Sync token = FDB versionstamp at the time of last sync
// Next sync: "give me all changes since versionstamp V"
// FDB key encoding: versionstamp → big-endian 10-byte key prefix
// GetRange from (V, MAX) returns all records created/modified after V

This is FDB’s versionstamp feature in action: each committed transaction gets a globally monotonic 10-byte version. Records written in that transaction embed the versionstamp in their key, enabling efficient “fetch all changes since X” queries with a single range scan.

Horizontal partitioning: Each CloudKit user’s data lives in a dedicated FDB keyspace (their userID is a subspace prefix). No two users’ transactions conflict because they operate on disjoint keys. This is how FDB achieves per-user isolation without database-per-user overhead.

Interview Questions

Q: Why would you use FDB over Postgres for metadata in a distributed system like Snowflake?

Postgres is a single-node database. To scale writes, you need sharding or replication, which introduces coordination complexity. FDB is horizontally scalable by design: adding machines increases throughput linearly. For metadata that needs global ACID guarantees across hundreds of nodes, a single-node Postgres (or even a replicated Postgres with synchronous replication) cannot match FDB’s throughput. FDB also runs in the same datacenter/cloud region as the application, eliminating cross-region round-trips for metadata operations.

Q: The Document Layer is deprecated — does that mean FDB isn’t good for document storage?

No. The Document Layer was deprecated because maintaining a MongoDB-compatible query engine (including the query planner, aggregation pipeline, and wire protocol) was expensive to maintain. The underlying pattern — document as FDB record, query predicates as range scans over index subspaces — is sound and is exactly what fdb-record-layer implements (with Protobuf instead of BSON). The Document Layer’s death was organizational, not architectural.

Q: How does mvsqlite enable multiple concurrent writers to the same SQLite database?

Standard SQLite uses POSIX file locks (or Windows file locks) to serialize writes. One writer holds a RESERVED lock; others wait. mvsqlite replaces file locks with FDB transactions on a “write lock” key. The writer that wins the FDB transaction proceeds; others retry. Because FDB’s transaction semantics are stronger than POSIX file locks (FDB provides serializable isolation; POSIX locks only prevent concurrent access), mvsqlite actually provides stronger guarantees than standard SQLite in a networked environment.

Q: What is a “continuation” in fdb-record-layer and why does every production FDB layer need one?

FDB limits transaction duration to 5 seconds. Any operation that might take longer — full-table scan, bulk export, index rebuild — must be broken into multiple transactions. A continuation is a serialized cursor: it records which keys have been processed so the next transaction can resume from the right place. Without continuations, a query that touches more records than fit in one 5-second window simply cannot be executed. In our labs, we ignore this constraint — all queries run in one transaction, which fails with a transaction-too-old error if the dataset is large enough. Adding continuation support is the single most important production readiness improvement for any of these layers.

This Repository — Five Implementations

• 5.1 Recommended Reading Order
• 5.2 Layer-by-Layer Architecture Map
  • option-a-leveldb (API Layer above FDB)
  • option-c-record-layer (Record + Index Layer)
  • option-a-sqlite (SQL Engine Layer)
  • option-b-sqlite (SQLite VFS Substrate)
  • option-b-leveldb (LevelDB Storage Substrate)
• 5.3 Running the Labs
• 5.4 Extending the Labs — Suggested Exercises

This repo contains five working implementations of FDB layers, ordered from conceptually simple to architecturally sophisticated. Each folder is self-contained: it has its own go.mod, source code, and docs/GUIDE.md with a deep dive into how and why that layer is built the way it is.

foundationdb-labs/
├── option-a-leveldb/     LevelDB API above FDB        → docs/GUIDE.md
├── option-a-sqlite/      SQL engine above FDB         → docs/GUIDE.md
├── option-c-record-layer/ Records + secondary indexes → docs/GUIDE.md
├── option-b-leveldb/     LevelDB on FDB storage       → docs/GUIDE.md
└── option-b-sqlite/      SQLite VFS substrate         → docs/GUIDE.md

5.1 Recommended Reading Order

Start with option-a-leveldb regardless of your background. It introduces all the core patterns — subspaces, range scans, snapshot reads, transaction retry loops — in the simplest possible context.

5.2 Layer-by-Layer Architecture Map

option-a-leveldb (API Layer above FDB)

LevelDB API (Get/Put/Delete/Batch/Iterator/Snapshot)
                        ↕
         FDB KV store (subspace-encoded keys)

The layer sits above FDB. User code calls db.Put(key, value). The layer encodes the key as ns + 0x00 + key and calls tr.Set(encodedKey, value). All LevelDB semantics (iterators, snapshots, batches) are implemented using FDB primitives.

Why this layer: LevelDB’s API is the lingua franca of embedded KV stores. By implementing it on FDB, existing code that uses LevelDB can be dropped into a distributed, replicated context without changing its API usage.

option-c-record-layer (Record + Index Layer)

Application (PutRecord / GetRecord / LookupByIndex)
                        ↕
   FDB (two subspaces: records and indexes)

The layer sits above FDB. A record is stored at a fixed key. Every indexed field also has an index key in a separate subspace. Both are written in the same transaction to keep records and indexes consistent.

Why this layer: This is the fundamental pattern for any “database” feature. Tables, collections, documents — all of them are “records with indexes.” Understanding how to keep a record and its indexes consistent under concurrent writes is the core of database internals.

option-a-sqlite (SQL Engine Layer)

SQL (CREATE TABLE / INSERT / SELECT / UPDATE / DELETE)
                        ↕
     Table catalog (FDB) + Row store (FDB)

A minimal SQL parser and query engine above FDB. Tables are defined in a catalog (a separate subspace). Rows are encoded as msgpack values keyed by primary key. SELECT * is a range scan over the table’s key range.

Why this layer: Shows that a full (if minimal) relational database is ~300 lines of Go above FDB. The interesting parts are the catalog encoding and the primary key sort-order problem.

option-b-sqlite (SQLite VFS Substrate)

SQLite (full SQL engine)
           ↕
   SQLite VFS API (xRead / xWrite / xSync / xLock)
           ↕
   FDB (page-keyed store: pageNum → 4096 bytes)

SQLite provides its own SQL engine, query planner, and transaction management. The VFS is just a storage substrate: given a logical page number, read or write 4096 bytes. FDB stores those pages.

Why this layer: Shows how to extend an existing system by replacing its storage interface rather than reimplementing its logic. The interesting parts are partial-page writes (SQLite calls xWrite with sub-page ranges, requiring read-modify-write) and locking (translating SQLite’s POSIX lock semantics to FDB key operations).

option-b-leveldb (LevelDB Storage Substrate)

LevelDB (full KV engine)
           ↕
   storage.Storage interface (Create / Open / Rename / List)
           ↕
   FDB (file metadata keys + chunked file data keys)

LevelDB provides its own full KV engine. The storage layer is just a filesystem abstraction: create/open/read/write/rename/delete files. FDB stores those files as key ranges.

Why this layer: Shows that even complex storage engines like LevelDB (which manage their own B-trees, compaction, and WAL) can be decoupled from the filesystem. The interesting parts are chunk encoding (64KB chunks to stay under FDB’s value limit), atomic rename (LevelDB uses rename to commit new SSTables), and why WAL redundancy is a non-issue (FDB’s own log subsystem replaces the LevelDB WAL’s durability purpose).

5.3 Running the Labs

Each lab has a demo/main.go that exercises the layer. To run any lab:

# Start FDB (requires Docker)
docker-compose up -d

# Run a specific demo
cd option-a-leveldb && go run demo/main.go
cd option-c-record-layer && go run demo/main.go
# ... etc.

Prerequisites:

• Docker + Docker Compose (for the FDB cluster)
• Go 1.21+
• The FDB C client library (installed by the bootstrap script: bash scripts/bootstrap-fdb.sh)

FDB API version: All labs use API version 710 (FDB 7.1.x). This is set at process start:

fdb.MustAPIVersion(710)

5.4 Extending the Labs — Suggested Exercises

Each docs/GUIDE.md has exercises specific to that lab. Here are cross-cutting improvements that apply to multiple labs:

1. Continuations (all labs): Add a ScanWithCursor(cursor []byte, limit int) function that returns at most limit results and a cursor byte string. The cursor encodes the last key seen. Next call resumes from cursor. This makes full-table scans safe under FDB’s 5-second transaction limit.

2. Transactions exposed to the caller (option-a-leveldb, option-a-sqlite): Wrap multiple operations in a user-visible transaction: db.Begin() → Tx, tx.Put(k, v), tx.Get(k), tx.Commit(). Internally, this is just db.Transact(func(tr) { ... }) — all operations share one FDB transaction.

3. Versionstamp-based change feed (option-c-record-layer): When writing a record, also write to a change log subspace: changelogKey = ns + changelogSubspace + versionstamp. A background consumer can do GetRange(lastCheckpoint, MAX) to efficiently read all changes since the last poll. This is how fdb-record-layer implements change feeds.

4. Range index queries (option-c-record-layer): Current index lookup finds exact matches (field = value). Add LookupByRange(schema, field, minValue, maxValue) that does GetRange(indexPrefix(field, minValue), indexPrefix(field, maxValue)). This requires your index value to be sort-preserving encoded (integers via encodeInt64, strings lexicographically).

5. Multi-file SQLite (option-b-sqlite): Support ATTACH DATABASE by mapping SQLite filenames to separate FDB subspaces. Two “attached” databases are just two different ns prefixes in FDB — no code duplication needed.

Reading Guide — Where to Go Next

• 6.1 FoundationDB — Going Deeper
  • Essential Reading
  • Source Code
• 6.2 Storage Engines — Going Deeper
  • Books
  • Papers
  • Source Code
• 6.3 SQLite VFS — Going Deeper
• 6.4 Distributed Systems Fundamentals
• 6.5 Building Something Real — A Progression

You’ve read through the five chapters of this guide and the lab implementation guides. By now you understand FDB’s architecture from the commit pipeline through every component of the cluster, the layer concept from byte-level encoding through production schema evolution, and how five real systems (Apple CloudKit, Snowflake, mvsqlite, Document Layer, Record Layer) deploy these exact patterns.

This chapter maps out the landscape of resources for going deeper in each direction.

6.1 FoundationDB — Going Deeper

Essential Reading

FDB Documentation — The official reference. The “Developer Guide” section covers API semantics in depth, including the precise semantics of read-your-writes consistency, exactly which operations conflict, and the full list of transaction options.

FDB White Paper — SIGMOD 2021 — “FoundationDB: A Distributed Unbundled Transactional Key Value Store.” The definitive technical reference for FDB’s internal architecture. Covers the simulation framework, commit pipeline, log system, and storage layer in academic depth. If you’ve read this guide, you have the vocabulary to understand every section of this paper.

FDB Forum — Design discussions, Q&A, and announcements. Some of the most insightful posts are from FDB’s core team explaining design decisions.

Source Code

apple/foundationdb — The full FDB C++ source. Key files:

• fdbserver/MasterProxyServer.actor.cpp → Commit Proxy implementation (the commit pipeline)
• fdbserver/Resolver.actor.cpp → Resolver (conflict detection)
• fdbserver/TLogServer.actor.cpp → Transaction Log (write-ahead log)
• fdbserver/StorageServer.actor.cpp → Storage Server (reads, MVCC)
• fdbclient/ReadYourWrites.actor.cpp → Client-side read-your-writes cache
• fdbserver/workloads/ → Simulation workloads (randomized fault testing)

FoundationDB/fdb-record-layer — Java. The most complete example of a production FDB layer. Study FDBRecordContext (transaction management), RecordQueryPlanner (query compilation to range scans), and OnlineIndexer (safe online index builds). Reading this source after this guide will be directly comprehensible.

6.2 Storage Engines — Going Deeper

Books

Designing Data-Intensive Applications by Martin Kleppmann — The best single-volume overview of storage trade-offs, replication, consistency, and distributed transactions. Chapters 3 (storage engines), 7 (transactions), and 9 (consistency) are directly relevant. If you haven’t read this book, stop here and read it. It contextualizes everything in this guide.

Database Internals by Alex Petrov — Deep dive into B-trees (B+ variants, page splits, page merges), LSM trees (all the internals: bloom filters, compaction algorithms, manifest management), and distributed consensus (Paxos, Raft, Multi-Paxos). The LSM chapter is the best public explanation of RocksDB/LevelDB internals.

Papers

The Log-Structured Merge Tree (1996) — The original LSM paper by O’Neil et al. Defines the C0/C1/C2 component model that LevelDB simplifies into level-0/level-1/…

Bigtable: A Distributed Storage System for Structured Data (2006) — The architecture paper for Google Bigtable. Introduced the tablet-server model, SSTable format, and hierarchical metadata server. Direct ancestor of HBase, Cassandra’s SSTables, and FDB’s storage layer design.

Spanner: Google’s Globally-Distributed Database (2012) — How Google built a globally distributed ACID database using TrueTime for external consistency. The “Paxos groups” concept is closely related to FDB’s shard-level durability. Essential reading for understanding distributed transactions.

Source Code

google/leveldb — The C++ LevelDB. Read: db/version_set.cc (compaction and manifest management), db/log_reader.cc + db/log_writer.cc (WAL format), table/block.cc + table/format.cc (SSTable on-disk format), util/arena.cc (memtable arena allocator).

syndtr/goleveldb — The Go LevelDB used in option-b-leveldb. The storage/ package defines the storage.Storage interface that our layer implements.

sqlite.org/sqlite — The SQLite source. Study: btree.c (B-tree implementation), vfs.c (VFS abstraction), pager.c (page cache and WAL). The SQLite source is famously well-commented.

facebook/rocksdb — RocksDB is LevelDB’s production successor. Much more complex (tiered compaction, bloom filters, column families, transactions) but the fundamental data model is identical.

6.3 SQLite VFS — Going Deeper

SQLite VFS Documentation — The full VFS API with method semantics. Essential for understanding what each of the xRead, xWrite, xSync, xLock, xUnlock methods must guarantee.

SQLite File Format — The page-by-page layout of a SQLite database file. After reading the option-b-sqlite guide, this document will make complete sense. The page header format, freelist, overflow pages, and B-tree cell formats are all documented here.

losfair/mvsqlite — The production FDB-backed SQLite VFS. The Rust code is clean and readable. The mvfs/ directory contains the VFS implementation; mvsqlite/ contains the page store. Compare with option-b-sqlite/pagestore/pagestore.go line by line.

SQLite WAL Mode — The WAL (Write-Ahead Logging) mode documentation. Explains how WAL provides concurrent reads and writes. Our pagestore uses rollback journal mode semantics (xSync is a no-op because FDB is durable), but understanding WAL mode helps you see what we’re not implementing.

6.4 Distributed Systems Fundamentals

The Part-Time Parliament (Paxos) — Lamport’s original Paxos paper. Dense, but the consensus problem it solves (how do N machines agree on one value when any machine can fail?) is the foundation of every distributed database.

In Search of an Understandable Consensus Algorithm (Raft) — The Raft paper. More accessible than Paxos. FDB uses a Paxos variant, not Raft, but the problems solved are identical. After this, read the Raft visualization.

Linearizability: A Correctness Condition for Concurrent Objects — The formal definition of linearizability (which FDB provides for reads and writes) and its relationship to sequential consistency and serializability.

A Critique of ANSI SQL Isolation Levels — The Berenson et al. paper that formalized isolation anomalies (dirty reads, non-repeatable reads, phantoms, lost updates, write skew) and showed that ANSI SQL’s definitions were imprecise. Essential vocabulary for discussing “serializable” vs “snapshot isolation” vs “repeatable read.”

6.5 Building Something Real — A Progression

If you want to go from “I understand these labs” to “I built something production-worthy,” here is a concrete progression:

Step 1: Add continuations to option-a-leveldb. Add IterateWithCursor(cursor []byte, limit int) ([]KV, nextCursor []byte). This is the single most important production feature missing from all the labs. Every real FDB application needs cursor-based pagination for range queries.

Step 2: Add range index queries to option-c-record-layer. Implement LookupByRange(schema, field string, minVal, maxVal interface{}) ([]Record, error). This requires: (a) sort-preserving encoding for all index value types, and (b) a range scan on the index subspace instead of a point lookup.

Step 3: Implement a simple query planner. For WHERE city='Paris' AND age >= 25, decide: which index scan is more selective? Scan city=‘Paris’ and filter age, or scan age >= 25 and filter city? Look at how fdb-record-layer’s RecordQueryPlanner.java makes this decision.

Step 4: Add online index building. Given a table with 1 million records, add a new index without downtime. The approach: (a) mark index as “building”, (b) background job scans records in cursor-paginated chunks and writes index entries, (c) any concurrent PutRecord/DeleteRecord writes to both old and new index state, (d) when background job finishes, mark index “ready”. This is how fdb-record-layer’s OnlineIndexer works.

Step 5: Read the fdb-record-layer source. After steps 1-4, open fdb-record-layer’s FDBRecordStore.java, RecordQueryPlanner.java, and OnlineIndexer.java. You will understand every design decision immediately. The gap between “labs exercise” and “production library used at Apple scale” is these four features: continuations, sort-preserving range indexes, query planning, and online index builds.

Contributing to FoundationDB

• 10.1 Building FoundationDB from Source
  • Prerequisites
  • The official path: build_run.sh
  • What gets built
  • Sanity check: run a simulation
• 10.2 Finding a First Bug to Fix
  • Source 1: The issue tracker, labeled appropriately
  • Source 2: Run the simulator until something breaks
  • Source 3: Performance investigation
• 10.3 The Contribution Workflow
  • Style and conventions
• 10.4 Where to Hang Out
• 10.5 First-PR Idea Catalog
• 10.6 Reading the Source — A Recommended Sequence
• 10.7 Beyond the Core: Layers and Tooling
• 10.8 The Long Game

“The best way to learn a database is to fix a bug in it.”

This chapter is the runway from “I read the book” to “I have a merged PR on apple/foundationdb.” It is opinionated and concrete. The official CONTRIBUTING.md is the canonical source; this chapter is the operational walkthrough.

10.1 Building FoundationDB from Source

The build is the gate that filters most would-be contributors. Block off an afternoon the first time. Once it works, it works forever.

Prerequisites

You need:

• A Linux machine (or a Linux VM / container on macOS). The macOS build is partially supported but most maintainers test on Linux. Use Linux.
• 16 GB RAM minimum, 32 GB recommended. The build is large.
• 40 GB free disk space.
• Docker (the official build uses a containerized toolchain).

The official path: build_run.sh

git clone https://github.com/apple/foundationdb.git
cd foundationdb

# Builds inside the official Docker toolchain image; output in build/
./contrib/Joshua/scripts/local_correctness.sh   # one-shot, prints a sim test result

For interactive development:

docker run -it --rm \
  -v $PWD:/foundationdb \
  -w /foundationdb \
  foundationdb/build:centos7-latest \
  bash

# Inside the container:
mkdir -p build && cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..
ninja -j$(nproc)

First build takes ~20 min on a fast laptop; incremental builds are usually < 1 min.

What gets built

Sanity check: run a simulation

./bin/fdbserver -r simulation -f ../tests/fast/CycleTest.toml

A successful run prints SimulatedFDBD: simulation finished successfully after 30 s to a few minutes. If you get an ASSERT failure, write down the seed — you may have already found a regression.

10.2 Finding a First Bug to Fix

Three reliable sources of approachable issues:

Source 1: The issue tracker, labeled appropriately

• good first issue label
• help wanted label
• documentation label

Documentation issues are the most under-served. The official docs at apple.github.io/foundationdb are maintained from documentation/sphinx/source/. A clear, well-tested documentation PR is almost always merged quickly and is excellent practice for the contribution workflow.

Source 2: Run the simulator until something breaks

The simulator is designed to find bugs that humans missed. Run it for hours:

for seed in $(seq 1 200); do
  ./bin/fdbserver -r simulation \
    -f tests/fast/RandomReadWriteTest.toml \
    --random-seed $seed \
    > /tmp/sim-$seed.log 2>&1 || echo "FAIL seed=$seed" 
done

If you find a reproducible failure on main that doesn’t appear on the most recent stable release, you’ve found a regression. File it with the seed; if you can also localize it via git bisect, you’re 80% to a merged PR.

Source 3: Performance investigation

The performance hotspots change with every release. Use perf against an fdbserver running a mako workload:

mako --mode build --rows 10000000 ...
perf record -g -p $(pgrep -f 'fdbserver.*storage') -- sleep 30
perf report --stdio | head -100

Anything taking > 5% of CPU in a place that doesn’t make architectural sense is a candidate for optimization. The storageserver.actor.cpp hot loop is a perennial source of incremental wins.

10.3 The Contribution Workflow

1. Open an issue first for anything non-trivial. The maintainers will tell you if (a) someone is already on it, (b) the approach is wrong, or (c) they’d prefer a different scope. Saves rework.
2. Sign the CLA. The bot will prompt on your first PR. Individual CLA is fine; corporate CLA needed if your employer holds your IP.
3. Branch from main. Topic branches like fix-storage-leak-12345.
4. One logical change per PR. Refactors stay separate from bug fixes. Maintainers reject mixed PRs quickly.
5. Add a simulation test for behavior changes. This is non-negotiable. The reviewer will ask. If you can’t write one, ask for help in the issue or on the forum — but the test must land in the same PR.
6. Run the relevant simulation tests locally before pushing:

   ctest -L fast -j$(nproc)         # ~15 min, runs the "fast" test suite
   

7. CI will run Joshua (the distributed sim runner) against your PR. Failures with seeds are surfaced in the PR comment. Reproduce locally with the printed seed.

Style and conventions

• C++17, formatted by clang-format using the repo’s .clang-format. Run ./contrib/clang-format-hook.sh or set up the pre-commit hook.
• No std::unordered_map, no raw gettimeofday, no std::thread. The determinism rules from Chapter 4 are enforced in review.
• Knobs go in Knobs.h with a sensible default; don’t add config flags unless absolutely necessary.
• Public Flow APIs (touched by the client library) need binding tests in bindings/c/test/ and equivalents for at least Python and Java.

10.4 Where to Hang Out

The FDB community is small but active.

Read at least one forum thread per week for a month before posting; you’ll absorb the team’s writing style and avoid asking already-answered questions.

10.5 First-PR Idea Catalog

If you want a concrete starting point, here are the kinds of contributions that historically have been merged from new contributors:

1. Documentation: clarify a confusing section. Pick a doc page that confused you while reading this book. Improve it. Submit it.
2. Add an example to the binding tests. Each language binding has a test/ subdirectory; many corner cases are under-tested. Add a test for tr.GetEstimatedRangeSize from your favorite binding.
3. Improve an error message. Run a test that hits an FDB error code, read what FDB tells you, and rewrite a confusing message in fdbclient/FDBTypes.h to be actionable.
4. Add a knob. Find a hardcoded constant in fdbserver/ that looks like it should be tunable (search for magic numbers in .actor.cpp files). Move it to Knobs.h, give it a default, and add a release note.
5. Write a workload. New simulation workloads live in fdbserver/workloads/. Pick a real-world pattern your job uses (e.g., a producer-consumer queue) and write a workload that exercises it under chaos. The simulator will probably find a bug for you.
6. Port a binding. There are unofficial bindings for Rust, Zig, Elixir. If your favorite language is missing, the C client API is small (~50 functions); a from-scratch binding is a great multi-month project.
7. Improve Redwood’s cold-start time. A known issue: opening a large Redwood store on cold startup can take minutes because the page metadata index is walked sequentially. Various heuristics (parallel walk, mmap, prefetch) have been proposed. Pick one, prototype it, benchmark it.

10.6 Reading the Source — A Recommended Sequence

Even before contributing, reading the source is the best self-education in distributed systems available outside graduate school. A recommended sequence:

1. flow/ — internalize the actor model.
   • flow.h, flow.cpp — Future, Promise, SAV.
   • Net2.actor.cpp — the event loop.
   • genericactors.actor.h — Flow’s “standard library.”
2. fdbrpc/ — RPC framework on top of Flow.
   • FlowTransport.actor.cpp — sending/receiving requests.
   • sim2.actor.cpp — the simulation backend.
3. fdbserver/ — read in this order:
   • MasterServer.actor.cpp (or SequencerServer.actor.cpp in newer versions) — version assignment, the heart of MVCC.
   • CommitProxyServer.actor.cpp — the commit pipeline.
   • Resolver.actor.cpp — conflict resolution.
   • TLogServer.actor.cpp — the distributed WAL.
   • storageserver.actor.cpp — the read/apply loop.
   • DataDistribution.actor.cpp — shard movement.
   • VersionedBTree.actor.cpp — Redwood, then DeltaTree.h.
   • ClusterController.actor.cpp — the role manager (last; it’s complex).
4. fdbclient/ — how clients see the cluster.
   • NativeAPI.actor.cpp — transaction lifecycle.
   • MultiVersionTransaction.actor.cpp — the client-version compatibility dance.
5. bindings/c/ — the C API your CGO Go binding actually calls.

If you give yourself one file per evening, you’ll read the whole essential codebase in a month — and you will be one of perhaps a few hundred people in the world who have.

10.7 Beyond the Core: Layers and Tooling

Contributing to the core engine is one path. Equally valuable:

• fdb-record-layer (FoundationDB/fdb-record-layer) — the Java layer used inside CloudKit. Less code than the core, very active.
• fdb-document-layer (FoundationDB/fdb-document-layer) — a MongoDB-compatible document store on FDB. Looking for maintainers.
• fdb-kubernetes-operator (FoundationDB/fdb-kubernetes-operator) — Go, very approachable, fast PR turnaround.
• Tigris (tigrisdata-community/tigris) — open-source serverless document DB on FDB.
• mvsqlite (losfair/mvsqlite) — runs SQLite on FDB, the inspiration for this repo’s option-b-sqlite lab.

These projects are smaller, faster-moving, and often very grateful for help. The patterns they use are the exact patterns this book covers.

10.8 The Long Game

The FoundationDB team is small. The maintainers know everyone who has landed more than a few patches. After 5–10 merged PRs you will be on a first-name basis with people whose names you originally saw in citations. After 50, you’ll be invited to design discussions and given commit privileges.

The path is real. The bar is high but not arcane: write correct C++, write deterministic Flow, run the simulator, follow the conventions, be patient in review. Every senior contributor started exactly where you are now.

“So long, and thanks for all the fish.”

Now go build something on FoundationDB — and then go fix something in it.

Option A — LevelDB API on top of FoundationDB

• Why this is interesting
• Files
• Key-space layout
• Mapping each LevelDB op to an FDB call
• Running

Pattern: “API layer above LevelDB” — we keep the familiar LevelDB surface (Put/Get/Delete/Batch/Iterator/Snapshot) but the bytes never touch LevelDB. They live in FoundationDB instead.

Why this is interesting

LevelDB is a local embedded KV. FoundationDB is a distributed transactional KV. Both speak (key, value) pairs over ordered byte keys, so the API shapes are nearly identical — but the guarantees underneath are wildly different:

The mapping is almost mechanical, and that’s the point: it makes FDB’s primitives concrete.

Files

layer/
  encoding.go   Subspace: ns + 0x00 + userKey → fdb.Key, plus range helpers
  db.go         Open / Close / Get / Put / Delete
  batch.go      Batch + DB.Write (atomic multi-op)
  iterator.go   Forward+backward cursor over GetRange results
  snapshot.go   MVCC snapshot via captured read version
demo/main.go    End-to-end: CRUD → batch → range → snapshot vs. live reads

Key-space layout

<namespace> 0x00 <userKey>  -> <value>

Subspace (see layer/encoding.go) keeps multiple logical databases isolated inside one FDB cluster. The 0x00 separator + the fact that the user-key portion is appended as opaque bytes gives us the same lexicographic ordering LevelDB users expect.

Mapping each LevelDB op to an FDB call

• Put(k,v) → db.Transact(func(tr){ tr.Set(ns.Pack(k), v) }). Transact handles automatic retry on conflict.
• Get(k) → db.ReadTransact(func(rt){ rt.Get(ns.Pack(k)).Get() }). An empty FDB result (nil) becomes our ErrNotFound.
• Delete(k) → tr.Clear(ns.Pack(k)) (no-op if absent).
• Batch.Write → a single Transact containing many Set/Clear ops. Either all of them land, or none do.
• Iterator → materialized list from tr.GetRange(subspace.Range()). Real production code would stream via RangeIterator, but materializing keeps the iterator usable outside a live transaction (the LevelDB shape).
• Snapshot → capture the read version with tr.GetReadVersion(), then on subsequent reads call tr.SetReadVersion(captured) so FDB serves the data as it was at that point in time.

Running

1. Start the shared FDB cluster (from the repo root):

   docker compose up -d
   ./scripts/bootstrap-fdb.sh
   

2. Install the FDB client library on the host (required by the Go bindings, which are CGO-linked to libfdb_c):

   brew install foundationdb        # macOS
   

3. Run the demo:

   cd option-a-leveldb
   

go run ./demo

The demo defaults to `../fdb.cluster` (or `FDB_CLUSTER_FILE` if set).

Expected output (abridged):

Get apple -> red batch applied (cherry+date inserted, banana deleted) range scan [a, z): apple = red cherry = red date = brown snapshot read version = 1234567 live Get apple -> green live Get cherry -> layer: not found snap Get apple -> red (was ‘red’ at snapshot) snap Get cherry -> red (err=)

That last block is the punchline: the snapshot sees the world before our
post-snapshot writes, because FDB just gives us older versions on demand.

## What this layer intentionally omits

- **Compaction / SST files** — there are none. FDB handles storage internally.
- **Bloom filters** — not needed; FDB indexes by key range natively.
- **Write batches with sequence numbers** — FDB's transaction commit version
  plays that role automatically (`tr.GetCommittedVersion()` after commit).

See `../option-b-leveldb` for the inverse experiment: keeping *real* LevelDB
code paths (memtable + SSTs) but storing the SST bytes in FDB.

Hitchhiker’s Guide — Option A: LevelDB API over FoundationDB

• Table of Contents
• 1. What LevelDB Is
• 2. The Five Concepts
  • Get — One Read, One Transaction
  • Put — One Write, One Transaction
  • Delete — Same as Put, but Clear
  • When Each One is Right
• 3. Subspaces — The Key Encoding
• 4. Write Batches — Atomicity Made Explicit
• 5. Iterators — Range Scans Without Cursors
• 6. Snapshots — MVCC Exposed to the Caller
• 7. The Demo, Step by Step
• 8. What the Real LevelDB Does That We Don’t
• 9. Real-World Analogue: goleveldb, RocksDB, PebbleDB
• 10. Exercises — Build on This
• 11. Source Code Deep Dive — Every Line Explained
  • db.go — The Core Database Type
  • encoding.go — The Full Subspace Implementation
  • iterator.go — Forward/Backward Navigation
  • snapshot.go — Pinning MVCC Versions
• 12. Production Considerations
  • 12.1 FDB Transaction Size Limits
  • 12.2 FDB Go Binding Concurrency Model
  • 12.3 Key Space Planning
  • 12.4 Monitoring and Observability
• 13. Interview Questions — LevelDB, MVCC, and FDB

The question this answers: “I have code that uses LevelDB. Can I swap in FoundationDB underneath with zero changes to my application?”

The deeper question: “What is LevelDB’s API contract, and how exactly does each piece of it map onto FDB primitives?”

Table of Contents

1. What LevelDB Is (and Isn’t)
2. The Five Concepts: Get, Put, Delete, Batch, Iterator, Snapshot
3. Subspaces — The Key Encoding
4. Write Batches — Atomicity Made Explicit
5. Iterators — Range Scans Without Cursors
6. Snapshots — MVCC Exposed to the Caller
7. How the Demo Works Step by Step
8. What the Real LevelDB Does That We Don’t
9. Real-World Analogue: goleveldb, RocksDB, PebbleDB
10. Exercises — Build on This

1. What LevelDB Is

LevelDB (released by Google in 2011) is an embedded, ordered, key–value store implemented as a Log-Structured Merge Tree (LSM tree). “Embedded” means it runs in the same process as your application — no server, no network. “Ordered” means the same thing as in FDB: keys are sorted lexicographically, and you can range-scan them efficiently.

LevelDB’s API surface is deliberately tiny:

db.Get(key)
db.Put(key, value)
db.Delete(key)
batch := new(leveldb.Batch)
batch.Put / batch.Delete
db.Write(batch)
iter := db.NewIterator(...)
snap := db.GetSnapshot()

This simplicity is why LevelDB became the embedded storage engine of choice for Chrome (IndexedDB), Bitcoin Core, and countless other applications.

Where LevelDB lives in the storage stack:

Application code
    ↓
LevelDB API (Get/Put/Delete/Batch/Iterator)
    ↓
LSM tree (MemTable + SSTables on disk)
    ↓
Filesystem / OS

Where our layer lives:

Application code
    ↓
[our layer] — same LevelDB-shaped API (Get/Put/Delete/Batch/Iterator)
    ↓
FDB transactions
    ↓
FDB cluster (distributed, replicated)

The application sees the same interface. The durability substrate changes completely.

2. The Five Concepts

Get — One Read, One Transaction

In LevelDB, Get opens a brief “read lock” (via a snapshot) and reads one key. There is no explicit transaction; LevelDB handles it internally.

In our layer:

func (d *DB) Get(key []byte) ([]byte, error) {
    v, err := d.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.Get(d.ns.Pack(key)).Get()
    })
    ...
}

ReadTransact opens a read-only FDB transaction (no conflict tracking on writes, cheaper than a full Transact). The read version is chosen by FDB to be a recent committed version — typically a few milliseconds behind real-time. This gives you a consistent view even if concurrent writers are active.

rt.Get(k) does not block on return. It returns a FutureByteSlice. Calling .Get() on the future is what blocks (sends the request to the FDB storage server and waits for the response). This two-phase call style is how FDB supports pipelining: you can call rt.Get(k1), rt.Get(k2), rt.Get(k3) in sequence, then .Get() all three — FDB sends all three requests before blocking on any response. This is critical for LookupByIndex in option-c, as we’ll see.

Put — One Write, One Transaction

func (d *DB) Put(key, value []byte) error {
    _, err := d.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        tr.Set(d.ns.Pack(key), value)
        return nil, nil
    })
    return err
}

Transact opens a read-write transaction. tr.Set adds the (key, value) pair to the transaction’s local write buffer. Nothing is sent to the cluster until the function returns without error, at which point FDB commits. If there was a conflict (another writer touched this key since our read version), FDB calls our function again with a fresh transaction automatically.

Delete — Same as Put, but Clear

tr.Clear(key) adds a tombstone to the write buffer. In FDB’s model, a cleared key is identical to a key that was never written — there is no “null” value. This is important: Get on a cleared key returns nil (which our layer translates to ErrNotFound), not a special tombstone value.

When Each One is Right

3. Subspaces — The Key Encoding

Every key that hits FDB is first passed through the Subspace.Pack method:

// encoding.go
type Subspace struct{ prefix []byte }

func (s Subspace) Pack(userKey []byte) fdb.Key {
    out := make([]byte, 0, len(s.prefix)+1+len(userKey))
    out = append(out, s.prefix...)
    out = append(out, 0x00)        // separator byte
    out = append(out, userKey...)
    return fdb.Key(out)
}

If your namespace is "demo" and your key is "apple", the actual FDB key is the byte string "demo\x00apple".

Why the separator byte?

Without a separator, two namespaces "foo" and "foobar" would collide: the key "foobar\x00somekey" would appear inside both foo’s range and foobar’s range. The separator \x00 prevents this because "foo\x00" is not a prefix of "foobar\x00".

Why 0x00 specifically?

Because 0x00 is the smallest possible byte value. When we compute the range end for a subspace scan, we copy the begin key and change the last byte from 0x00 to 0x01:

func (s Subspace) Range() fdb.KeyRange {
    begin := append([]byte{}, s.prefix...)
    begin = append(begin, 0x00)
    end := append([]byte{}, s.prefix...)
    end = append(end, 0x01)
    return fdb.KeyRange{Begin: fdb.Key(begin), End: fdb.Key(end)}
}

The range ["demo\x00", "demo\x01") contains exactly and only the keys packed by this subspace. This is a simple, efficient way to express “all keys in this namespace” without needing a sentinel end key.

The tuple layer alternative:

Official FDB client libraries encode subspace ranges using the Tuple encoding, which handles nested subspaces, escaping, and typed values. Our hand-rolled encoding is simpler but less general — for production use, adopt the Tuple layer.

4. Write Batches — Atomicity Made Explicit

LevelDB’s WriteBatch is the mechanism for writing multiple keys atomically. Without a batch, each Put is a separate transaction — if your process crashes between two Puts, the second one is missing.

b := layer.NewBatch()
b.Put([]byte("user:1:name"), []byte("Alice"))
b.Put([]byte("user:1:email"), []byte("alice@example.com"))
b.Put([]byte("user:1:score"), []byte("100"))
db.Write(b)

After Write, either all three keys exist or none do.

Our implementation:

// batch.go
func (d *DB) Write(b *Batch) error {
    _, err := d.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        for _, op := range b.ops {
            if op.clear {
                tr.Clear(d.ns.Pack(op.key))
            } else {
                tr.Set(d.ns.Pack(op.key), op.value)
            }
        }
        return nil, nil
    })
    return err
}

One Transact call. All ops go into the transaction buffer together. FDB commits them atomically. This is exactly what LevelDB’s WriteBatch does internally — it writes all ops to the Write-Ahead Log in one fsync, then applies them to the MemTable.

FDB’s size limits:

FDB transactions are capped at approximately 10 MB of reads + writes. For most use cases this is not a limit, but if you’re writing millions of keys you’ll need to split into multiple transactions. See option-b-leveldb for the chunking pattern.

5. Iterators — Range Scans Without Cursors

LevelDB’s iterator is a cursor over the sorted key space. It supports bidirectional movement and seeking to arbitrary positions.

The streaming vs. materializing decision:

A “streaming” iterator would keep a live FDB transaction open and use fdb.RangeIterator to fetch keys page by page as you call Next(). This is efficient for large ranges but ties the iterator’s lifetime to an open transaction.

FDB transactions have a ~5 second timeout. LevelDB iterators are frequently held open much longer (e.g., while a background compaction reads a full SSTable). Forcing a 5-second limit would break drop-in compatibility.

Our solution: materialize the entire range into a slice upfront:

func newIteratorAt(fdbDB fdb.Database, ns Subspace, start, end []byte, readVersion int64) *Iterator {
    v, _ := fdbDB.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(ns.RangeWithin(start, end), fdb.RangeOptions{}).GetSliceWithError()
    })
    it.kvs = v.([]fdb.KeyValue)
    ...
}

The entire GetSliceWithError call happens inside one transaction. The transaction closes. The iterator holds the materialized slice — it can outlive the transaction indefinitely.

Trade-off: We read all matching keys eagerly, even if the caller only needs the first few. For small ranges (as in most LevelDB use cases) this is fine. For ranges spanning millions of keys, a streaming approach would be necessary.

Navigation:

it.First()          // idx = 0
it.Next()           // idx++
it.Prev()           // idx--
it.Last()           // idx = len(kvs)-1
it.Seek(target)     // binary (or linear) search for key >= target
it.Key()            // kvs[idx].Key unpacked from subspace
it.Value()          // kvs[idx].Value
it.Valid()          // 0 <= idx < len(kvs)

The Seek in our implementation is a linear scan for pedagogical clarity. A production implementation would use sort.Search (binary search) since the slice is sorted.

6. Snapshots — MVCC Exposed to the Caller

This is where FDB’s MVCC machinery becomes directly visible.

A LevelDB snapshot captures the database state at a point in time. Reads through the snapshot always see that exact state, regardless of subsequent writes.

snap := db.NewSnapshot()
// ... later, after writes have occurred ...
v1, _ := db.Get(key)    // sees current state
v2, _ := snap.Get(key)  // sees state at snapshot time
// v1 != v2 if the key was mutated after the snapshot

How we implement this:

// snapshot.go
type Snapshot struct {
    db          fdb.Database
    readVersion int64
}

func (d *DB) NewSnapshot() (*Snapshot, error) {
    rv, err := d.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetReadVersion().Get()
    })
    ...
    return &Snapshot{db: d.fdb, readVersion: rv.(int64)}, nil
}

GetReadVersion() returns the logical timestamp FDB assigned to our transaction. This is a monotonically increasing integer — FDB’s “version clock”. We store it.

Later, when the snapshot is asked for a key:

func (s *Snapshot) Get(key []byte) ([]byte, error) {
    v, err := s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        tr.SetReadVersion(s.readVersion)   // ← the key line
        return tr.Get(s.ns.Pack(key)).Get()
    })
    ...
}

SetReadVersion pins the transaction to read from the exact version we captured. FDB’s MVCC machinery ensures that the storage servers still have the old version in their version history — provided it hasn’t been garbage collected (the ~5 second window).

Why SetReadVersion requires a writable Transaction:

This is a quirk of the FDB Go API: SetReadVersion is only available on fdb.Transaction (read-write), not on fdb.ReadTransaction (read-only). So we open a writable transaction but never write anything — effectively using it as a “pinnable read transaction”. The transaction is automatically abandoned when the closure returns.

What this maps to in real databases:

• PostgreSQL: BEGIN; SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; creates a snapshot valid for the transaction’s duration
• MySQL InnoDB: START TRANSACTION WITH CONSISTENT SNAPSHOT
• Spanner: BeginTransaction(mode: READ_ONLY, timestamp: exact_staleness: Duration)
• CockroachDB: AS OF SYSTEM TIME <timestamp> clause on SELECT

All of these are, at their core, “read from this specific version of the MVCC chain.”

7. The Demo, Step by Step

The demo in demo/main.go exercises every feature and prints the results. Here is the internal FDB call sequence for the MVCC section:

1. db.Put("color", "red")
   → Transact: Set("demo\x00color", "red"), commit version v1

2. snap = db.NewSnapshot()
   → ReadTransact: GetReadVersion() → returns v1 (or very close)
   → snap.readVersion = v1

3. db.Put("color", "blue")
   → Transact: Set("demo\x00color", "blue"), commit version v2

4. snap.Get("color")
   → Transact: SetReadVersion(v1), Get("demo\x00color")
   → FDB returns "red"  (value at v1)

5. db.Get("color")
   → ReadTransact: Get("demo\x00color")
   → FDB returns "blue" (latest committed value)

This demonstrates that the snapshot truly pins to v1, seeing the pre-mutation value.

8. What the Real LevelDB Does That We Don’t

LevelDB has significant machinery we skip:

LSM Tree internals:

• MemTable (skip list in memory) + WAL (write-ahead log on disk)
• SSTable files (sorted, immutable, bloom-filter indexed)
• Compaction (background merging of SSTables to reclaim space and bound read amplification)
• Bloom filters (avoid disk reads for non-existent keys)

None of this applies to our layer because FDB handles all of it. FDB’s storage servers use their own B-tree-like storage (a custom tree called the “FDB B-tree” that supports MVCC) and their own write-ahead logs. We simply call tr.Set and FDB’s internals handle the rest.

What we deliberately emulate:

• The API surface (same method names and semantics as goleveldb)
• Namespace isolation (subspace = virtual database)
• Atomic batches
• Forward/backward iteration
• MVCC snapshots

What we don’t emulate:

• Compaction (irrelevant — FDB handles it)
• Bloom filters (FDB has its own read optimization)
• File-level operations (no such thing in FDB)
• Checksums (FDB provides end-to-end data integrity)

9. Real-World Analogue: goleveldb, RocksDB, PebbleDB

goleveldb (github.com/syndtr/goleveldb) is the Go port of LevelDB that option-b-leveldb uses as a consumer. Its storage.Storage interface is the pluggable storage backend. We’ll see this in option-b’s guide.

RocksDB (Facebook/Meta, 2013) is the evolution of LevelDB: more configurable, multi-threaded compaction, column families, transactional API. It is the storage engine inside MySQL 8 (MyRocks), CockroachDB, TiKV, and many others. Every concept from our layer applies directly to RocksDB’s API.

PebbleDB (CockroachDB, 2019) is a Go implementation of RocksDB’s key ideas, designed for CockroachDB’s specific workload. CockroachDB switched from RocksDB to Pebble in 2021 for improved performance and simpler operations.

The common thread: all of these expose the same Get/Put/Delete/Batch/ Iterator/Snapshot interface. Building this interface over FDB means you understand the contract deeply, because you have to implement it rather than just use it.

10. Exercises — Build on This

These are not hypothetical. Each one adds a real capability:

Exercise 1 — Atomic Compare-And-Swap (CAS)

func (d *DB) CAS(key, expected, next []byte) (bool, error)

Read key, compare to expected, write next — all in one transaction. If the key changed since you read it, FDB retries automatically (conflict detection does this for you — you don’t need to loop).

Exercise 2 — TTL (Time-To-Live) Keys Store expiry timestamps alongside values. Modify Get to return ErrNotFound for expired keys, and add a Sweep() method that clears all expired entries with a ClearRange.

Exercise 3 — Transactions Across Multiple Keys

tx := db.Begin()
tx.Put("account:alice:balance", "900")
tx.Put("account:bob:balance", "100")
tx.Commit()

This is just a Batch.Write today. Extend it to include optimistic reads (read alice’s balance, check it’s >= 100 before subtracting) — you’ll need a real Transact closure, not just a batch.

Exercise 4 — Prefix Scan

rows, err := db.Scan(prefix []byte) ([]KV, error)

Use RangeWithin(prefix, nil) and filter server-side. This is the basis for the Record Layer’s ScanRecords.

Exercise 5 — Size Estimate Use FDB’s GetEstimatedRangeSizeBytes (an atomic op) to estimate how many bytes live in your subspace. This is how database engines implement SHOW TABLE STATUS without a full scan.

11. Source Code Deep Dive — Every Line Explained

This section walks through the full source of layer/db.go, layer/encoding.go, layer/iterator.go, and layer/snapshot.go with annotations about the non-obvious decisions.

db.go — The Core Database Type

type DB struct {
    fdb fdb.Database
    ns  Subspace
}

Two fields. fdb is the FDB connection (goroutine-safe, long-lived). ns is the namespace: a byte prefix prepended to every key. Multiple DB instances on the same FDB cluster with different ns values are completely isolated — their key ranges do not overlap.

func Open(fdbDB fdb.Database, namespace []byte) *DB {
    return &DB{fdb: fdbDB, ns: NewSubspace(namespace)}
}

Open does not contact FDB. It’s a pure in-memory initialization. The connection to FDB was already established when fdb.OpenDefault() was called in main.go. Open just associates this DB with a prefix.

Why take fdb.Database rather than a cluster address string? This lets the caller decide how the FDB connection is configured (API version, cluster file path, network options) and share one connection across multiple DB instances. Multiple DB instances share one FDB network thread and one connection pool.

func (d *DB) FDB() fdb.Database { return d.fdb }
func (d *DB) Namespace() []byte { return d.ns.prefix }

Accessors for embedding and testing. FDB() lets a consumer pass the FDB connection to another layer (e.g., a Record Layer built on top of this DB). Namespace() lets tests inspect the key prefix.

encoding.go — The Full Subspace Implementation

func (s Subspace) RangeWithin(start, end []byte) fdb.KeyRange {
    var begin, endKey fdb.Key
    if start == nil {
        begin = s.Range().Begin
    } else {
        begin = s.Pack(start)
    }
    if end == nil {
        endKey = s.Range().End
    } else {
        endKey = s.Pack(end)
    }
    return fdb.KeyRange{Begin: begin, End: endKey}
}

RangeWithin lets callers specify a sub-range within the subspace. If start = nil, the range starts at the beginning of the subspace. If end = nil, the range ends at the end of the subspace. This is used by the iterator to implement LevelDB’s NewIterator(slice *util.Range) — an iterator over a restricted key range.

The subtle end encoding: When end is provided, we use Pack(end) as the upper bound. This is an exclusive upper bound in FDB (same as Python’s range() — GetRange(begin, end) returns keys where begin <= key < end). Since Pack(end) = prefix + 0x00 + end, this correctly excludes the end key itself.

iterator.go — Forward/Backward Navigation

func (it *Iterator) compareKeys(a, b []byte) int {
    return bytes.Compare(a, b)
}

One line, but critical: key comparison is lexicographic byte order, not string collation, not numeric order. "9" > "10" under this comparison. This is identical to LevelDB’s default comparator (BytewiseComparator). If you need a different sort order, you need a sort-preserving encoding — which is why encodeInt64 (used in option-c’s index keys) exists.

func (it *Iterator) Seek(target []byte) {
    for it.idx = 0; it.idx < len(it.kvs); it.idx++ {
        if it.compareKeys(it.Key(), target) >= 0 {
            return
        }
    }
}

Linear scan for Seek. Correct but O(n). A production implementation uses binary search:

it.idx = sort.Search(len(it.kvs), func(i int) bool {
    return bytes.Compare(it.kvs[i].Key, target) >= 0
})

This is O(log n) — important for large result sets. The linear scan is kept here for readability.

snapshot.go — Pinning MVCC Versions

func (s *Snapshot) NewIterator(start, end []byte) *Iterator {
    return newIteratorAt(s.db, s.ns, start, end, s.readVersion)
}

The iterator constructor takes readVersion and passes it to the internal newIteratorAt. Inside newIteratorAt:

func newIteratorAt(fdbDB fdb.Database, ns Subspace, start, end []byte, readVersion int64) *Iterator {
    v, _ := fdbDB.Transact(func(tr fdb.Transaction) (interface{}, error) {
        if readVersion > 0 {
            tr.SetReadVersion(readVersion)
        }
        return tr.GetRange(ns.RangeWithin(start, end), fdb.RangeOptions{}).GetSliceWithError()
    })
    ...
}

If readVersion > 0, we pin the transaction to that version. The range scan returns results as of that exact version. This enables snapshot-consistent iteration — the iterator sees a stable, non-changing view even while concurrent writers are active.

The Transact retry loop and SetReadVersion: Transact retries on conflict. But SetReadVersion sets a fixed read version — the transaction’s read set is pinned. Can we still get a conflict on a read-only operation that reads from a pinned version? No — conflicts are about write-write and read-write conflicts. A transaction that only reads (even via Transact) cannot be retried due to conflict unless it also writes. Our snapshot transactions only read, so Transact will not retry them for conflict reasons. The only reason for retry would be a transaction_too_old error (if readVersion is too old and the data has been GC’d), which surfaces as an error to the caller.

12. Production Considerations

12.1 FDB Transaction Size Limits

FDB enforces hard limits on transactions:

• 10 MB total mutation size (sum of all Set and Clear calls in one transaction)
• 10 MB total read size (sum of all values read)
• 5 seconds maximum transaction duration (from first operation to commit)

For option-a-leveldb, the most likely hit is the iterator: GetSliceWithError() reads the entire range into memory in one transaction. If a namespace contains 50 MB of data and you create an iterator over it, you’ll get a transaction too-large error.

Production solution: Paginated iteration with cursors:

// Instead of reading everything at once:
var cursor fdb.Key = ns.Range().Begin
const pageSize = 10_000
for {
    kvs, _ := db.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(fdb.KeyRange{Begin: cursor, End: ns.Range().End},
            fdb.RangeOptions{Limit: pageSize}).GetSliceWithError()
    })
    if len(kvs) == 0 {
        break
    }
    // process kvs...
    cursor = fdb.Key(append(kvs[len(kvs)-1].Key, 0x00)) // next page starts after last key
}

12.2 FDB Go Binding Concurrency Model

The FDB Go binding uses a single-threaded network loop internally (the FDB C library has one network thread). All transactions are multiplexed over this thread. This means:

• Multiple goroutines can each have their own ReadTransact or Transact calls concurrently — these are correctly serialized by the binding
• fdb.Database is goroutine-safe
• But the network thread is single-threaded: if you saturate it (thousands of concurrent transactions), you’ll see latency increase

For high concurrency, FDB recommends batching multiple operations within one transaction rather than creating many small transactions.

12.3 Key Space Planning

Before deploying, decide on your namespace structure. Once data is in production, renaming a namespace (changing the prefix) requires migrating all data — a potentially days-long background job.

Good practice:

// Use a versioned namespace prefix
db := layer.Open(fdb, []byte("myapp:v1:users"))
// If schema changes require a new encoding:
newDB := layer.Open(fdb, []byte("myapp:v2:users"))
// Migrate in background; dual-write during migration window

12.4 Monitoring and Observability

FDB exposes cluster health via its status API:

fdbcli --exec "status json"

Key metrics to monitor:

• Transactions committed/second — throughput
• Conflicts/second — high values indicate hot keys or poorly structured transactions
• Storage server read/write latency — P99 should be < 10ms
• Data distribution lag — if a shard is being moved, latency spikes

For application-level monitoring, instrument every Transact call with a timer and tag the metric with the operation name.

13. Interview Questions — LevelDB, MVCC, and FDB

Q: What is the difference between ReadTransact and Transact in the FDB Go binding?

ReadTransact opens a read-only transaction: no write buffer, no conflict tracking, no commit phase. It’s cheaper than Transact because it skips the commit round-trip. Use ReadTransact whenever you’re only reading. Transact opens a read-write transaction: it tracks the read key set for conflict detection and requires a commit round-trip to apply writes. For single-key reads like Get, using Transact instead of ReadTransact works but wastes latency and cluster resources.

Q: If FDB’s MVCC window is ~5 seconds, what happens to a snapshot older than 5 seconds?

Reading from that snapshot returns a transaction_too_old error. FDB garbage-collects old MVCC versions after the configured version history window (default ~5 seconds). Any transaction — including read-only snapshot reads — that tries to read a version older than the GC horizon fails with a retriable error. In our Snapshot implementation, this means that a snapshot held open longer than ~5 seconds will start returning errors on the next Get or NewIterator call. Production code must handle this by recreating the snapshot.

Q: LevelDB supports custom comparators. What would you need to change to support a different sort order in this layer?

The sort order is determined by FDB’s key ordering, which is always lexicographic byte order. To support a different sort order (e.g., “integers sort numerically”), you would change the key encoding: encode integer keys using encodeInt64 (sign-bit-flipped big-endian) so that their byte order matches their numeric order. You cannot change FDB’s comparator; you can only change what bytes you write as keys. This is why sort-preserving encoding is the fundamental concept in layer design.

Q: How does FDB’s Transact retry loop interact with side effects?

Transact retries the closure function if the transaction conflicts. If the closure has side effects outside FDB (e.g., incrementing a counter, logging, sending an HTTP request), those side effects will execute multiple times on retry. The FDB convention is: the Transact closure must be idempotent or side-effect-free. For logging, use a separate post-commit hook. For counters, use FDB atomic operations (tr.Add) which are commutative and do not require retry logic.

Q: How does an iterator over a snapshot differ from an iterator over the current database state?

A snapshot iterator is pinned to the read version captured at NewSnapshot() time. Even if concurrent writers modify or delete keys between when the snapshot was taken and when the iterator is created, the iterator sees the state at snapshot time. A current-state iterator uses the latest committed version, so it sees all mutations up to the moment the GetRange call is sent to the cluster. In our implementation, the difference is one line: tr.SetReadVersion(readVersion) in the snapshot path.

Option A — “SQL Layer” over FoundationDB

• What we ship (and don’t)
• Key layout
• Transactionality
• Why this is not a real SQLite layer
• Running

Pattern: relational engine above FDB. Tables → subspaces, rows → KV pairs, ACID inherited from FDB transactions. This is conceptually what Apple’s old (now-archived) fdb-sql-layer did.

What we ship (and don’t)

We ship the storage half of a SQL engine: a catalog, row encoding, table scans, primary-key lookup, and atomic inserts/updates/deletes — all in about 250 lines.

We do not ship a SQL parser. The Go API itself is the query language:

db.CreateTable(sqllayer.TableDef{Name:"users", PK:"id", Columns: ...})
db.Insert("users", sqllayer.Row{"id":1, "name":"Alice", "city":"Paris"})
db.SelectWhere("users", func(r sqllayer.Row) bool { return r["city"] == "Paris" })

A real SQL frontend would compile SELECT ... WHERE city='Paris' into exactly the same backend call.

Key layout

<ns> 0x00 <tableName>                     -> msgpack(TableDef)    [catalog]
<ns> 0x01 <tableName> 0x00 <encodedPK>    -> msgpack(Row)         [data]
<ns> 0x02 <tableName>                     -> uint64 (LE counter)  [rowid seq]

• Catalog lets a fresh Open() reconstruct the in-memory schema cache.
• Data rows live under a per-table subspace, so GetRange on <ns> 0x01 <tableName> 0x00 is a full table scan in key order.
• Seq is used only when the table has no declared PK (we mint a rowid).

PK values are encoded with the same sign-flipping big-endian trick used in option-c-record-layer: integer rows sort numerically, strings sort lexicographically. This is what makes table scans produce ordered output “for free”.

Transactionality

Insert does the seq read+write and the row write in the same fdb.Transact, so two concurrent inserts can never collide on the same rowid: FDB will retry one of them. DropTable clears catalog, all rows, and the seq counter atomically — no orphan rows can be observed after a drop.

Why this is not a real SQLite layer

A real SQLite vtab would let you write SELECT name FROM users WHERE city='Paris' in standard SQL and have SQLite’s planner call back into FDB. Building that requires:

• Implementing the sqlite3_module C ABI (via CGO with mattn/go-sqlite3, or using modernc.org/sqlite’s yet-undocumented vtab hooks).
• Mapping SQLite’s xBestIndex / xFilter / xColumn callbacks onto FDB range scans.
• Round-tripping SQLite’s rowids to our PK encoding.

That’s an order of magnitude more code (~1.5k lines) and adds little to your understanding of FDB itself; the interesting parts — how does the relational model fit on top of an ordered KV store? — are all here.

Running

cd option-a-sqlite
go mod tidy
go run ./demo -cluster ../fdb.cluster

Expected output:

SELECT * FROM users;
  map[city:Paris id:1 name:Alice]
  map[city:Tokyo id:2 name:Bob]
  map[city:Paris id:3 name:Carol]

SELECT * FROM users WHERE city='Paris';
  map[city:Paris id:1 name:Alice]
  map[city:Paris id:3 name:Carol]

After UPDATE users SET city='Tokyo' WHERE id=1;
  map[city:Tokyo id:1 name:Alice]
  map[city:Tokyo id:2 name:Bob]
  map[city:Paris id:3 name:Carol]

After DELETE FROM users WHERE id=3;
  map[city:Tokyo id:1 name:Alice]
  map[city:Tokyo id:2 name:Bob]

Hitchhiker’s Guide — Option A: SQL Engine over FoundationDB

• Table of Contents
• 1. What a Relational Database Actually Stores
• 2. The Catalog — A Database About Databases
• 3. Row Encoding — Turning a Map into Bytes
• 4. Primary Keys and Sort Order
• 5. Full Table Scan — The Honest Query
• 6. INSERT as Rowid Allocation + Write
• 7. UPDATE = Overwrite (Why it Works)
• 8. DELETE = Clear (Why it Works)
• 9. Key Layout In Detail
• 10. What We Didn’t Build (and Why Those Parts Are Hard)
• 11. Real-World Analogues
  • SQLite
  • MySQL InnoDB
  • PostgreSQL
  • CockroachDB / TiKV / YugabyteDB
• 12. Exercises
• 13. Source Code Deep Dive — sqllayer/db.go
  • The DB Type
  • Key Helpers
  • The Sequence Counter Pattern
  • The encodePK Function
• 14. Production Architecture — What a Real SQL-on-FDB Looks Like
  • Component 1: SQL Parser
  • Component 2: Query Planner
  • Component 3: Encoding Layer (what we built)
  • Component 4: FDB Transaction Management
  • Component 5: Network Protocol
• 15. Interview Questions — Relational Databases and FDB

The question this answers: “What does a relational database actually store? If you strip away the SQL parser and the query planner, what is left?”

The deeper question: “How do tables, rows, primary keys, and catalogs map onto an ordered key–value store?”

Table of Contents

1. What a Relational Database Actually Stores
2. The Catalog — A Database About Databases
3. Row Encoding — Turning a Map into Bytes
4. Primary Keys and Sort Order
5. Full Table Scan — The Honest Query
6. INSERT as a Rowid Allocation + Write
7. UPDATE = Overwrite (Why it Works)
8. DELETE = Clear (Why it Works)
9. Key Layout In Detail
10. What We Didn’t Build (and Why Those Parts Are Hard)
11. Real-World Analogues: SQLite, MySQL, PostgreSQL
12. Exercises

1. What a Relational Database Actually Stores

A relational database is, at its core, a system for:

1. Storing rows (collections of typed values) under a primary key.
2. Keeping a catalog (metadata about what tables exist and their schemas).
3. Maintaining indexes (secondary data structures for fast lookup by non-primary-key fields).
4. Executing queries (finding rows matching a predicate).

Strip away SQL parsing, the query planner, authentication, and network protocols, and you have a structured key–value store with a catalog.

Our layer implements steps 1 and 2. Step 3 is in option-c-record-layer. Step 4 is a SelectWhere(pred) that does a full scan — honest but slow, which makes the need for indexes obvious.

2. The Catalog — A Database About Databases

Every real database has a catalog: a table of tables. PostgreSQL stores it in pg_catalog.pg_class. MySQL uses information_schema.TABLES. SQLite writes the catalog to the first page of the database file as sqlite_master.

Our catalog lives in FDB under a dedicated subspace:

<ns> 0x00 <tableName>  →  msgpack(TableDef)

TableDef contains the table name, column definitions (name + type), and the declared primary-key column name (if any). When Open() is called, we scan this subspace and reconstruct the in-memory schemas map:

func (d *DB) loadCatalog() error {
    // range-scan the catalog subspace
    kvs, _ := d.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(d.catalogRange(), fdb.RangeOptions{}).GetSliceWithError()
    })
    for _, kv := range kvs {
        var def TableDef
        msgpack.Unmarshal(kv.Value, &def)
        d.schemas[def.Name] = def
    }
}

This means the catalog is self-describing — a fresh process connecting to an existing FDB namespace automatically discovers all tables. You don’t need to pass schemas at startup. This mirrors how SQLite opens a database file and reads its catalog from sqlite_master before accepting any queries.

CreateTable is atomic: the catalog write and any initial setup happen inside one FDB transaction. Two concurrent CreateTable("users") calls will either both succeed (if they have identical schemas) or one will fail with a conflict. There is no possibility of a half-created table.

3. Row Encoding — Turning a Map into Bytes

Our Row type is map[string]any. We serialize it with msgpack, which produces a compact binary representation:

{"name": "Alice", "city": "Paris", "id": 1}
→  83 a4 6e 61 6d 65 a5 41 6c 69 63 65 a4 63 69 74 79 a5 50 61 72 69 73 a2 69 64 01
   (msgpack map: 3 entries, name="Alice", city="Paris", id=1)

The row bytes become the value in FDB. The key is derived from the primary key value (see below).

Why msgpack, not JSON or Protobuf?

• JSON is human-readable but wastes space (quoted keys, text numbers).
• Protobuf is compact but requires a schema file and code generation.
• msgpack is compact, self-describing (no external schema), and has first- class Go support.

For a teaching implementation, msgpack hits the right trade-off. For production, Protobuf or Cap’n Proto would be better choices, since they support schema evolution (adding fields without breaking old readers) and are more efficient for fixed schemas.

4. Primary Keys and Sort Order

A primary key must be unique and determines the storage order of rows.

In FDB, “storage order” = “key byte order”. So we need to encode primary key values into bytes in a way that preserves the intended sort order.

String PKs: store as raw UTF-8 bytes. UTF-8’s byte ordering preserves lexicographic order for ASCII characters. Enough for a demo; a production system would use Unicode collation.

Integer PKs: this is where it gets interesting.

Naive approach: encode as decimal string. Problem: "9" > "30" in byte comparison. Rows would sort as: 1, 10, 100, 2, 20, 3… — wrong.

Better: big-endian 64-bit integer. Problem: negative numbers have the high bit set, so -1 encodes as 0xFF... which sorts after all positive numbers.

Our solution (and the industry standard):

func encodeInt64(x int64) []byte {
    b := make([]byte, 8)
    // Flip the sign bit. This turns the two's-complement representation into
    // an unsigned representation that sorts correctly:
    //  0 → 0x8000_0000_0000_0000 (sorts between negatives and positives)
    //  1 → 0x8000_0000_0000_0001
    // -1 → 0x7FFF_FFFF_FFFF_FFFF (sorts before 0)
    // -2 → 0x7FFF_FFFF_FFFF_FFFE (sorts before -1)
    binary.BigEndian.PutUint64(b, uint64(x)^(1<<63))
    return b
}

After this encoding, integer byte strings sort in the same order as their numeric values. A GetRange on the rows subspace returns rows in ascending PK order, for free, by FDB’s native byte-order scan.

This is the same trick used by:

• Apache HBase (for row key ordering)
• Google Cloud Bigtable
• FoundationDB’s official Tuple layer (for negative integers)
• Apache Cassandra’s blob serialization

5. Full Table Scan — The Honest Query

SelectAll and SelectWhere both scan the entire rows subspace:

func (d *DB) SelectWhere(table string, pred func(Row) bool) ([]Row, error) {
    v, _ := d.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(d.tableRange(table), fdb.RangeOptions{}).GetSliceWithError()
    })
    for _, kv := range v.([]fdb.KeyValue) {
        var row Row
        msgpack.Unmarshal(kv.Value, &row)
        if pred == nil || pred(row) {
            out = append(out, row)
        }
    }
}

The predicate is applied in Go, after fetching all rows from FDB. This is a full table scan — the same operation as SELECT * FROM users WHERE city='Paris' without any index.

Why not push the predicate to FDB?

FDB has no server-side filtering. Its API is: “give me this key range.” All filtering must happen on the client. This is a deliberate design choice by FDB: the cluster is a dumb-fast KV store; smarts live in the layer.

Real SQL databases push predicates “down” to the storage engine to avoid fetching unnecessary rows over the network. In our setup, “the network” is the connection to FDB’s storage servers — for large tables, a full scan incurs real latency. This is the motivation for secondary indexes (option-c).

What a real SQL engine does instead:

A query planner looks at the predicate city='Paris' and checks whether there is an index on city. If yes, it rewrites the scan as:

scan index(city = 'Paris') → [pk1, pk2, pk3]
fetch rows by pk: Get(pk1), Get(pk2), Get(pk3)

This is exactly what LookupByIndex does in option-c.

6. INSERT as Rowid Allocation + Write

When a table has no declared PK, we allocate a monotonic rowid:

_, err := d.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
    curBytes, _ := tr.Get(d.seqKey(table)).Get()
    var cur uint64
    if len(curBytes) == 8 {
        cur = binary.LittleEndian.Uint64(curBytes)
    }
    next := cur + 1
    // Write the new counter value
    binary.LittleEndian.PutUint64(buf, next)
    tr.Set(d.seqKey(table), buf)
    // Write the row
    tr.Set(d.rowKey(table, encodeInt64(int64(next))), encoded)
    return nil, nil
})

The counter read + increment + row write happen in one transaction. Two concurrent inserts cannot get the same rowid — FDB’s conflict detection will cause one to retry with a fresh counter read.

This is identical to how SQLite allocates its rowid (a monotonic integer stored in the B-tree page header) and how PostgreSQL allocates OIDs and ctids. The key insight: the counter is a row, and updating it is a transaction.

7. UPDATE = Overwrite (Why it Works)

LevelDB and FDB share a property: Set(key, newValue) on an existing key simply replaces it. There is no “update in place” at the storage level.

Our “UPDATE” is just another Insert call with the same PK:

db.Insert("users", Row{"id": 1, "name": "Alice", "city": "Tokyo"})
// This calls tr.Set(rowKey(table, encode(1)), msgpack(new_row))
// The old row bytes under the same key are replaced.

FDB does not copy the old value to a TOAST or undo segment — it just writes the new value. The old version is kept in FDB’s MVCC history for ~5 seconds (for in-flight read transactions) and then garbage-collected.

What changes when you have indexes:

Once you have secondary indexes (option-c), an update is no longer just a single key write. You must:

1. Read the old row (to find out what old index entries to remove)
2. Clear the old index entries
3. Write the new row
4. Write the new index entries

All in one transaction. This is why option-c’s PutRecord is more complex than our Insert.

8. DELETE = Clear (Why it Works)

func (d *DB) Delete(table string, pk any) error {
    encodedPK, _ := encodePK(pk)
    _, err := d.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        tr.Clear(d.rowKey(table, encodedPK))
        return nil, nil
    })
    return err
}

Clear adds a tombstone to the transaction buffer. After commit, the key disappears from FDB as if it was never written. Future reads return nil.

No cascades: unlike SQL with ON DELETE CASCADE, our engine has no foreign key enforcement. This is an intentional simplification. Real SQL engines resolve cascades by finding related rows (via indexes on the FK column) and deleting them in the same transaction.

What about DropTable?

func (d *DB) DropTable(name string) error {
    _, err := d.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        tr.Clear(d.catalogKey(name))         // remove from catalog
        tr.ClearRange(d.tableRange(name))    // delete all rows
        tr.Clear(d.seqKey(name))             // reset rowid counter
        return nil, nil
    })
}

One transaction that deletes the catalog entry, all data rows, and the sequence counter. Atomically. A reader that has an open transaction might still see the table (MVCC), but any new transaction started after commit sees the table as gone.

9. Key Layout In Detail

Let’s trace exactly what happens when you call:

db.CreateTable(TableDef{Name: "users", PK: "id", Columns: [...]})
db.Insert("users", Row{"id": 1, "name": "Alice", "city": "Paris"})
db.Insert("users", Row{"id": 2, "name": "Bob",   "city": "Tokyo"})

With namespace "demo-sql":

Catalog key:
  "demo-sql" + 0x00 + "users"
  = 64 65 6d 6f 2d 73 71 6c 00 75 73 65 72 73
  → value: msgpack({Name:"users", PK:"id", Columns:[...]})

Row key for id=1:
  "demo-sql" + 0x01 + "users" + 0x00 + encodeInt64(1)
  = 64 65 6d 6f 2d 73 71 6c 01 75 73 65 72 73 00 80 00 00 00 00 00 00 01
                                               ↑ encoded 1 (sign bit flipped)
  → value: msgpack({id:1, name:"Alice", city:"Paris"})

Row key for id=2:
  "demo-sql" + 0x01 + "users" + 0x00 + 80 00 00 00 00 00 00 02
  → value: msgpack({id:2, name:"Bob", city:"Tokyo"})

Table scan range:
  begin: "demo-sql" + 0x01 + "users" + 0x00
  end:   "demo-sql" + 0x01 + "users" + 0x01
  → returns id=1 row, then id=2 row (in ascending id order)

The 0x00 separator between tag, table name, and PK ensures:

• Table “user” doesn’t collide with table “users”
• Row key doesn’t collide with catalog key (different tag byte: 0x01 vs 0x00)
• Rows for table “users” don’t collide with rows for table “orders” (different table name segment)

10. What We Didn’t Build (and Why Those Parts Are Hard)

Secondary indexes: We have SelectWhere but it does a full scan. Adding a secondary index (e.g., “find all users where city=‘Paris’”) requires maintaining an extra key for every row that maps (city value, pk) → "". The hard part is keeping the index consistent when rows change — see option-c.

SQL parsing: A SQL parser turns SELECT name FROM users WHERE city='Paris' into an AST. This is ~1000 lines of code for a minimal SQL grammar. We skip it because it adds no FDB-specific insight.

Query planner: Given multiple indexes, which one should we use? This is a graph problem (estimating cost of each query plan) that modern databases spend enormous effort on. The basic version is simple (pick the most selective index), but getting it right for all cases is a career.

JOIN: Joining two tables in SQL is conceptually a nested loop (for each row in table A, find matching rows in table B). The hard part is making this fast when both tables are large. Most JOIN algorithms (hash join, sort-merge join) still reduce to: “scan one table, look up matching rows from the other.”

MVCC for uncommitted readers: We provide MVCC through FDB’s read versions, but we don’t expose BEGIN TRANSACTION / COMMIT / ROLLBACK at the SQL level. Adding this requires keeping an open FDB transaction for the duration of the user transaction — which is possible but requires careful lifecycle management.

11. Real-World Analogues

SQLite

SQLite stores its catalog in sqlite_master, a special table on page 1:

CREATE TABLE sqlite_master (
    type TEXT,    -- "table", "index", "view", "trigger"
    name TEXT,
    tbl_name TEXT,
    rootpage INTEGER,  -- page number of the B-tree root for this table
    sql TEXT           -- the original CREATE statement
);

Every table’s rows are stored in a B-tree rooted at rootpage. The B-tree keys are the rowid values (64-bit integers, always). The row data (all columns) is packed into the B-tree leaf value using SQLite’s own compact encoding.

Our rowKey(table, encodedPK) is the FDB equivalent of the B-tree rowid. Our msgpack row serialization is the FDB equivalent of SQLite’s row record format.

MySQL InnoDB

InnoDB uses a clustered index — the primary key IS the B-tree. All columns are stored in the PK leaf node. Secondary indexes store the PK value (not the physical row location) as their “pointer”:

PRIMARY KEY index: pk → all_columns
SECONDARY INDEX on city: city → pk

To look up a row by city:

1. Scan secondary index for city = 'Paris' → get pk values
2. Look up each pk in the primary index → get the full row

This two-step lookup is called a “key lookup” in MySQL’s EXPLAIN output. Our option-c-record-layer does exactly this.

PostgreSQL

PostgreSQL uses “heap files” — rows are stored in pages in insertion order, not sorted by PK. The PK is just another index (a B-tree) that maps pk → ctid (page number + slot within page). Secondary indexes also map field_value → ctid.

This is subtly different from InnoDB: in PostgreSQL, all indexes point to physical page locations, not to the PK. This makes index maintenance easier (an update to a non-indexed column doesn’t touch any index), but makes VACUUM (garbage collection of old row versions) more complex.

CockroachDB / TiKV / YugabyteDB

These “NewSQL” databases are essentially what we’re building here: a SQL query engine on top of a distributed, ordered, transactional KV store. Their storage architecture is:

SQL layer (parsing, planning, execution)
    ↓
Encoding layer (rows → keys, indexes → keys)
    ↓
Distributed KV layer (RocksDB + Raft replication per shard)

CockroachDB’s row encoding is:

/<table_id>/<index_id>/<col1_value>/<col2_value>/...  →  rest_of_columns

This is the same pattern as our <ns>/<tag>/<table_name>/<pk> — just with more type information and a smarter Tuple encoding.

12. Exercises

Exercise 1 — Add a Secondary Index

Modify Insert to also write index entries:

ns + 0x02 + tableName + 0x00 + columnName + 0x00 + encodedValue + 0x00 + pk → ""

Modify Delete to also clear those entries (requires reading the row first). Add SelectByIndex(table, column, value) ([]Row, error).

Exercise 2 — Foreign Keys

Add a References field to Column:

type Column struct {
    Name       string
    Type       ColumnType
    References *ForeignKey  // nil if no FK
}

In Insert, check that the referenced PK exists (a Get inside the same transaction). In Delete, check that no child rows reference this PK (a GetRange on the FK index).

Exercise 3 — Transactions for Callers

Add a Begin() *Tx method that returns a transaction object. Tx should buffer operations and commit on tx.Commit() — all inside one FDB db.Transact() call.

Exercise 4 — DISTINCT and COUNT

Implement SelectDistinct(table, column) and Count(table) using a single FDB GetRange and in-process aggregation. This is your first aggregation operator — the same one query engines implement with hash tables or sort passes.

Exercise 5 — Schema Evolution

Add a version number to TableDef. When Open() loads a table with version < current, run a migration function that adds default values for new columns.

13. Source Code Deep Dive — sqllayer/db.go

The entire engine is in sqllayer/db.go. Let’s trace the critical paths.

The DB Type

type DB struct {
    fdb     fdb.Database
    ns      []byte
    schemas map[string]TableDef
    mu      sync.RWMutex
}

schemas is an in-memory cache of the catalog. It is loaded at Open() time from FDB and updated by CreateTable/DropTable. The mutex protects concurrent access to schemas.

Design question: Why cache schemas in memory at all? Because every query needs the schema to know column types (for encoding/decoding) and the PK column name. Fetching the schema from FDB on every query would add a round-trip. The trade-off: the in-memory cache can be stale if another process creates/drops a table. Production systems solve this with schema versioning: every DDL operation increments a global version; processes re-sync when they detect a version mismatch.

Key Helpers

func (d *DB) catalogKey(table string) fdb.Key {
    return fdb.Key(append(append([]byte{}, d.ns...), append([]byte{0x00}, []byte(table)...)...))
}

func (d *DB) rowKey(table string, encodedPK []byte) fdb.Key {
    prefix := append(append([]byte{}, d.ns...), 0x01)
    prefix = append(prefix, []byte(table)...)
    prefix = append(prefix, 0x00)
    return fdb.Key(append(prefix, encodedPK...))
}

Tag bytes distinguish subspaces:

• 0x00 → catalog entries
• 0x01 → row data
• 0x02 → sequence (rowid counter)

This tag-byte approach is simpler than the full Tuple encoding but achieves the same goal: prevent catalog keys from colliding with row keys even if a table is named “users” and another subspace prefix also uses “users”.

The Sequence Counter Pattern

func (d *DB) nextRowid(tr fdb.Transaction, table string) (int64, error) {
    key := d.seqKey(table)
    cur, _ := tr.Get(key).Get()
    var n int64 = 1
    if len(cur) == 8 {
        n = int64(binary.LittleEndian.Uint64(cur)) + 1
    }
    b := make([]byte, 8)
    binary.LittleEndian.PutUint64(b, uint64(n))
    tr.Set(key, b)
    return n, nil
}

This is a read-modify-write within one transaction. The caller passes tr (an active transaction). The Get reads the current counter value. The Set increments it. Both are in the same transaction as the row write.

If two concurrent inserts both read the same counter value, they’ll both try to write n+1 — but FDB’s conflict detection will catch this: both transactions read the same key, and both tried to write it. One will conflict and retry, which re-reads the counter and gets a fresh value. The retry is invisible to the caller.

Why not use FDB’s atomic add?

tr.Add(key, encodeInt64(1))  // atomic increment, no conflict

tr.Add is a commutative atomic operation — it doesn’t cause read-write conflicts. But it returns no value — you can’t know what rowid was assigned. If you need to use the rowid in the same transaction (to build the row key), you need the read-modify-write pattern. If the rowid is just an opaque row identifier that callers don’t use, tr.Add plus a versionstamp (to create a unique, ordered key without knowing the exact value) is a better approach.

The encodePK Function

func encodePK(v any) ([]byte, error) {
    switch x := v.(type) {
    case int:
        return encodeInt64(int64(x)), nil
    case int64:
        return encodeInt64(x), nil
    case string:
        return []byte(x), nil
    case []byte:
        return x, nil
    default:
        return nil, fmt.Errorf("unsupported PK type %T", v)
    }
}

String PKs are stored as raw UTF-8 bytes. This sorts correctly for ASCII but not for Unicode (e.g., “é” sorts after “z” in byte order but before “z” in French locale). For a production system, use the ICU library for locale-aware collation, or restrict PKs to ASCII.

Integer PKs use encodeInt64: big-endian with sign-bit flipped. Read Chapter 3 of this guide for the derivation.

What about composite PKs? (user_id, order_id) as a PK means the row key is encode(user_id) + 0x00 + encode(order_id). The separator between components is 0x00 (the Tuple encoding style). This naturally makes rows for the same user adjacent in key order — a range scan GetRange(encode(user_id) + 0x00, encode(user_id) + 0x01) returns all orders for that user. Our implementation only supports single-column PKs, but the extension is straightforward.

14. Production Architecture — What a Real SQL-on-FDB Looks Like

The five components of a production SQL-on-FDB system:

Component 1: SQL Parser

Takes SQL text, produces an AST. For SELECT name FROM users WHERE city='Paris':

SelectStatement {
    Columns: ["name"],
    Table:   "users",
    Where:   EqualExpr{Column: "city", Value: "Paris"},
}

Libraries: pingcap/tidb/parser (Go, production-grade), xwb1989/sqlparser (Go, simpler), or write your own using gocc or pigeon.

Component 2: Query Planner

Takes the AST, looks at available indexes, estimates row counts, and picks the cheapest plan:

Without index on city:
  plan: TableScan("users") → Filter(city='Paris')
  cost: O(n_rows) reads

With index on city:
  plan: IndexScan(city='Paris') → PKLookup("users", [pk1, pk2, ...])
  cost: O(index_entries) + O(result_rows) reads

The planner is the hardest part of a query engine. Modern planners use statistics (histograms of column value distributions) stored in a catalog table to estimate n_rows for each plan.

Component 3: Encoding Layer (what we built)

Rows → keys in FDB. Indexes → keys in FDB. All with sort-preserving encodings.

Component 4: FDB Transaction Management

SQL has BEGIN / COMMIT / ROLLBACK. FDB has Transact. Bridging these requires holding an FDB Transaction object open for the duration of the SQL transaction. The FDB 5-second limit means long-running SQL transactions must be broken up or use FDB’s “grv” caching.

Component 5: Network Protocol

Postgres wire protocol (libpq-compatible), MySQL wire protocol, or a custom protocol. This is what lets standard clients (psql, JDBC, SQLAlchemy) connect. CockroachDB implements the Postgres wire protocol on top of their FDB-like storage.

15. Interview Questions — Relational Databases and FDB

Q: What is a clustered index, and how does option-a-sqlite implement it?

A clustered index is an index where the data rows are stored at the index leaf nodes, sorted by the index key. In option-a-sqlite, the FDB key for each row IS the primary key, and the row data is the value at that key. Range scanning by primary key returns rows in PK order and fetches the data in the same operation. This is a clustered index: the data is organized by PK. InnoDB uses the same design (the primary key B-tree is the table).

Q: What happens when two concurrent transactions try to insert a row with the same primary key?

Both transactions read the same row key (which doesn’t exist yet — they get nil). Both write a new value to that key. FDB’s conflict detection: one transaction commits; the other conflicts (it wrote to a key that was also written by the committed transaction). The conflicting transaction retries from the beginning. On retry, it reads the row key again — this time it exists. Depending on application semantics, the retry can: (a) return a duplicate key error, (b) return the existing row (SELECT or INSERT/IGNORE), or (c) replace it (INSERT OR REPLACE).

Q: Why does PostgreSQL store rows in heap order rather than clustered by PK?

PostgreSQL’s original design assumed that MVCC updates would frequently need to move rows (updating a row creates a new row version at a new physical location; the old version stays until VACUUM). If rows were clustered by PK, a moved row would need to update all indexes to point to the new physical location. By using heap files with stable physical locations (ctid), updates only create a new heap entry and update the visibility chain — indexes don’t need to change. The trade-off: PK range scans in PostgreSQL require index traversal + random I/O into the heap, while InnoDB’s clustered index gives sequential I/O for PK range scans.

Q: How does your row encoding handle schema migrations — adding a new column to a table with existing rows?

With msgpack, adding a new column to the schema is safe for new rows (they include the new column). Old rows, when read back, will simply not have the column in the map. Application code should handle missing fields with defaults. This is “forward compatibility”: old data is compatible with new code. The hard case is “backward compatibility”: if you need all rows to have the new column (for a NOT NULL constraint), you must run a background migration job that reads every old row, adds the default value, and writes it back — one paginated batch per transaction.

This is how every production database handles ALTER TABLE ADD COLUMN.

Option B — LevelDB on top of FoundationDB

• How LevelDB sees the world
• Key layout
• Why this is interesting
• Running
• What this implementation skips

Pattern: existing storage engine, FDB as the disk. We give the unmodified goleveldb library an FDB-backed implementation of its storage.Storage interface. LevelDB still does its LSM thing (memtables, SSTables, compaction, MANIFEST), but every byte ends up in FDB key ranges instead of on local disk.

This is the mirror image of option-a: the storage engine sits below FDB rather than above it.

How LevelDB sees the world

goleveldb accesses persistence exclusively through a small interface:

type Storage interface {
    Lock() (Locker, error)
    Log(str string)
    SetMeta(FileDesc) error
    GetMeta() (FileDesc, error)
    List(FileType) ([]FileDesc, error)
    Open(FileDesc) (Reader, error)
    Create(FileDesc) (Writer, error)
    Remove(FileDesc) error
    Rename(old, new FileDesc) error
    Close() error
}

Each FileDesc is {Type, Num} — e.g. {TypeTable, 42} for SST #42 or {TypeManifest, 7} for the 7th manifest. Filenames are an implementation detail; LevelDB never looks at strings.

Our fdbstorage package implements that interface against FDB. The whole file is ~250 lines.

Key layout

<ns> 0x01 <ftype:1B> <num:int64 BE>                 -> uint64 BE file size
<ns> 0x02 <ftype:1B> <num:int64 BE> <chunk:uint32 BE> -> 64 KiB chunk
<ns> 0x03                                             -> current MANIFEST {ftype,num}
<ns> 0x04                                             -> lock marker

• Files are split into 64 KiB chunks so we stay well under FDB’s 100 KiB-per- value soft limit and the 10 MB-per-transaction hard limit.
• Create returns a Writer that buffers in memory and flushes on Sync / Close. We split the flush across multiple transactions (100 chunks each ≈ 6 MiB) to safely handle files larger than 10 MB.
• Rename is implemented as copy-then-clear inside one transaction. LevelDB only renames small files (temp → real on flush completion), so the inefficiency doesn’t matter.
• SetMeta writes the manifest pointer atomically. Because FDB transactions are serializable, two concurrent flushes can’t observe a half-rotated manifest.

Why this is interesting

You get a real LevelDB instance — with bloom filters, compaction, snapshots, the works — whose durability story is “whatever FDB’s durability story is.” That means:

• Geo-replication and read scaling come for free from the FDB cluster.
• Backups are FDB backups.
• The local node has no on-disk state at all; it can crash and restart against a different FDB coordinator without losing anything.

The cost is latency: every SST read is at least one FDB round-trip, every flush is many. This isn’t a production architecture; it’s a teaching artifact that proves how cleanly the layers separate.

Running

cd option-b-leveldb
go mod tidy
go run ./demo -cluster ../fdb.cluster

Expected output (the second session re-opens and reads the persisted data):

First session: wrote 3 keys, then closed.
Reopening LevelDB on the same FDB namespace...

  apple -> red
  banana -> yellow
  cherry -> red

Iterating the whole DB:
  apple -> red
  banana -> yellow
  cherry -> red

What this implementation skips

• Locker isn’t multi-process safe across long-lived processes — if a holder crashes the lock key stays set. A production version would attach the lock to a client UUID and TTL it via FDB watches.
• Reader loads the whole file into memory. LevelDB SSTs are bounded (default 2 MB), so this is fine for a demo but not for huge tables.
• No caching layer. Every Open is a fresh FDB scan. A real impl would cache hot SSTs.

Read fdbstorage/storage.go — the whole thing is one file deliberately, so you can follow the data flow end to end.

Hitchhiker’s Guide — Option B: LevelDB on FDB Storage

• Table of Contents
• 1. The Storage Abstraction: Why LevelDB Has a Plugin Point
• 2. What LevelDB Actually Writes to Disk
• 3. The storage.Storage Interface — Dissected
  • 3.1 There Are No Filesystem Calls in the Data Path
• 4. How We Map LevelDB Files to FDB Keys
  • 4.1 FileDesc: (Type, Num) — Not a Filename
  • 4.2 Namespace = Your Database Name
  • 4.3 Your User Data is NOT an FDB Key
  • 4.4 How db.Get("bob") Actually Works — The Two-Level Lookup
  • 4.5 Many Writes, One Flush — How Records Accumulate
• 5. Key Layout Deep Dive: From Function Call to FDB Bytes
  • 5.1 The Four Tag Bytes
  • 5.2 Anatomy of Every Key Type
  • 5.3 Journey: db.Put("hello", "world") → FDB keys
  • 5.4 Journey: stor.Open(FileDesc{TypeJournal, 2})
  • 5.5 Journey: stor.List(TypeAll)
  • 5.6 Journey: stor.SetMeta(FileDesc{TypeManifest, 1})
  • 5.7 What the FDB keyspace looks like for a small database
  • 5.8 Verifying with a test
  • 5.9 Live Simulation — Annotated Output
    • Phase 1 — Write
    • Phase 2 — FDB Keyspace Dump
    • Phase 3 — Read Back
    • Phase 4 — GetMeta + List
    • Summary table
• 6. Chunking: Overcoming the 100 KiB Value Limit
• 7. Atomic Rename — Durability’s Secret Weapon
• 8. The Writer: Batching Chunks into Transactions
• 9. Why the WAL is Redundant With FDB
• 10. The Blob Layer Pattern
• 11. Real-World Analogues
  • RocksDB Remote Compaction (Project Titan, Ripple)
  • TiKV on Disaggregated Storage
  • Pebble (CockroachDB)
  • The Pattern’s Universality
• 12. Exercises
• 13. Source Code Deep Dive — fdbstorage/storage.go
  • The Storage Struct
  • Key Encoding Helpers
  • The dataRange Helper
  • The List() Implementation
  • The Lock Implementation
• 14. Production Considerations
  • 14.1 Transaction Size for Large SSTables
  • 14.2 Read Performance for Large SSTables
  • 14.3 Write Amplification
  • 14.4 Monitoring
• 15. Interview Questions — Storage Abstractions and LSM Trees
• 16. Bugs Encountered and Lessons Learned
  • Bug 1 — storage.ErrNotExist Does Not Exist
  • Bug 2 — GetMeta Returning a Custom Error on Empty Namespace
  • Bug 3 — List() Always Returned Empty
  • Bug 4 — FDB-Persisted Lock Survives Process Crashes
  • Bug 5 — os.IsNotExist Ignores Custom .Is() Methods (Go 1.16+)
  • Bug 6 — Stale FDB State Across Runs

The question this answers: “Can I run a real LSM-tree storage engine — the actual LevelDB binary with all its compaction logic — with its files stored in FoundationDB instead of a local disk?”

The deeper question: “What is the storage.Storage interface, why does it exist, and what does it tell us about how databases handle file I/O?”

Table of Contents

1. The Storage Abstraction: Why LevelDB Has a Plugin Point
2. What LevelDB Actually Writes to Disk
3. The storage.Storage Interface — Dissected
4. How We Map LevelDB Files to FDB Keys
5. Key Layout Deep Dive: From Function Call to FDB Bytes
6. Chunking: Overcoming the 100 KiB Value Limit
7. Atomic Rename — Durability’s Secret Weapon
8. The Writer: Batching Chunks into Transactions
9. Why the WAL is Redundant With FDB
10. The Blob Layer Pattern
11. Real-World Analogues: RocksDB on Cloud Storage
12. Exercises
13. Source Code Deep Dive — fdbstorage/storage.go
14. Production Considerations
15. Interview Questions — Storage Abstractions and LSM Trees
16. Bugs Encountered and Lessons Learned

1. The Storage Abstraction: Why LevelDB Has a Plugin Point

LevelDB’s storage.Storage interface exists because the original LevelDB authors (Jeff Dean and Sanjay Ghemawat) designed it for portability. Not every environment has a POSIX filesystem. Google has internal systems where storage might be Bigtable, Colossus, or a custom log-structured store.

The interface says: “if you can implement these 8 methods, LevelDB will run on your storage.” The application code (goleveldb) doesn’t know whether it’s writing to ext4, NTFS, GCS, or FDB — it just calls the interface.

type Storage interface {
    Lock() (util.Releaser, error)
    Log(m storage.FileDesc) (storage.Writer, error)
    Open(fd storage.FileDesc) (storage.Reader, error)
    Create(fd storage.FileDesc) (storage.Writer, error)
    Remove(fd storage.FileDesc) error
    Rename(oldfd, newfd storage.FileDesc) error
    GetMeta() (storage.FileDesc, error)
    SetMeta(fd storage.FileDesc) error
    List() ([]storage.FileDesc, error)
    Close() error
}

Our fdbstorage.Storage implements all of these, storing files as FDB key-value pairs. LevelDB itself (in the syndtr/goleveldb package) calls these methods. It has no idea the “files” are actually chunks in a distributed database.

2. What LevelDB Actually Writes to Disk

To understand what we need to store, let’s look at what LevelDB writes:

File types:
  TypeJournal  (.log)  — Write-Ahead Log: records every write before the
                         MemTable is flushed. Used to recover unflushed writes
                         after a crash.
  TypeManifest (.MANIFEST) — Lists which SSTables are "live" (not yet
                         garbage-collected). Updated at each compaction.
  TypeTable    (.ldb / .sst) — Sorted String Tables. Immutable, sorted KV
                         data files produced by compaction.
  TypeCurrent  (CURRENT) — A single file containing the name of the latest
                         MANIFEST file.
  TypeTemp     (.tmp)   — Temporary files used during compaction.
  TypeLock     (LOCK)   — A file held open to prevent two processes from
                         opening the same database simultaneously.

File descriptor:
type FileDesc struct {
    Type FileType  // TypeJournal, TypeManifest, etc.
    Num  int64     // unique file number (monotonically increasing)
}

A LevelDB database directory looks like:

000003.log       ← journal (WAL)
000004.ldb       ← SSTable level 0
000005.ldb       ← SSTable level 0
MANIFEST-000002  ← current manifest
CURRENT          ← "MANIFEST-000002\n"
LOCK             ← lockfile

When compaction happens:

1. LevelDB picks some SSTables, merges and sorts them into a new SSTable.
2. It writes the new SSTable as a .tmp file (via Create(TypeTemp, ...))
3. It renames the .tmp to the final .ldb name (via Rename)
4. It updates the MANIFEST to list the new SSTable and de-list the old ones.
5. It removes the old SSTables (via Remove).

This is the temp-then-rename durability pattern: create a new file atomically, then rename it into place. POSIX rename is atomic — the old name or the new name is visible, never a partial file. Our FDB implementation must replicate this property.

3. The storage.Storage Interface — Dissected

Let’s look at each method and what it does:

Lock() (util.Releaser, error) Prevents two processes from opening the same database simultaneously. We implement this by writing a “lock” key to FDB. The Releaser clears it.

Create(fd FileDesc) (Writer, error) and Open(fd FileDesc) (Reader, error) Create starts a new file (for writing). Open opens an existing file (for reading). In FDB terms: Create returns a writer that buffers bytes; Open reads all chunks for the file into memory and returns a bytes.Reader.

Remove(fd FileDesc) error Deletes a file. In FDB: ClearRange over all chunk keys for this file.

Rename(oldfd, newfd FileDesc) error Renames a file atomically. In FDB: copy all chunks from oldfd keys to newfd keys, then clear all oldfd keys — in one transaction. This is the atomic rename.

GetMeta() (FileDesc, error) and SetMeta(fd FileDesc) error Get/set the “current” file pointer — which MANIFEST is current. In FDB: a single key (ns + tagManifest + 0x00) stores the current FileDesc. This replaces the CURRENT file in LevelDB’s original design.

List() ([]FileDesc, error) List all files. We implement this as a range scan over the meta key prefix. We store a meta key for each file alongside its data.

3.1 There Are No Filesystem Calls in the Data Path

This is the most important thing to understand about the implementation. LevelDB calls our methods thinking it is talking to a real filesystem. But inside every method, instead of OS syscalls, we make FDB transactions.

// What a normal disk-backed implementation would do:
os.Open(path)             // syscall — reads from disk
os.Create(path)           // syscall — writes to disk

// What fdbstorage does instead:
rt.Get(metaKey(fd))                          // FDB read: does this file exist?
rt.GetRange(dataRange(fd)).GetSliceWithError() // FDB read: fetch all chunks
tr.Set(metaKey(fd), sizeBytes)               // FDB write: file metadata
tr.Set(chunkKey(fd, i), chunk)               // FDB write: file content

The only os package usage in the entire file is os.ErrNotExist — borrowed for its error semantics. There is no os.Open, no os.Create, no os.Read, no os.Write, no file descriptor, no inode. The filesystem is entirely replaced by the key naming scheme.

The exact translation points are:

4. How We Map LevelDB Files to FDB Keys

4.1 FileDesc: (Type, Num) — Not a Filename

LevelDB never uses string filenames internally. Every file is a FileDesc struct — just two numbers:

type FileDesc struct {
    Type FileType  // what category of file
    Num  int64     // which one — a counter, never reused
}

The String() method (MANIFEST-000001, 000002.log, 000003.ldb) is a display-only format for humans and log messages. Those strings never appear in FDB keys — only the raw integer values do.

Num is assigned by LevelDB’s internal file number counter, which only ever increases. If file 3 is deleted and a new SSTable is created, it gets number 5, not 3. This makes (Type, Num) a safe unique identifier for a key:

{TypeJournal,  2}  →  000002.log         (WAL)
{TypeManifest, 1}  →  MANIFEST-000001
{TypeTable,    3}  →  000003.ldb         (first SSTable)
{TypeTable,    4}  →  000004.ldb
       ↑                                 file 3 deleted → next is 5, not 3
{TypeTable,    5}  →  000005.ldb

The four file types and their byte values:

4.2 Namespace = Your Database Name

The ns prefix you pass to New() is the entire database identity. Two Storage instances with different namespaces share the same FDB cluster but never see each other’s keys:

storA := fdbstorage.New(db, "alice")  // all keys start with 0x616c6963 65
storB := fdbstorage.New(db, "bob")    // all keys start with 0x626f62

Both write SSTable num=3 (TypeTable=4). Their FDB keys are completely different:

alice's SSTable:  61 6c 69 63 65  01  04  00 00 00 00 00 00 00 03
                  └── "alice" ──┘
bob's   SSTable:  62 6f 62  01  04  00 00 00 00 00 00 00 03
                  └─ "bob" ┘

FDB sees them as unrelated keys in a flat sorted list. No schema, no directory, no tenant table — just different byte prefixes.

4.3 Your User Data is NOT an FDB Key

This surprises most people. When you write:

db.Put([]byte("apple"),  []byte("red"),    nil)
db.Put([]byte("banana"), []byte("yellow"), nil)
db.Put([]byte("cherry"), []byte("red"),    nil)

apple, banana, and cherry never appear as FDB keys. LevelDB packs them together into an SSTable file (a sorted binary format), then our flush() stores that binary blob across 64KB FDB chunk values:

User writes:   apple → red
               banana → yellow          ← LevelDB holds these in memory
               cherry → red
                     ↓  (on db.Close or memtable flush)
LevelDB creates one SSTable file: {TypeTable, Num=3}
                     ↓
FDB sees:
  "mydb" 01 04 0000000000000003  →  size (8 bytes)      ← file exists
  "mydb" 02 04 0000000000000003 00000000  →  <SSTable binary blob>
              ↑ TypeTable=4                ↑ the blob contains apple+banana+cherry
              chunk 0                        encoded in LevelDB's internal format

FDB stores opaque bytes. It has no idea what’s inside the chunk values. The apple/banana/cherry data is invisible to FDB — it can only be decoded by LevelDB when it reads the SSTable back.

4.4 How db.Get("bob") Actually Works — The Two-Level Lookup

If FDB only stores opaque SSTable blobs, how does LevelDB ever find bob? The answer is a two-level search: LevelDB decides which file, then FDB delivers that file’s bytes, then LevelDB searches inside the bytes.

db.Get([]byte("bob"))
         │
         ▼
╔══════════════════════════════════════╗
║  LEVEL 1: LevelDB finds the FILE     ║  ← no FDB I/O yet
╠══════════════════════════════════════╣
║  1. Check memtable (unflushed writes)
║  2. Read MANIFEST → live SSTables: #3, #5, #7
║  3. Check each SSTable's Bloom filter
║     → "bob is probably in SSTable #5"
║  4. Read SSTable #5's index block
║     → "bob lives in the block at byte offset 4096"
╚══════════════════════════════════════╝
         │
         ▼  stor.Open(FileDesc{TypeTable, 5}) ← FDB call here
╔════════════════════════════════════════╗
║  LEVEL 2: FDB reassembles the file     ║
╠════════════════════════════════════════╣
║  Range scan: ns 02 04 0000000000000005 *
║   chunk 0 → 64 KB of SSTable bytes
║   chunk 1 → 64 KB of SSTable bytes
║   chunk 2 → remainder
║  → reassemble → bytes.Reader
╚════════════════════════════════════════╝
         │
         ▼  (back in LevelDB, no more FDB I/O)
  binary-search SSTable at byte offset 4096
  find key "bob" → return value "smith"

What the FDB keys encode is only the file address, never the user key:

ns + 0x02 + 0x04 + 0000000000000005 + 00000000
│    │      │      │                  └─ chunk index
│    │      │      └─ file number (5)
│    │      └─ FileType (0x04 = TypeTable)
│    └─ tag byte (0x02 = file data)
└─ namespace ("mydb")

bob is packed inside the value bytes of those chunks, in LevelDB’s own sorted-block binary format, alongside every other key in that SSTable.

The three layers of indexing, summarized:

FDB is a file store. LevelDB is a key–value store built on top of it. The bob → smith lookup is entirely LevelDB’s responsibility. FDB just delivers SSTable #5’s bytes on demand.

4.5 Many Writes, One Flush — How Records Accumulate

Say you write 22 records:

db.Put([]byte("alex"), []byte("20"), nil)
db.Put([]byte("bob"),  []byte("25"), nil)
// ... 20 more

Do each of those create a new FDB key? No. Here is the full path:

db.Put("alex", "20")
db.Put("bob",  "25")      ← both go here immediately:
db.Put(...)               ↓
                    ┌─────────────────────┐
                    │  WAL (TypeJournal)  │  ← append each record as binary
                    │  in-memory buffer   │     Sync() → written to FDB
                    └─────────────────────┘
                    ┌─────────────────────┐
                    │  Memtable           │  ← sorted in-memory skip-list
                    │  alex=20, bob=25... │     all 22 records live here
                    └─────────────────────┘
                          │ (memtable full, ~4 MB, or db.Close())
                          ▼
                    LevelDB flushes the ENTIRE memtable
                    as ONE SSTable file (TypeTable, Num=3)
                          │
                          ▼  stor.Create({TypeTable,3}) → flush() → FDB
       ┌──────────────────────────────────────────────────────────────┐
       │  FDB keys written (all in one or two transactions):          │
       │  ns 01 04 0000000000000003          → meta (size)            │
       │  ns 02 04 0000000000000003 00000000 → chunk 0: sorted blob   │
       │           ↑ all 22 records (alex, bob, ...) packed in here   │
       └──────────────────────────────────────────────────────────────┘

Key rules:

What about the WAL?

The WAL (TypeJournal) is different — it IS append-like. As you call db.Put(), LevelDB appends binary records to its Journal writer. In our FDB implementation, every Sync() call rewrites the Journal’s chunks from scratch (chunk 0, 1, 2…) because our writer accumulates into a bytes.Buffer and flush() always writes the full buffer. There is no partial append in FDB — we over-write all chunks with the latest snapshot of the buffer.

after db.Put("alex"):  WAL buffer = [record-for-alex]
                       Sync() → FDB chunk 0 = [record-for-alex] (32 bytes)

after db.Put("bob"):   WAL buffer = [record-for-alex | record-for-bob]
                       Sync() → FDB chunk 0 = [record-for-alex | record-for-bob]
                                 (entire buffer, overwriting previous chunk 0)

Once the memtable is flushed to an SSTable, the WAL is deleted (it’s no longer needed for recovery — the data is in the immutable SSTable).

Each LevelDB file is identified by (FileType, FileNum). We encode this as:

Meta key (file existence + type):
  ns + tagFileMeta(0x01) + fileType(1 byte) + fileNum(8 bytes BE)
  → msgpack({Type: ft, Num: n, size: totalBytes})

Data chunks:
  ns + tagFileData(0x02) + fileType(1 byte) + fileNum(8 bytes BE) + chunkNum(8 bytes BE)
  → up to 64 KiB of file data

Manifest pointer (replaces CURRENT file):
  ns + tagManifest(0x03)
  → msgpack(FileDesc{Type: TypeManifest, Num: n})

Lock key:
  ns + tagLock(0x04)
  → "locked" (any non-empty value means locked)

Why big-endian for file numbers?

Big-endian encoding preserves sort order. File numbers are monotonically increasing (LevelDB never reuses a file number). By storing them big-endian, a range scan over ns+tagFileData+ft+num+* returns chunks in chunk-number order — which is the correct order to reassemble the file. Without big-endian encoding, chunk 10 would sort before chunk 2 (0x0A < 0x02 is false in big-endian but 0x0000000A < 0x00000002 is also false — you need lexicographic order over big-endian bytes).

File number and type as part of the key:

This means all chunks of file (TypeJournal, 3) sort together, before all chunks of (TypeJournal, 4), which sort before (TypeTable, 5). Clean, hierarchical key organization.

5. Key Layout Deep Dive: From Function Call to FDB Bytes

This section traces the complete journey of every operation — from a Go function call in storage.go down to the raw bytes written in FDB. If you read only one section, read this one.

5.1 The Four Tag Bytes

const (
    tagFileMeta byte = 0x01  // "does this file exist, and how big is it?"
    tagFileData byte = 0x02  // "here are the actual file contents"
    tagManifest byte = 0x03  // "which MANIFEST file is currently active?"
    tagLock     byte = 0x04  // "is this database open by a process?"
)

FDB is a flat key→value store. There are no tables, folders, or schemas — just a sorted sequence of byte keys. To store four conceptually different things (file metadata, file data chunks, the manifest pointer, the lock) without collisions, the very first byte after the namespace prefix tells you what kind of record you are looking at. This is the tag byte.

Think of it like a URL path prefix:

• /meta/... → tag 0x01
• /data/... → tag 0x02
• /manifest → tag 0x03
• /lock → tag 0x04

5.2 Anatomy of Every Key Type

ns = "mydb"  (4 bytes: 0x6D 0x79 0x64 0x62)

── tagFileMeta (0x01) ──────────────────────────────────────────────────────
Key:   6D 79 64 62  01  04  00 00 00 00 00 00 00 03
       └── ns ───┘  ↑   ↑   └────── num (int64 BE) ──┘
                    │   └── fd.Type = TypeTable (4 = 0x04)
                    └── tagFileMeta
Value: 00 00 00 00 00 20 00 00   (uint64 BE = 2097152 = 2 MB)

── tagFileData (0x02) ──────────────────────────────────────────────────────
Key:   6D 79 64 62  02  04  00 00 00 00 00 00 00 03  00 00 00 01
       └── ns ───┘  ↑   ↑   └────── num (int64 BE) ──┘  └chunk┘
                    │   └── fd.Type = TypeTable (4)
                    └── tagFileData
Value: <65536 bytes of SSTable data, chunk index 1>

── tagManifest (0x03) ──────────────────────────────────────────────────────
Key:   6D 79 64 62  03
       └── ns ───┘  └── tagManifest (no other fields — there is only one)
Value: 01  00 00 00 00 00 00 00 01
       ↑   └────── Num = 1 (int64 BE) ──────────────────────────────────┘
       └── fd.Type = TypeManifest (1 = 0x01)

── tagLock (0x04) ──────────────────────────────────────────────────────────
Key:   6D 79 64 62  04
       └── ns ───┘  └── tagLock
Value: (empty — presence of the key = locked; absent = unlocked)

5.3 Journey: db.Put("hello", "world") → FDB keys

Here is the full call chain from a single LevelDB write to the bytes that land in FDB.

User code
  db.Put([]byte("hello"), []byte("world"), nil)
    │
    ▼ goleveldb internal
  memTable.Put(...)          ← stored in memory only
    │
    ▼  (on MemTable flush or db.Close)
  compaction goroutine
    │  calls our storage interface:
    ├─ stor.Create(FileDesc{Type: TypeJournal, Num: 2})
    │    └─ returns &writer{fd: {TypeJournal, 2}}
    │
    ├─ writer.Write(journalRecord)    ← buffered in writer.buf
    ├─ writer.Sync()                  ← calls flush()
    │    └─ fdb.Transact:
    │         tr.ClearRange(dataRange({TypeJournal, 2}))        ← wipe old
    │         tr.Set(metaKey({TypeJournal, 2}), size_8bytes)    ← tag 0x01
    │         tr.Set(chunkKey({TypeJournal, 2}, 0), chunk0)     ← tag 0x02
    │
    └─ writer.Close()

At this point two FDB keys exist:

"mydb" 0x01 0x02 0x00000000_00000002   →  size (8 bytes)
"mydb" 0x02 0x02 0x00000000_00000002 0x00000000  →  journal bytes

(0x02 in the 3rd byte position is TypeJournal = 2, not tagFileData.)

5.4 Journey: stor.Open(FileDesc{TypeJournal, 2})

When goleveldb recovers after a restart it calls Open on each journal file it found via List. Here is what happens:

// In storage.go — Open()
func (s *Storage) Open(fd storage.FileDesc) (storage.Reader, error) {
    v, err := s.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {

        // Step 1 — fetch the meta key (tag 0x01)
        // Key: "mydb" 0x01 0x02 0x00000000_00000002
        meta, err := rt.Get(s.metaKey(fd)).Get()
        // meta == []byte{0x00,0x00,0x00,0x00,0x00,0x00,0x04,0x00}  (1024 bytes)

        size := binary.BigEndian.Uint64(meta)   // = 1024

        // Step 2 — range-scan all chunk keys (tag 0x02)
        // Range: ["mydb" 0x02 0x02 0x00000000_00000002]
        //         ──────────────────────────────────────────────────────────────
        //        ["mydb" 0x02 0x02 0x00000000_00000002 0xFF 0xFF 0xFF 0xFF 0xFF]
        kvs, _ := rt.GetRange(s.dataRange(fd), fdb.RangeOptions{}).GetSliceWithError()
        // kvs[0].Key   = "mydb" 0x02 0x02 ... 0x00000000  (chunk 0)
        // kvs[0].Value = <1024 bytes>

        buf := make([]byte, 0, size)
        for _, kv := range kvs {
            buf = append(buf, kv.Value...)   // reassemble from chunks in order
        }
        return buf, nil
    })
    return &reader{bytes.NewReader(v.([]byte))}, nil
}

Two FDB reads, one round-trip (both happen inside one ReadTransact).

5.5 Journey: stor.List(TypeAll)

Called at startup. goleveldb needs to know every file that exists so it can decide which journal files to replay and which SSTables are live.

// In storage.go — List()
func (s *Storage) List(ft storage.FileType) ([]storage.FileDesc, error) {
    // Scan ONLY the tag-0x01 band — never touch the data keys.
    allMeta := fdb.KeyRange{
        Begin: fdb.Key(append([]byte(s.ns), tagFileMeta)),      // "mydb" 0x01
        End:   fdb.Key(append([]byte(s.ns), tagFileMeta+1)),    // "mydb" 0x02
    }
    // Returns all keys that start with "mydb" 0x01 ...
    // Example keys returned:
    //   "mydb" 0x01 0x01 0x00000000_00000001  → MANIFEST-000001
    //   "mydb" 0x01 0x02 0x00000000_00000002  → 000002.log
    //   "mydb" 0x01 0x04 0x00000000_00000003  → 000003.ldb
    //   "mydb" 0x01 0x04 0x00000000_00000004  → 000004.ldb

    v, _ := s.fdb.ReadTransact(...)

    prefixLen := len(s.ns) + 2   // skip: ns + tagFileMeta + ftype byte
    for _, kv := range v.([]fdb.KeyValue) {
        k := []byte(kv.Key)
        thisFT := storage.FileType(k[len(s.ns)+1])    // byte after the tag
        if ft&thisFT == 0 { continue }                // bitmask filter
        num := int64(binary.BigEndian.Uint64(k[prefixLen : prefixLen+8]))
        out = append(out, storage.FileDesc{Type: thisFT, Num: num})
    }
}

One range scan, one round-trip. Notice that the data chunks (tag 0x02) are never touched — List only reads the tiny meta keys (tag 0x01). This is why having separate tags for metadata vs data matters: you can enumerate all files without reading any file contents.

5.6 Journey: stor.SetMeta(FileDesc{TypeManifest, 1})

Called by goleveldb after it writes a new MANIFEST file, to record “this is now the current MANIFEST”.

func (s *Storage) SetMeta(fd storage.FileDesc) error {
    _, err := s.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        // encodeFD packs 9 bytes: 1 byte type + 8 bytes num BE
        // fd = {TypeManifest=1, Num=1}
        // encoded = [0x01, 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x01]
        tr.Set(s.manifestKey(), encodeFD(fd))
        //      └── key: "mydb" 0x03 (just the ns + tag, no other fields)
        return nil, nil
    })
    return err
}

And the reverse, GetMeta():

func (s *Storage) GetMeta() (storage.FileDesc, error) {
    v, _ := s.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.Get(s.manifestKey()).Get()   // key: "mydb" 0x03
    })
    b := v.([]byte)
    if b == nil {
        return storage.FileDesc{}, os.ErrNotExist   // fresh DB — no manifest yet
    }
    // b = [0x01, 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x01]
    return decodeFD(b)   // → {TypeManifest, Num: 1}
}

5.7 What the FDB keyspace looks like for a small database

After running demo/main.go and writing a few keys, the entire FDB namespace "mydb" might look like this (printed as hexdump | ascii):

KEY                                                     VALUE
───────────────────────────────────────────────────────────────────────────
mydb·03                                                 \x01·\x00\x00\x00\x00\x00\x00\x00\x01
                                                        ↑ MANIFEST pointer → {TypeManifest, 1}

mydb·01·01·\x00\x00\x00\x00\x00\x00\x00\x01           \x00\x00\x00\x00\x00\x00\x04\xA0
                                                        ↑ meta for MANIFEST-000001, size=1184

mydb·02·01·\x00\x00\x00\x00\x00\x00\x00\x01·\x00\x00\x00\x00   <1184 bytes>
                                                        ↑ MANIFEST-000001 data, chunk 0

mydb·01·02·\x00\x00\x00\x00\x00\x00\x00\x02           \x00\x00\x00\x00\x00\x00\x10\x00
                                                        ↑ meta for 000002.log, size=4096

mydb·02·02·\x00\x00\x00\x00\x00\x00\x00\x02·\x00\x00\x00\x00   <4096 bytes>
                                                        ↑ 000002.log data, chunk 0

mydb·01·04·\x00\x00\x00\x00\x00\x00\x00\x03           \x00\x00\x00\x00\x00\x20\x00\x00
                                                        ↑ meta for 000003.ldb, size=2MB

mydb·02·04·\x00\x00\x00\x00\x00\x00\x00\x03·\x00\x00\x00\x00   <65536 bytes>
mydb·02·04·\x00\x00\x00\x00\x00\x00\x00\x03·\x00\x00\x00\x01   <65536 bytes>
...                                                     ↑ 000003.ldb data, 32 chunks
mydb·02·04·\x00\x00\x00\x00\x00\x00\x00\x03·\x00\x00\x00\x1F   <last chunk>

Key observations:

• All 0x01 (meta) keys sort before all 0x02 (data) keys — because 0x01 < 0x02.
• Within the 0x01 band, TypeManifest (0x01) sorts before TypeJournal (0x02) sorts before TypeTable (0x04) — because their type bytes are in numeric order.
• Chunks within a file sort by chunk index — because BE integers preserve numeric order.
• List() range-scans only [mydb·01, mydb·02) — skipping all chunk data entirely.

5.8 Verifying with a test

Here is a self-contained Go test that exercises every code path above and lets you inspect the raw FDB keys to verify they match the byte layout:

// fdbstorage/storage_layout_test.go
package fdbstorage_test

import (
    "bytes"
    "encoding/hex"
    "fmt"
    "testing"

    "github.com/apple/foundationdb/bindings/go/src/fdb"
    "github.com/syndtr/goleveldb/leveldb/storage"

    fdbstorage "github.com/your-module/fdbstorage"
)

func TestKeyLayout(t *testing.T) {
    fdb.MustAPIVersion(620)
    db := fdb.MustOpenDefault()

    stor := fdbstorage.New(db, "testlayout")
    stor.Wipe() // always start from a clean slate

    // ── 1. Write a small file ────────────────────────────────────────────
    fd := storage.FileDesc{Type: storage.TypeJournal, Num: 7}
    w, err := stor.Create(fd)
    if err != nil {
        t.Fatal(err)
    }
    payload := []byte("hello from the journal")
    w.Write(payload)
    if err := w.Close(); err != nil {
        t.Fatal(err)
    }

    // ── 2. SetMeta (manifest pointer) ───────────────────────────────────
    manifest := storage.FileDesc{Type: storage.TypeManifest, Num: 1}
    if err := stor.SetMeta(manifest); err != nil {
        t.Fatal(err)
    }

    // ── 3. Dump all raw FDB keys in the namespace ────────────────────────
    // Expected output:
    //   74657374 6c61796f 7574 03                             ← manifest key
    //   74657374 6c61796f 7574 01 02 0000000000000007         ← meta key
    //   74657374 6c61796f 7574 02 02 0000000000000007 00000000 ← data chunk 0
    allKeys, err := stor.DumpKeys()
    if err != nil {
        t.Fatal(err)
    }
    fmt.Println("\n=== raw FDB keys in namespace ===")
    for _, k := range allKeys {
        fmt.Printf("  %s\n", hex.EncodeToString(k))
    }
    if len(allKeys) != 3 {
        t.Fatalf("want 3 keys (manifest + meta + 1 chunk), got %d", len(allKeys))
    }

    // ── 4. List round-trip ───────────────────────────────────────────────
    fds, err := stor.List(storage.TypeAll)
    if err != nil {
        t.Fatal(err)
    }
    if len(fds) != 1 || fds[0] != fd {
        t.Fatalf("List: want [%v], got %v", fd, fds)
    }

    // ── 5. Open round-trip (bytes must match) ────────────────────────────
    r, err := stor.Open(fd)
    if err != nil {
        t.Fatal(err)
    }
    buf := make([]byte, len(payload))
    if _, err := r.Read(buf); err != nil {
        t.Fatal(err)
    }
    if !bytes.Equal(buf, payload) {
        t.Fatalf("Open: want %q, got %q", payload, buf)
    }

    // ── 6. GetMeta round-trip ────────────────────────────────────────────
    got, err := stor.GetMeta()
    if err != nil {
        t.Fatal(err)
    }
    if got != manifest {
        t.Fatalf("GetMeta: want %v, got %v", manifest, got)
    }

    // ── 7. Remove clears both meta and data keys ─────────────────────────
    if err := stor.Remove(fd); err != nil {
        t.Fatal(err)
    }
    remaining, _ := stor.DumpKeys()
    // Only the manifest key should remain
    if len(remaining) != 1 {
        t.Fatalf("after Remove: want 1 key, got %d: %v", len(remaining),
            func() []string {
                var s []string
                for _, k := range remaining {
                    s = append(s, hex.EncodeToString(k))
                }
                return s
            }())
    }

    stor.Wipe()
}

To use DumpKeys you need to add this helper to fdbstorage/storage.go:

// DumpKeys returns all raw FDB keys in this storage's namespace, in order.
// Intended for tests and debugging only.
func (s *Storage) DumpKeys() ([][]byte, error) {
    nsEnd := append(append([]byte{}, s.ns...), 0xff)
    v, err := s.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(fdb.KeyRange{
            Begin: fdb.Key(s.ns),
            End:   fdb.Key(nsEnd),
        }, fdb.RangeOptions{}).GetSliceWithError()
    })
    if err != nil {
        return nil, err
    }
    kvs := v.([]fdb.KeyValue)
    out := make([][]byte, len(kvs))
    for i, kv := range kvs {
        out[i] = []byte(kv.Key)
    }
    return out, nil
}

The test confirms:

1. Create + Close produces exactly one meta key (0x01) and one chunk key (0x02).
2. List finds the file by scanning only 0x01 keys.
3. Open reconstructs the payload exactly by reading the 0x02 chunk.
4. SetMeta / GetMeta round-trips through the single 0x03 key.
5. Remove clears both the 0x01 meta key and all 0x02 chunk keys, leaving nothing behind.

5.9 Live Simulation — Annotated Output

The demo/simulate.go program writes three files, dumps every raw FDB key, reads them back, and calls GetMeta + List. Run it with:

go run ./demo/ -sim

Here is the full output with every byte explained.

Phase 1 — Write

wrote  MANIFEST-000001        "MANIFEST-content"
wrote  000002.log             "WAL-entry-bytes"
wrote  000005.ldb             "SST-block-bytes"

Each call is:

stor.Create(fd)        // returns a *writer with an empty bytes.Buffer — no FDB I/O yet
w.Write([]byte(...))   // appends into writer.buf — still no FDB I/O
w.Close()              // calls flush() → fires fdb.Transact
                       //   tr.Set(metaKey(fd),  sizeBytes)  ← tag 0x01 key
                       //   tr.Set(chunkKey(fd,0), data)     ← tag 0x02 key

Nothing reaches FDB until Close() (or Sync()). LevelDB thinks it has a real file. FDB only learns about it at flush time.

Phase 2 — FDB Keyspace Dump

META   key=73696d0001010000000000000001  →  type=Manifest  num=1   size=16 B
META   key=73696d0001020000000000000002  →  type=Journal   num=2   size=15 B
META   key=73696d0001040000000000000005  →  type=Table     num=5   size=15 B

Byte-by-byte for the first META key:

73 69 6d 00        ← namespace "sim\x00"  (4 bytes)
           01      ← tagFileMeta  (the "directory entry" tag)
              01   ← fd.Type = TypeManifest = 1
                 00 00 00 00 00 00 00 01
                 └────── fd.Num = 1, big-endian int64 ───┘

Value: 00 00 00 00 00 00 00 10  ← uint64 BE = 16 — file size in bytes

The three CHUNK keys follow immediately after in the sorted order:

CHUNK  key=73696d000201000000000000000100000000  →  type=Manifest  chunk=0  16 B

73 69 6d 00        ← namespace
           02      ← tagFileData  (actual content)
              01   ← TypeManifest
                 00 00 00 00 00 00 00 01   ← num=1
                                         00 00 00 00  ← chunk index 0 (uint32 BE)

Value: "MANIFEST-content"  (16 bytes, the raw file payload)

The sorted order in FDB is:

All META  (0x01) keys come first because 0x01 < 0x02
  ├── TypeManifest (0x01) meta    ← 0x01·0x01·...
  ├── TypeJournal  (0x02) meta    ← 0x01·0x02·...
  └── TypeTable    (0x04) meta    ← 0x01·0x04·...

All CHUNK (0x02) keys come next
  ├── TypeManifest (0x01) chunk 0 ← 0x02·0x01·...·0x00000000
  ├── TypeJournal  (0x02) chunk 0 ← 0x02·0x02·...·0x00000000
  └── TypeTable    (0x04) chunk 0 ← 0x02·0x04·...·0x00000000

MANIF (0x03) key  ← single key, no extra fields

The MANIF key:

MANIF  key=73696d0003  →  points to type=Manifest num=1

73 69 6d 00        ← namespace
           03      ← tagManifest  (that's it — only one manifest at a time)

Value: 01  00 00 00 00 00 00 00 01
       ↑   └────── Num = 1 ─────┘
       └── fd.Type = TypeManifest = 1

This is exactly what encodeFD produces and decodeFD expects.

Phase 3 — Read Back

open   MANIFEST-000001        → "MANIFEST-content"
open   000002.log             → "WAL-entry-bytes"
open   000005.ldb             → "SST-block-bytes"

Each Open(fd) fires one FDB ReadTransact with two operations:

1. rt.Get(metaKey(fd))           ← fetch the 0x01 key  → confirms file exists + size
2. rt.GetRange(dataRange(fd))    ← fetch all 0x02 keys  → reassemble bytes in chunk order

Both happen inside the same snapshot. FDB guarantees you see a consistent view: no partial writes, no torn reads across chunks.

Phase 4 — GetMeta + List

GetMeta() → MANIFEST-000001

List(TypeAll):
  MANIFEST-000001
  000002.log
  000005.ldb

GetMeta() is one point-read on the single 0x03 key.

List(TypeAll) is one range scan over [ns·0x01, ns·0x02) — the entire META band. It decodes the type byte and num from each key, never touching a single CHUNK key:

thisFT := storage.FileType(k[len(s.ns)+1])   // byte at position: ns + tagByte + HERE
num    := int64(binary.BigEndian.Uint64(k[prefixLen : prefixLen+8]))

Three files → three META keys scanned → three FileDesc returned. The nine CHUNK keys (one per file) are not even requested.

Summary table

6. Chunking: Overcoming the 100 KiB Value Limit

FDB has a hard limit: values may not exceed 100 KiB (102,400 bytes). A typical LevelDB SSTable is 2–4 MB. We cannot store it in one FDB value.

Our solution: split each file into 64 KiB chunks:

const chunkSize = 64 * 1024  // 65,536 bytes

// Writing a 200 KiB file:
// Chunk 0: bytes [0, 65536)
// Chunk 1: bytes [65536, 131072)
// Chunk 2: bytes [131072, 200000)  (partial last chunk)

Each chunk is stored as a separate FDB key-value pair:

ns+0x02+ft+num+00000000_00000000  → 65536 bytes
ns+0x02+ft+num+00000000_00000001  → 65536 bytes
ns+0x02+ft+num+00000000_00000002  → 68528 bytes (partial)

Reading a file: range-scan all chunk keys for (ft, num), sort by chunk number (already in order due to big-endian encoding), concatenate the values.

func (s *Storage) Open(fd storage.FileDesc) (storage.Reader, error) {
    kvs, _ := s.db.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(s.dataRange(fd), fdb.RangeOptions{}).GetSliceWithError()
    })
    var buf []byte
    for _, kv := range kvs.([]fdb.KeyValue) {
        buf = append(buf, kv.Value...)
    }
    return io.NopCloser(bytes.NewReader(buf)), nil
}

Why 64 KiB chunks?

• Smaller than FDB’s 100 KiB value limit ✓
• Large enough to minimize key overhead (a 4 MB SSTable = 64 chunks, not thousands)
• Aligns with filesystem block sizes (4–64 KiB typical)

7. Atomic Rename — Durability’s Secret Weapon

POSIX rename(src, dst) is the single most important durability primitive in filesystems. Its contract: after rename returns, dst exists and src does not, with no window where neither exists. This is atomic replacement.

LevelDB uses rename heavily:

• Rename(TypeTemp, n, TypeTable, n): promote temp SSTable to final name
• Rename(TypeTemp, n, TypeManifest, n): promote temp manifest

Without atomicity, a crash during rename could leave:

• Neither file existing → data loss
• Both files existing → ambiguity about which is current
• A partial file at dst → corruption

In FDB, we implement atomic rename as copy + clear in one transaction:

func (s *Storage) Rename(oldfd, newfd storage.FileDesc) error {
    _, err := s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        // 1. Read all old chunks
        kvs, _ := tr.GetRange(s.dataRange(oldfd), fdb.RangeOptions{}).GetSliceWithError()

        // 2. Clear old data
        tr.ClearRange(s.dataRange(oldfd))
        tr.Clear(s.metaKey(oldfd))

        // 3. Write new data
        for _, kv := range kvs {
            newKey := s.translateChunkKey(kv.Key, oldfd, newfd)
            tr.Set(newKey, kv.Value)
        }
        tr.Set(s.metaKey(newfd), metaBytes)
        return nil, nil
    })
    return err
}

One transaction. The cluster either commits all of this (old is gone, new is present) or none of it (crash safety). The atomicity guarantee is identical to POSIX rename — and arguably stronger, since FDB replicates the commit across multiple machines before returning.

The 10 MB transaction limit:

FDB transactions are limited to ~10 MB of reads + writes. A large SSTable (4 MB) would have chunks adding up to 4 MB of writes in one transaction. That’s under the 10 MB limit. But 8 MB SSTables would be risky.

Our Rename reads all chunks in the transaction (4 MB reads) and writes them all back (4 MB writes) — totaling 8 MB. Safe for typical LevelDB files.

For larger files, we’d need to either:

1. Break the rename into multiple transactions (violating atomicity), or
2. Use a two-phase approach: write new chunks in a first transaction, then atomically swap the meta key in a second transaction (using a PENDING state key as the “in-progress rename” marker).

8. The Writer: Batching Chunks into Transactions

When LevelDB writes a new SSTable, it calls Create(fd) which returns a Writer. The writer accumulates bytes via Write(p []byte). When Close() is called, we flush everything to FDB.

Batch size:

const maxChunksPerTx = 100  // 100 × 64 KiB = 6.4 MB per transaction

We flush up to 100 chunks per FDB transaction. This stays well within the 10 MB limit. A 20 MB SSTable would be flushed in 4 transactions of 5 MB each.

func (w *writer) flush(final bool) error {
    start := w.flushedChunks
    end := start + maxChunksPerTx
    if end > len(w.chunks) {
        end = len(w.chunks)
    }
    _, err := w.s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        for i := start; i < end; i++ {
            tr.Set(w.s.chunkKey(w.fd, i), w.chunks[i])
        }
        if final && end == len(w.chunks) {
            // Write the meta key only on the final flush
            tr.Set(w.s.metaKey(w.fd), metaBytes)
        }
        return nil, nil
    })
    w.flushedChunks = end
    return err
}

The meta-key-last invariant:

We write the meta key (the file’s “directory entry”) only in the last batch of chunks. This ensures that List() never returns a file whose chunks are only partially written — the file is only “visible” once all its chunks exist.

This is the FDB equivalent of:

1. Write all content to a temp file
2. rename(temp, final) atomically

9. Why the WAL is Redundant With FDB

LevelDB’s Write-Ahead Log (WAL / journal, TypeJournal) exists for one reason: crash recovery. If the process crashes after writing to the in-memory MemTable but before flushing the MemTable to an SSTable on disk, the WAL is replayed to reconstruct the MemTable.

With FDB as the storage backend:

Every write is already durable before Write(p) returns.

Our writer.Write buffers bytes in memory. Our writer.Close flushes to FDB in transactions. Each FDB Transact call does not return until the commit is confirmed by FDB’s replication protocol — the data is on at least f+1 machines (where f is the fault tolerance level, typically 2). A process crash after Close() returns means the data is safe.

The WAL is protecting against “data written to OS memory but not yet on disk.” FDB’s Transact eliminates this window. By the time the WAL file is written through our fdbstorage.Writer, the bytes are already in FDB.

A production implementation would patch goleveldb to skip WAL writes entirely (or use Options.DisableSeeksCompaction and a custom journal implementation that’s a no-op). This would improve write throughput by 50% or more and reduce FDB key usage.

10. The Blob Layer Pattern

FDB’s core team documented the “Blob Layer” pattern: storing binary blobs (arbitrary large byte arrays) in FDB by chunking them. Our file storage is an instance of this pattern.

The Blob Layer pattern:

blob_key + chunkNum  →  chunk_data

It solves the 100 KiB value limit while preserving atomic operations on the whole blob (via FDB transactions) and efficient byte-range access (read only the chunks you need, e.g., for seeking within a large file).

Applications:

• Store large media files (> 100 KiB) in FDB for atomic metadata-plus-content updates
• Store ML model weights alongside their metadata records
• Store serialized protocol buffers larger than 100 KiB
• Back any file system abstraction (exactly what we’re doing)

11. Real-World Analogues

RocksDB Remote Compaction (Project Titan, Ripple)

Meta (Facebook) runs RocksDB on distributed storage in some configurations. Their “Ripple” project stores RocksDB SSTables in a distributed block store (similar to HDFS or GFS). The storage interface they use is exactly the same concept: RocksDB writes “files” via an abstract interface; the implementation stores chunks in a distributed system.

TiKV on Disaggregated Storage

TiDB (PingCAP) is moving toward a disaggregated architecture where TiKV (which uses RocksDB internally) stores its SSTables in object storage (S3, GCS). The TiKV storage engine writes SSTables through an abstract file interface to S3. This is identical to our pattern.

Pebble (CockroachDB)

CockroachDB replaced RocksDB with Pebble (a Go implementation) in 2021. Pebble has a vfs.FS interface — a virtual filesystem abstraction — that allows swapping the storage backend. CockroachDB uses this for testing (an in-memory FS) and is exploring using it for cloud storage.

The Pattern’s Universality

Every LSM-tree engine eventually adds a pluggable storage interface:

• LevelDB: storage.Storage
• RocksDB: Env (virtual filesystem)
• Pebble: vfs.FS
• WiredTiger: WT_FILE_SYSTEM

Why? Because running the compaction engine without worrying about where data lives is architecturally clean. The engine is responsible for LSM semantics; the storage interface is responsible for durability. Separation of concerns.

12. Exercises

Exercise 1 — Streaming Reader

Instead of materializing the entire file into memory in Open(), return an io.ReadSeekCloser that fetches chunks lazily. A read at offset 128 KiB should only fetch chunks 2–3, not chunk 0 and 1.

This reduces memory usage for large SSTables and enables efficient Seek(offset, io.SeekStart) for random-access reads.

Exercise 2 — File Size Cache

List() currently returns all file descriptors by scanning the meta keys. Open(fd) reads the meta key to get the file size, then reads all chunk keys.

Add a small in-memory LRU cache mapping FileDesc → size. On Open, check the cache first. Invalidate the cache entry on Remove and Rename.

Measure the reduction in FDB round-trips for a workload with many small reads on recently-opened files.

Exercise 3 — Compression

Before storing each 64 KiB chunk, compress it with compress/flate or github.com/golang/snappy. Store a compression-type byte in the meta key. On read, decompress transparently.

LevelDB SSTables are already internally compressed (Snappy by default), so this may not reduce size much for TypeTable files. But TypeJournal files are not compressed and might benefit.

Exercise 4 — Two-Phase Large Rename

For files larger than 5 MB (which would exceed the transaction limit in our current Rename), implement the two-phase rename:

Phase 1: Write all new chunks in multiple transactions. Write a “rename-pending” key: ns+tagPending+oldfd → newfd.

Phase 2: In one transaction, atomically: clear the pending key, clear all old chunks and meta, set the meta for new fd (chunks already exist).

On startup, check for any pending keys and complete or roll back the rename. This is essentially a two-phase commit for large file renames.

Exercise 5 — Multi-Tenant Databases

Add a namespace concept: allow multiple LevelDB databases to share one FDB cluster with independent key spaces. Each New(fdb, namespace) call returns a storage implementation that is completely isolated from others.

This is how mvsqlite handles multiple SQLite “database files” — each is an FDB namespace.

13. Source Code Deep Dive — fdbstorage/storage.go

The Storage Struct

type Storage struct {
    db  fdb.Database
    ns  []byte
}

Minimal. db is the FDB connection; ns is the byte prefix for all keys. The entire storage is two fields. All complexity lives in the key encoding and transaction logic.

Key Encoding Helpers

func (s *Storage) metaKey(fd storage.FileDesc) fdb.Key {
    // ns + 0x01 + type(1 byte) + num(8 bytes big-endian)
    key := make([]byte, len(s.ns)+10)
    copy(key, s.ns)
    key[len(s.ns)] = 0x01
    key[len(s.ns)+1] = byte(fd.Type)
    binary.BigEndian.PutUint64(key[len(s.ns)+2:], uint64(fd.Num))
    return fdb.Key(key)
}

func (s *Storage) chunkKey(fd storage.FileDesc, chunkNum int) fdb.Key {
    // ns + 0x02 + type(1 byte) + num(8 bytes) + chunk(8 bytes)
    key := make([]byte, len(s.ns)+18)
    copy(key, s.ns)
    key[len(s.ns)] = 0x02
    key[len(s.ns)+1] = byte(fd.Type)
    binary.BigEndian.PutUint64(key[len(s.ns)+2:], uint64(fd.Num))
    binary.BigEndian.PutUint64(key[len(s.ns)+10:], uint64(chunkNum))
    return fdb.Key(key)
}

Why 8-byte big-endian for chunkNum? Chunk numbers are read back via GetRange which returns chunks in key order. Big-endian ensures key order equals chunk number order. If we used little-endian, chunk 256 (LE: 00 01 00 00 00 00 00 00) would sort before chunk 1 (LE: 01 00 00 00 00 00 00 00) — wrong.

The dataRange Helper

func (s *Storage) dataRange(fd storage.FileDesc) fdb.KeyRange {
    begin := s.chunkKey(fd, 0)
    // end: same prefix but with chunkNum = MaxUint64 + 1 — use next-prefix trick
    endPrefix := make([]byte, len(s.ns)+10) // ns + 0x02 + type + num
    copy(endPrefix, s.ns)
    endPrefix[len(s.ns)] = 0x02
    endPrefix[len(s.ns)+1] = byte(fd.Type)
    binary.BigEndian.PutUint64(endPrefix[len(s.ns)+2:], uint64(fd.Num))
    end := append(endPrefix, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF)
    return fdb.KeyRange{Begin: begin, End: fdb.Key(append(end, 0x01))}
}

This range covers all chunk keys for (type, num) regardless of chunkNum. GetRange(dataRange(fd)) fetches all chunks in order.

The List() Implementation

func (s *Storage) List() ([]storage.FileDesc, error) {
    kvs, _ := s.db.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        return rt.GetRange(s.metaRange(), fdb.RangeOptions{}).GetSliceWithError()
    })
    var fds []storage.FileDesc
    for _, kv := range kvs.([]fdb.KeyValue) {
        var fd storage.FileDesc
        msgpack.Unmarshal(kv.Value, &fd)
        fds = append(fds, fd)
    }
    return fds, nil
}

A single range scan over all meta keys returns all files in one round-trip. LevelDB calls List() at startup to find all existing files. With FDB, this is O(1) round-trips regardless of file count.

With a local filesystem, List() is an opendir/readdir syscall — also O(1) in latency, but I/O must go through the local disk controller. With FDB, the I/O goes to the closest FDB storage server over the network, with similar or lower latency than a rotational disk.

The Lock Implementation

func (s *Storage) Lock() (util.Releaser, error) {
    _, err := s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        existing, _ := tr.Get(s.lockKey()).Get()
        if len(existing) > 0 {
            return nil, errors.New("storage: already locked")
        }
        tr.Set(s.lockKey(), []byte("locked"))
        return nil, nil
    })
    if err != nil {
        return nil, err
    }
    return util.ReleaserFunc(func() {
        s.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
            tr.Clear(s.lockKey())
            return nil, nil
        })
    }), nil
}

The lock is a FDB key. Acquiring the lock: check if the key exists; if not, set it — in one atomic transaction. This check-then-set is race-free because FDB’s optimistic concurrency ensures that if two processes both read “no lock” and both try to write “locked”, only one will commit (the other will conflict and retry, then find the lock held).

Limitation: This is a process-level lock, not a durable lease. If the lock-holding process crashes without calling Release(), the lock remains set until manually cleared. For production, use a lock with an expiry: store the lock as {holder: processID, expires: time.Now().Add(30*time.Second)} and have each lock holder refresh it periodically. A lock that isn’t refreshed is treated as expired.

14. Production Considerations

14.1 Transaction Size for Large SSTables

LevelDB level-0 SSTables are 2–4 MB. Level-1 SSTables are larger (up to L1_target_size, configurable). For a L1_target_size of 64 MB, level-1 SSTables are 64 MB each. Our current Rename would fail for files this large (exceeds the 10 MB transaction limit).

Solution: For production, configure LevelDB’s CompactionTableSize to keep SSTables small:

opts := &opt.Options{
    CompactionTableSize: 2 * 1024 * 1024,  // 2 MB SSTables
}

2 MB SSTables = 32 chunks of 64 KiB. Rename transaction: 32 reads + 32 writes = 4 MB total. Well within limits.

14.2 Read Performance for Large SSTables

Reading a 2 MB SSTable requires fetching 32 chunks from FDB. Our current implementation reads them in one GetRange — one round-trip, 32 key-value pairs returned. Latency: ~1–5 ms (FDB cluster local read latency).

A local filesystem read of 2 MB: ~1–3 ms on SSD, ~10–20 ms on HDD.

For a warm FDB cluster, FDB storage is competitive with SSDs and dramatically better than spinning disks. For random chunk access (seeking within large files), FDB may be faster because it can pipeline multiple point reads, while a spinning disk requires physical seeking.

14.3 Write Amplification

Our chunking adds write amplification: writing a 64 KiB chunk requires writing the chunk key (18 bytes) + value (64 KiB) = 64 KiB + 18 bytes. The key overhead is <0.03%, negligible.

But FDB itself adds write amplification internally: each committed transaction is written to the Transaction Log (TLog), then asynchronously applied to Storage Servers. The TLog write is sequential (fast). The Storage Server write is to FDB’s B-tree (with its own write amplification). FDB’s overall write amplification is roughly 3–5x — comparable to RocksDB’s LSM write amplification.

14.4 Monitoring

Key metrics for a fdbstorage-backed LevelDB deployment:

• FDB transaction latency P99: should be < 10ms for small transactions (meta reads)
• FDB range scan bytes/second: correlates with compaction throughput
• FDB conflict rate: if high, indicates concurrent compaction and write contention
• LevelDB metrics via db.GetProperty("leveldb.stats"): still valid — LevelDB reports its own view of compaction and SSTable counts, just the “disk I/O” is actually FDB I/O

15. Interview Questions — Storage Abstractions and LSM Trees

Q: What is the purpose of the Rename operation in LevelDB’s storage interface, and how does your FDB implementation preserve its atomicity guarantee?

Rename is LevelDB’s way of atomically promoting a new SSTable (or MANIFEST) into production. During compaction, LevelDB writes the new SSTable to a temp file, then renames it to its final name. POSIX rename is atomic: either the old name or the new name is visible, never a half-written file. Our FDB implementation reads all chunks with the old file descriptor, writes them with the new file descriptor, and clears the old keys — all in one FDB transaction. FDB’s transaction atomicity provides the same guarantee: either the old keys or the new keys are visible, never both or neither.

Q: Why does LevelDB use a Write-Ahead Log, and is it still necessary when using FDB as the storage backend?

The WAL protects against crash scenarios where data was written to the in-memory MemTable but not yet flushed to an SSTable on disk. Without a WAL, a crash after the MemTable write but before the SSTable flush would lose those writes. With FDB as storage, our writer.Close() writes chunks to FDB in transactions. Each committed FDB transaction is durable (replicated to at least two machines). A crash after Close() returns has no data loss. The WAL’s durability purpose is already provided by FDB. A production implementation would use a no-op WAL to skip the overhead.

Q: What is the 10 MB transaction limit in FDB, and what design patterns avoid hitting it?

FDB limits the total read + write size per transaction to approximately 10 MB to bound the memory required on Commit Proxies and to keep transaction resolution fast. Patterns to stay within the limit: (1) chunk large values (as we do with 64 KiB chunks), (2) break bulk writes into multiple transactions with cursor-based pagination, (3) configure LevelDB’s compaction to keep SSTable sizes small (< 2 MB), (4) use FDB atomic operations (tr.Add, tr.SetVersionstampedKey) where possible — atomic operations don’t count against the read portion of the limit.

Q: How would you extend this implementation to support multiple concurrent LevelDB instances sharing the same FDB cluster?

Give each LevelDB instance its own ns prefix. The FDB key space is naturally partitioned: ns1 + ... keys and ns2 + ... keys are completely disjoint. Multiple instances can read and write concurrently with no coordination overhead — FDB’s conflict detection only fires when two transactions write the same key, and different namespaces use different keys. The lock key (ns + tagLock) is also per-namespace, so locking one instance doesn’t affect others.

16. Bugs Encountered and Lessons Learned

Implementing storage.Storage from scratch surfaces several non-obvious contracts that the goleveldb source doesn’t make obvious. These are real bugs found while getting the demo to work end-to-end.

Bug 1 — storage.ErrNotExist Does Not Exist

Symptom:

fdbstorage/storage.go:265:27: undefined: storage.ErrNotExist

What happened:

The initial implementation tried to use storage.ErrNotExist (by analogy with os.ErrNotExist), assuming goleveldb exported a sentinel error value for “file not found”. It doesn’t — goleveldb v1.0.0 exports no such symbol.

The contract (from the goleveldb source comments):

// Open opens file with the given 'file descriptor' read-only.
// Returns os.ErrNotExist error if the file does not exist.
Open(fd FileDesc) (Reader, error)

// GetMeta returns 'file descriptor' stored in meta.
// Returns os.ErrNotExist if meta doesn't store any 'file descriptor'.
GetMeta() (FileDesc, error)

The interface contract is written in English comments, not in types. Both Open and GetMeta must signal absence with os.ErrNotExist — the standard library sentinel, not a goleveldb one.

Fix: Use os.ErrNotExist directly everywhere the storage contract requires “not found”.

Bug 2 — GetMeta Returning a Custom Error on Empty Namespace

Symptom:

fdbstorage: no manifest set
exit status 1

What happened:

GetMeta read the manifest key from FDB. When the namespace was empty (first run, no DB created yet), the key was nil. The original code returned:

return storage.FileDesc{}, errors.New("fdbstorage: no manifest set")

goleveldb’s Open distinguishes two cases after calling s.recover():

err = s.recover()
if err != nil {
    if !os.IsNotExist(err) || s.o.GetErrorIfMissing() {
        return  // real error → abort
    }
    err = s.create()  // not-exist → create a fresh DB ← this is what we want
}

The custom error doesn’t satisfy os.IsNotExist, so goleveldb took the “real error → abort” path and surfaced the message to the user.

Fix:

if b == nil {
    return storage.FileDesc{}, os.ErrNotExist
}

Return the standard sentinel. goleveldb then calls s.create() and initialises a fresh database.

Bug 3 — List() Always Returned Empty

Symptom:

First session wrote three keys successfully. Second session opened the DB without error, but db.Get("apple") returned leveldb: not found.

What happened:

The List function is supposed to enumerate all files in the namespace. goleveldb calls List(TypeJournal) during recovery to find WAL files that need to be replayed. If the list is empty, no WAL is replayed — the memtable stays empty — and all keys written in the previous session appear missing.

The bug was subtle. List always passed storage.TypeAll to the internal range helper regardless of the ft argument:

func (s *Storage) List(ft storage.FileType) ([]storage.FileDesc, error) {
    v, err := s.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        // BUG: always uses TypeAll as the range prefix byte
        return rt.GetRange(s.metaRangeForType(storage.TypeAll), ...).GetSliceWithError()
    })

storage.TypeAll is a bitmask (TypeManifest | TypeJournal | TypeTable | TypeTemp = 1 | 2 | 4 | 8 = 0x0F). It is not a real file type. The meta keys in FDB use individual type bytes (0x01, 0x02, 0x04, 0x08). The range starting at <ns> 0x01 0x0F matched nothing because 0x0F is larger than all real type bytes.

The in-loop filter ft & thisFT == 0 was correct; only the FDB range prefix was wrong.

Fix: Scan all meta keys (from <ns> tagFileMeta to <ns> tagFileMeta+1) and let the existing per-item ft filter handle selection:

allMeta := fdb.KeyRange{
    Begin: fdb.Key(append(append([]byte{}, s.ns...), tagFileMeta)),
    End:   fdb.Key(append(append([]byte{}, s.ns...), tagFileMeta+1)),
}

Lesson: A bitmask sentinel (TypeAll) and a concrete type byte are different things. Never use a bitmask as a key prefix component.

Bug 4 — FDB-Persisted Lock Survives Process Crashes

Symptom:

fdbstorage: namespace already locked
exit status 1

on every run after any run that didn’t exit cleanly.

What happened:

The original Lock() wrote a key to FDB. Unlock() (called by goleveldb inside db.Close()) cleared it. This worked as long as db.Close() ran. But log.Fatal / log.Fatalf calls os.Exit(1), which does not run deferred functions. The demo’s second session had:

db, err := leveldb.Open(stor, nil)
// ...
defer db.Close()   // ← registered but ...

v, err := db.Get([]byte(k), nil)
if err != nil {
    log.Fatalf(...)  // ← os.Exit kills the process here; defer never runs
}

goleveldb stores the locker returned by Lock() and calls locker.Unlock() inside db.Close(). When os.Exit fires, db.Close() is never called, the unlock never happens, and the FDB lock key persists until the next manual cleanup.

Two fixes applied:

1. Switch to an in-process lock — replace the FDB key with a sync.Mutex flag on the Storage struct. An in-process flag resets automatically every time the process starts; stale state from a crashed run is impossible. The trade-off is losing cross-process exclusion, which is acceptable for a single-process demo.

2. Avoid log.Fatal after defer db.Close() — extract the code into a function that returns an error. Deferred cleanup runs normally when the function returns, even on the error path. The caller then does log.Fatal after cleanup is complete.

Lesson: log.Fatal, log.Fatalf, os.Exit, and runtime.Goexit all bypass defer. Any resource that must be released (DB handles, locks, network connections) should be closed by a return-based error path, not a fatal exit inside a function that registered defer.

Bug 5 — os.IsNotExist Ignores Custom .Is() Methods (Go 1.16+)

Symptom:

fdbstorage: file MANIFEST-000003 does not exist
exit status 1

after stale FDB state was left by a prior failed run.

What happened:

The initial fix for Bug 1 used a custom error type with an Is method:

type fileNotExistError struct{ fd storage.FileDesc }

func (e *fileNotExistError) Is(target error) bool {
    return target == os.ErrNotExist
}

errors.Is(err, os.ErrNotExist) correctly returns true for this type. However, goleveldb’s session.recover() uses os.IsNotExist(err), not errors.Is. In Go 1.25, os.IsNotExist is implemented as:

func underlyingErrorIs(err, target error) bool {
    err = underlyingError(err)  // unwraps *PathError, *LinkError, *SyscallError only
    if err == target { return true }
    e, ok := err.(syscallErrorType)
    return ok && e.Is(target)  // only calls .Is() on syscall errors
}

The comment in the Go source is explicit: underlyingErrorIs “only examines syscall errors” to preserve historical behaviour. A custom error type’s Is method is called by errors.Is but not by os.IsNotExist.

Because os.IsNotExist returned false for fileNotExistError, goleveldb surfaced the error directly instead of creating a fresh DB.

Fix: Use *os.PathError as the wrapper — os.IsNotExist explicitly unwraps *PathError in its underlyingError switch:

return nil, &os.PathError{Op: "open", Path: fd.String(), Err: os.ErrNotExist}

underlyingError extracts Err (which is os.ErrNotExist), the == target check passes, and os.IsNotExist returns true.

Lesson: errors.Is and os.IsNotExist are not equivalent in Go 1.13+. errors.Is traverses the full chain calling .Is() methods. os.IsNotExist only inspects a fixed set of OS error wrapper types. Always use *os.PathError{Err: os.ErrNotExist} (not a custom Is method) when you need os.IsNotExist compatibility.

Bug 6 — Stale FDB State Across Runs

Root cause: Every bug above caused the demo to exit abnormally, leaving partial or inconsistent FDB state: a manifest pointer referencing a file whose data chunks were never written, or data from a session that was never cleanly closed.

Fix: Add Storage.Wipe() — a single range-clear of the entire namespace — and call it at demo startup. This makes each run deterministic. In a real system, Wipe would be replaced with a proper recovery procedure:

1. Check for a “rename-pending” key (Exercise 4 in this guide) and complete or roll back any in-flight rename.
2. Use goleveldb’s Recover function (instead of Open) which reads whatever SSTables it can find and rebuilds the MANIFEST from scratch.
3. Remove orphaned files (those in FDB but not referenced by the MANIFEST) via List + cross-reference with the MANIFEST’s live file set.

Option B — SQLite VFS substrate on FoundationDB

• What a SQLite VFS actually needs
• Key layout
• Where the transactional magic lives
• Hooking this into a real SQLite
• Running the demo
• What’s deliberately omitted

Pattern: SQLite (or any pager-style engine) with FDB as the disk. We implement the byte-range storage primitive a SQLite VFS sits on top of — fixed-size pages keyed by page number, atomic page-level updates — and demonstrate it through a small Go API. Wiring the C-level VFS hooks is mechanical glue that’s left for a follow-up.

What a SQLite VFS actually needs

SQLite’s pager talks to “the OS” through a thin C interface defined in vfs.h. Boiled down, a VFS file handle exposes:

SQLite always reads and writes in multiples of the page size (4096 by default) once it’s past the 100-byte file header, so storing one FDB KV per page is a natural fit and makes the pager-to-storage mapping 1:1.

Key layout

<ns> 0x00                            -> uint64 BE  file size in bytes
<ns> 0x01 <pageNum:uint64 BE>        -> 4096-byte page
<ns> 0x02                            -> lock holder name (or absent)

Where the transactional magic lives

The interesting method is WriteAt. For partial-page writes, we:

1. Issue tr.Get for every affected page.
2. Merge the existing page bytes with the new bytes.
3. tr.Set the resulting full page.
4. Update the file-size key if we grew.

All inside one FDB transaction. That gives SQLite a property it cannot get from a normal filesystem: multi-page writes are atomic. SQLite has elaborate journal/WAL machinery to recover from “we crashed halfway through updating pages 17, 18, and 19.” On this VFS that recovery code becomes dead — either all three pages flipped or none did.

In practice you’d still set PRAGMA journal_mode = MEMORY (so SQLite skips the rollback journal it doesn’t need) and rely on FDB’s transactional commit as the single durability point.

Hooking this into a real SQLite

There are two paths:

1. cgo + mattn/go-sqlite3 or zombiezen.com/go/sqlite: register a custom sqlite3_vfs whose xRead/xWrite/... thunks call into our pagestore.File methods via a CGO bridge. ~300 lines of glue.
2. modernc.org/sqlite: pure-Go SQLite. Its vfs subpackage exposes Register / VFS types. Same wiring, no CGO.

Either way the interesting code is what’s already here — the C/Go thunks add no further insight into how FDB serves as the storage tier.

Running the demo

cd option-b-sqlite
go mod tidy
go run ./demo -cluster ../fdb.cluster

Expected output:

After writing header (16 B): size=16
After 100-byte cross-page write at offset 4090: size=4190
Read 100 B back, all 'A'? true
Header preserved? "SQLite format 3\x00"
After truncate to 4096: size=4096
Lock contention as expected: pagestore: locked by "conn-A"
Lock handoff OK.

The cross-page write at offset 4090 is the key correctness test: it spans page 0 (bytes 4090–4095) and page 1 (bytes 4096–4189). The output above proves that:

• The pre-existing 16-byte header on page 0 was preserved during the read- modify-write merge.
• Both halves of the payload made it to disk.
• A subsequent Truncate(4096) cleared page 1 entirely.

What’s deliberately omitted

• WAL mode. Skipping it forces SQLite into rollback-journal mode, which emits ordinary writes only — exactly what our VFS supports. WAL would require a second “shared-memory” backing store (xShmMap) and is the single largest source of complexity in real VFS implementations.
• Multi-process locking. The lock key works for cooperating clients but has no liveness guarantee on holder crash. Production code would combine it with FDB watches and a heartbeat.
• The C bridge itself. See docs/README.md above for the wiring sketch.

Hitchhiker’s Guide — Option B: SQLite VFS over FoundationDB

• Table of Contents
• 1. SQLite’s Architecture — From SQL to Bytes
• 2. The Virtual File System (VFS) — SQLite’s Plugin Point
• 3. The Page Model — How SQLite Organizes Storage
• 4. Our pagestore — FDB as a Page Store
• 5. The Partial-Page Write Problem — And Its Atomic Solution
• 6. xSync is a No-Op — And Why That’s Correct
• 7. Locking — From POSIX Flock to FDB Keys
• 8. Journal Modes: DELETE, WAL, MEMORY
• 9. mvsqlite — The Production Version of This
• 10. Real-World Analogues
  • libSQL (Turso)
  • Litestream
  • LiteFS (Fly.io)
  • The Pattern’s Significance
• 11. Exercises
• 12. Source Code Deep Dive — pagestore/pagestore.go
  • The File Struct
  • Key Construction
  • ReadAt — Handling Partial-Page Reads
  • WriteAt — The Read-Modify-Write Pattern
  • pageRange — For Bulk Operations
• 13. Production Considerations
  • 13.1 Page Size Selection
  • 13.2 Hot Page Contention
  • 13.3 Crash Recovery Testing
• 14. Interview Questions — SQLite, VFS, and Page Models

The question this answers: “How does SQLite actually read and write its database file? Can we replace the file with FoundationDB?”

The deeper question: “What is a database page, how does the pager work, and why is the page model the right abstraction for a VFS?”

Table of Contents

1. SQLite’s Architecture — From SQL to Bytes
2. The Virtual File System (VFS) — SQLite’s Plugin Point
3. The Page Model — How SQLite Organizes Storage
4. Our pagestore — FDB as a Page Store
5. The Partial-Page Write Problem — And Its Atomic Solution
6. xSync is a No-Op — And Why That’s Correct
7. Locking — From POSIX Flock to FDB Keys
8. Journal Modes: DELETE, WAL, MEMORY
9. mvsqlite — The Production Version of This
10. Real-World Analogues: libSQL, Litestream, LiteFS
11. Exercises

1. SQLite’s Architecture — From SQL to Bytes

SQLite processes queries through seven layers:

SQL text
   ↓ Tokenizer + Parser
   Abstract Syntax Tree (AST)
   ↓ Code Generator
   Bytecode program (VDBE instructions)
   ↓ Virtual Database Engine (VDBE)
   B-tree operations (seek, insert, delete on B-tree pages)
   ↓ Pager
   Page cache: reads/writes logical page numbers
   ↓ OS Interface (VFS)
   File reads/writes at byte offsets
   ↓ Actual storage

We plug in at the VFS layer. Everything above (SQL parsing, the B-tree, the pager) runs as normal. The VFS is called for all I/O, and our implementation redirects those calls to FDB.

The key insight: SQLite doesn’t know or care whether the VFS talks to a local file, a network mount, or a distributed database. It only needs the VFS contract to be upheld.

2. The Virtual File System (VFS) — SQLite’s Plugin Point

The VFS is documented in SQLite’s C API. The key functions (C signatures simplified):

// File operations (called on each open file):
int xRead(file, buf, amount, offset);   // read `amount` bytes at `offset`
int xWrite(file, buf, amount, offset);  // write `amount` bytes at `offset`
int xTruncate(file, size);              // truncate to `size` bytes
int xSync(file, flags);                 // flush to durable storage
int xFileSize(file, *size);             // return current size
int xLock(file, locktype);             // acquire SHARED/EXCLUSIVE/etc
int xUnlock(file, locktype);           // release lock
int xCheckReservedLock(file, *result); // is anyone holding RESERVED lock?
int xShmMap(file, region, size, ...);  // map shared memory region (WAL mode)

Our Go pagestore implements the subset needed for non-WAL mode: ReadAt, WriteAt, Truncate, FileSize, Lock, Unlock.

Why a Go struct instead of a C VFS?

A real SQLite VFS requires implementing C structs (sqlite3_vfs, sqlite3_file) and using CGO to register them. This is correct but adds ~200 lines of glue code that would obscure the core concept. Our pagestore is a Go struct that exercises the exact same I/O patterns as SQLite would use. The demo calls ReadAt/WriteAt directly, simulating what the SQLite pager would call through the VFS.

3. The Page Model — How SQLite Organizes Storage

SQLite’s database file is divided into fixed-size pages (default 4096 bytes). Every structure in a SQLite database — B-tree interior nodes, B-tree leaf nodes, overflow pages, free-list pages — is exactly one page. The pager manages a cache of these pages and issues I/O to the VFS in page-aligned operations.

SQLite database file layout:
  Offset  0 – 4095:   Page 1 (database header + B-tree root)
  Offset  4096 – 8191:  Page 2
  Offset  8192 – 12287: Page 3
  ...
  Offset (N-1)*4096 – N*4096-1: Page N

Page 1 is special: its first 100 bytes are the database header:

Offset  0: "SQLite format 3\000" (16 bytes, magic string)
Offset 16: page size (2 bytes)
Offset 18: file format write version (1 byte)
Offset 19: file format read version (1 byte)
...
Offset 28: file change counter (4 bytes)
...
Offset 52: schema format number (4 bytes)
...

The partial-page write problem arises here: the header is 100 bytes, but SQLite might write just the header (100 bytes at offset 0), while the rest of page 1 (bytes 100–4095) contains B-tree data. An xWrite(file, header, 100, 0) call writes only 100 bytes — not the full page.

4. Our pagestore — FDB as a Page Store

We store pages as FDB keys:

const PageSize = 4096

// Page data:
//   ns + 0x01 + pageNum(uint64 big-endian)  →  4096 bytes

// Database size (in pages):
//   ns + 0x00  →  uint64 (big-endian)

// Exclusive lock:
//   ns + 0x02  →  "locked" (any non-empty value)

Page numbers:

SQLite page numbers are 1-based (page 1 is the first page). We store them as 0-padded big-endian uint64s:

Page 1:  ns + 0x01 + 0000000000000001
Page 2:  ns + 0x01 + 0000000000000002
Page 1000: ns + 0x01 + 00000000000003E8

Big-endian encoding ensures that a GetRange over ns+0x01+0 to ns+0x01+FFFFFFFFFFFFFFFF returns pages in ascending page-number order — useful for scanning the entire database.

Size tracking:

SQLite calls xFileSize to determine how many pages exist. We track this explicitly with a size key rather than scanning for the highest-numbered page key (which would be an O(N) scan). The size key is updated atomically in the same transaction as every WriteAt and Truncate.

5. The Partial-Page Write Problem — And Its Atomic Solution

This is the trickiest part of implementing a page-based storage engine.

SQLite’s VFS contract says:

xWrite(offset, amount, data):
  Write `amount` bytes starting at `offset`.
  `offset` and `amount` may be any values (not necessarily page-aligned).

Examples of partial-page writes:

• Write header (100 bytes at offset 0) to page 1
• Write a record that spans a 4-byte boundary at the end of a page
• Write a single integer (4 bytes at offset 12) to a page

We cannot simply map WriteAt(offset, data) to Set(pageKey(offset/4096), data) because that would overwrite the entire 4096-byte page with only the partial data — corrupting the rest of the page.

Our solution: read-modify-write inside one transaction.

func (p *PageStore) WriteAt(data []byte, off int64) (int, error) {
    firstPage := off / PageSize
    lastPage := (off + int64(len(data)) - 1) / PageSize

    _, err := p.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        for pageNum := firstPage; pageNum <= lastPage; pageNum++ {
            // 1. Read current page content (or zeros if it doesn't exist yet)
            existing, _ := tr.Get(p.pageKey(pageNum)).Get()
            page := make([]byte, PageSize)
            copy(page, existing)  // pad to 4096 with zeros if shorter

            // 2. Compute which bytes of `data` go into this page
            pageStart := pageNum * PageSize
            pageEnd := pageStart + PageSize
            writeStart := max(off, pageStart) - pageStart
            writeEnd := min(off+int64(len(data)), pageEnd) - pageStart
            dataStart := max(off, pageStart) - off
            dataEnd := min(off+int64(len(data)), pageEnd) - off

            // 3. Overlay the new bytes onto the existing page
            copy(page[writeStart:writeEnd], data[dataStart:dataEnd])

            // 4. Write the modified page back
            tr.Set(p.pageKey(pageNum), page)
        }
        // 5. Update size key if this write extends the file
        ...
        return nil, nil
    })
    return len(data), err
}

Steps 1–4 happen inside one FDB transaction. The read and the write are atomic: no concurrent writer can modify the page between our read and write. No partial update is visible to other transactions.

The atomicity guarantee:

If the process crashes after step 1 (read) but before step 4 (write), the transaction is never committed. FDB abandons it. The page is unchanged. No corruption.

If two concurrent writers both try to read-modify-write the same page, FDB’s conflict detection will cause one to retry. The second writer will re-read the page (now including the first writer’s change) and overlay its own data on top. Correct behavior, no coordination required.

6. xSync is a No-Op — And Why That’s Correct

SQLite calls xSync to tell the VFS: “make sure all previous writes are durable on physical storage before returning.” On a normal filesystem, this calls fsync() which flushes the OS page cache to the disk controller and waits for the disk to confirm durability.

Why do we implement it as a no-op?

FDB’s Transact is synchronous and durable.

When WriteAt calls p.db.Transact(...) and it returns without error, the data is already:

1. Written to FDB’s transaction log on f+1 machines (durably replicated)
2. Committed (visible to new transactions)

By the time WriteAt returns to the SQLite pager, the durability guarantee is already satisfied — stronger than fsync on a local disk.

xSync is a no-op because the work it would do has already been done during xWrite. There is nothing left to flush.

Analogy: Imagine a bank transfer. xWrite is “debit Alice, credit Bob, commit to ledger.” xSync is “make sure the ledger is durable.” If the ledger is already in a replicated distributed database, xSync has nothing to do.

7. Locking — From POSIX Flock to FDB Keys

SQLite uses file locking to coordinate concurrent access. The POSIX lock levels, in order of increasing exclusivity:

UNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE

• SHARED: reader has this. Many readers can hold SHARED simultaneously.
• RESERVED: writer intending to modify. One writer can hold RESERVED while readers continue holding SHARED.
• PENDING: writer waiting for all SHARED holders to release.
• EXCLUSIVE: sole writer, no readers. Required before writing pages.

A full VFS implementation would map these levels to FDB lock keys with appropriate semantics (e.g., a counter for SHARED, a flag for EXCLUSIVE).

Our implementation is simplified: we use one “exclusive lock” key. Setting it means exclusive ownership; clearing it means unlocked. This correctly implements single-writer semantics but doesn’t allow concurrent readers.

For multi-reader/single-writer, a production implementation would:

SHARED lock: counter key (increment on lock, decrement on unlock)
EXCLUSIVE lock: flag key
  → acquire EXCLUSIVE: check counter == 0, set flag key
  → acquire SHARED: check flag key is unset, increment counter

Both checks-and-sets would use FDB transactions to ensure atomicity.

8. Journal Modes: DELETE, WAL, MEMORY

SQLite has several journal modes, selected with PRAGMA journal_mode=MODE. The journal is SQLite’s crash recovery mechanism (distinct from FDB’s own recovery).

DELETE (default rollback journal): Before modifying a page, SQLite copies the original page content to a separate journal file. If a crash occurs during a write, SQLite replays the journal to restore the original page content.

With FDB, this is redundant: FDB’s transactions provide rollback for free. A partial WriteAt that crashes mid-transaction rolls back automatically. The journal file would be stored in FDB via our VFS, adding overhead for no benefit.

WAL (Write-Ahead Log): Instead of copying original pages before modification, SQLite appends new page versions to a WAL file. Readers check the WAL before reading the main database file. A “checkpoint” operation copies WAL pages back to the main file.

WAL mode requires xShmMap — shared memory for the WAL index. This is a memory-mapped file that multiple processes share to coordinate which WAL frames are valid. Implementing xShmMap in FDB would require:

1. A small FDB key range to store the WAL index state.
2. Synchronized access to that state via FDB transactions. This is complex but possible — mvsqlite does it.

MEMORY: SQLite uses an in-memory journal, not written to any file. Rollback is possible within a transaction but not after a crash.

For our FDB VFS, PRAGMA journal_mode=MEMORY is the right choice: SQLite won’t try to create journal files (which would trigger extra VFS calls), and crash recovery is handled by FDB. This is what our demo uses implicitly by not specifying a journal mode (we would need to open the database and run PRAGMA journal_mode=MEMORY before any writes).

9. mvsqlite — The Production Version of This

mvsqlite (by losfair, Rust) is the production-grade implementation of the same idea. Its architecture:

Namespace mapping: Each SQLite “database file” path is mapped to an FDB namespace prefix. Multiple processes can open the same “file” (namespace) simultaneously with MVCC isolation.

Page-level MVCC: mvsqlite keeps multiple versions of each page, similar to how PostgreSQL keeps multiple row versions. When a reader opens the database, it captures an FDB read version. Page reads are served from FDB at that version. New writes create new page versions. Old versions are kept until all readers that need them complete.

Our pagestore is a simplified, non-MVCC version: every read sees the latest page version. Adding page-level MVCC would require:

ns + 0x01 + pageNum + version  →  4096 bytes  (instead of just pageNum)

With a cleanup process that removes old versions when no readers hold them.

Write-ahead log in FDB: mvsqlite implements WAL mode by storing the WAL log pages in FDB itself:

ns + 0x03 + txnId + pageNum  →  4096 bytes  (uncommitted WAL pages)
ns + 0x04 + txnId           →  commit record

This allows SQLite’s WAL mode to work without xShmMap (the shared memory region) — instead, the WAL index state lives in FDB and is accessed via transactions.

10. Real-World Analogues

libSQL (Turso)

libSQL is a fork of SQLite that adds multi-tenancy and remote storage. Turso’s cloud product stores SQLite databases in a distributed object store (similar to our FDB approach). The architecture:

• Local replicas for low-latency reads
• A primary writes to the distributed store
• Followers pull changes from the store and apply them locally

Litestream

Litestream continuously replicates SQLite databases to S3 or other cloud storage by intercepting the WAL. It monitors the SQLite WAL file and copies new frames to cloud storage in near-real-time. Restore is done by downloading frames and applying them.

Litestream does NOT modify SQLite’s VFS — it runs as a separate process that monitors the WAL file. This is simpler but means it can’t provide multi-writer consistency (only one process can write to a SQLite database at a time).

LiteFS (Fly.io)

LiteFS mounts a FUSE filesystem that intercepts SQLite writes, replicates them to a primary node, and distributes to replicas. It does intercept at the filesystem level (FUSE = Filesystem in Userspace), which is essentially implementing the VFS pattern at the OS level.

The Pattern’s Significance

All of these tools (mvsqlite, libSQL, Litestream, LiteFS) are attacking the same problem: SQLite is an excellent embedded database, but it’s tied to a single file on a single machine. The solution in each case is to intercept the storage layer and redirect I/O to a distributed, replicated system.

Our pagestore is the minimal proof-of-concept for this idea: 150 lines of Go that demonstrate the core page-store primitives.

11. Exercises

Exercise 1 — Page Cache

Add an in-process LRU page cache:

type PageStore struct {
    ...
    cache *lru.Cache  // maps pageNum → [PageSize]byte
}

ReadAt checks the cache before going to FDB. WriteAt updates the cache after writing to FDB. Evict on Truncate.

Measure the cache hit rate for a typical SQLite workload (a mix of reads and writes). You’ll find that B-tree root pages have very high hit rates — they’re accessed on every query.

Exercise 2 — Implement xShmMap for WAL Mode

WAL mode requires shared memory for the WAL index. In mvsqlite, this is done with FDB keys. Implement a simplified version:

1. Add a “WAL region” subspace: ns + 0x03 + regionNum → 32768 bytes
2. xShmMap(region, size) reads the FDB key and returns a []byte
3. xShmBarrier() writes the modified byte slice back to FDB
4. xShmLock maps to a FDB lock key per-region

With this, enable PRAGMA journal_mode=WAL in the demo and verify reads and writes still work.

Exercise 3 — Multi-Version Pages

Change the page key layout to:

ns + 0x01 + pageNum + readVersion  →  4096 bytes

WriteAt reads the current version, writes a new version at the current FDB commit version. ReadAt scans backwards from the requested read version to find the most recent page version ≤ that version.

Add a Compact(olderThan version) that clears all page versions older than a given version. This is the MVCC GC mechanism.

Exercise 4 — Register as a Real SQLite VFS

Using CGO, implement the actual sqlite3_vfs and sqlite3_file C structs backed by our pagestore. Register the VFS with sqlite3_vfs_register. Open a real SQLite connection using this VFS:

db, err := sql.Open("sqlite3", "file:demo.db?vfs=fdbvfs")
db.Exec("CREATE TABLE t (id INTEGER PRIMARY KEY, val TEXT)")
db.Exec("INSERT INTO t VALUES (1, 'hello')")

All I/O will go through pagestore to FDB. You have now replaced SQLite’s storage engine while keeping the full SQL query interface.

Exercise 5 — Multi-Database Isolation

The pagestore uses a namespace prefix to isolate one database. Add a DatabaseManager that:

1. Lists all databases (range scan over a well-known metadata prefix)
2. Creates a new database (allocates a namespace, writes a metadata entry)
3. Deletes a database (one ClearRange + metadata clear)
4. Returns a pagestore for a given database name

12. Source Code Deep Dive — pagestore/pagestore.go

The File Struct

type File struct {
    db   fdb.Database
    ns   []byte
    mu   sync.Mutex
}

ns is the namespace byte prefix. mu protects concurrent calls to WriteAt and Truncate from a single File instance (though FDB’s own transactions provide the real isolation for concurrent processes).

Key Construction

func (f *File) sizeKey() fdb.Key {
    key := make([]byte, len(f.ns)+1)
    copy(key, f.ns)
    key[len(f.ns)] = 0x00
    return fdb.Key(key)
}

func (f *File) pageKey(pageNum int64) fdb.Key {
    key := make([]byte, len(f.ns)+9)
    copy(key, f.ns)
    key[len(f.ns)] = 0x01
    binary.BigEndian.PutUint64(key[len(f.ns)+1:], uint64(pageNum))
    return fdb.Key(key)
}

sizeKey() stores the total file size in bytes. pageKey(n) stores the 4096-byte page at page number n. The big-endian 8-byte encoding of pageNum ensures that GetRange over sizeKey to the end of the 0x01 subspace returns pages in ascending page-number order.

ReadAt — Handling Partial-Page Reads

func (f *File) ReadAt(p []byte, off int64) (int, error) {
    firstPage := off / PageSize
    lastPage := (off + int64(len(p)) - 1) / PageSize

    futures := make([]fdb.FutureByteSlice, lastPage-firstPage+1)

    _, err := f.db.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
        for i := firstPage; i <= lastPage; i++ {
            futures[i-firstPage] = rt.Get(f.pageKey(i))
        }
        return nil, nil
    })
    // Wait for all futures and assemble the result
    for i, fut := range futures {
        pageData, _ := fut.Get()
        // copy relevant bytes from page into p...
    }
}

Pipelining multiple page reads: All rt.Get(pageKey(i)) calls are issued inside one ReadTransact closure. FDB sends all read requests to the storage servers before waiting for any response. If the read spans 3 pages (common for reads that cross page boundaries), all 3 are fetched in one round-trip. Without pipelining, each page would require a separate round-trip — 3x the latency.

This is the same pipelining pattern used in option-c-record-layer’s LookupByIndex.

WriteAt — The Read-Modify-Write Pattern

func (f *File) WriteAt(p []byte, off int64) (int, error) {
    f.mu.Lock()
    defer f.mu.Unlock()

    firstPage := off / PageSize
    lastPage := (off + int64(len(p)) - 1) / PageSize

    _, err := f.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        // Issue all reads first (pipelined)
        futures := make([]fdb.FutureByteSlice, lastPage-firstPage+1)
        for i := firstPage; i <= lastPage; i++ {
            futures[i-firstPage] = tr.Get(f.pageKey(i))
        }
        sizeF := tr.Get(f.sizeKey())

        // Wait for reads and apply writes
        for i := firstPage; i <= lastPage; i++ {
            existing, _ := futures[i-firstPage].Get()
            page := make([]byte, PageSize)
            copy(page, existing)
            // overlay p bytes onto page...
            tr.Set(f.pageKey(i), page)
        }

        // Update size
        curSize := int64(binary.BigEndian.Uint64(sizeF.MustGet()))
        newEnd := off + int64(len(p))
        if newEnd > curSize {
            b := make([]byte, 8)
            binary.BigEndian.PutUint64(b, uint64(newEnd))
            tr.Set(f.sizeKey(), b)
        }

        return nil, nil
    })
    return len(p), err
}

The read futures are pipelined: all tr.Get() calls are issued before any .Get() is called to block. This means all page reads happen in parallel — one round-trip for up to N pages regardless of N.

The size update is in the same transaction: The page writes and the size update are atomic. If the process crashes mid-transaction, FDB abandons it: neither the page updates nor the size update are applied. The database remains in its previous state — consistent, readable, no corruption.

pageRange — For Bulk Operations

func (f *File) pageRange() fdb.KeyRange {
    begin := f.pageKey(0)
    end := make([]byte, len(f.ns)+1)
    copy(end, f.ns)
    end[len(f.ns)] = 0x02  // one past the page subspace tag (0x01)
    return fdb.KeyRange{Begin: begin, End: fdb.Key(end)}
}

Used by Truncate to clear all pages above the new size:

func (f *File) Truncate(size int64) error {
    newLastPage := size / PageSize
    _, err := f.db.Transact(func(tr fdb.Transaction) (interface{}, error) {
        // Clear all pages above newLastPage
        clearBegin := f.pageKey(newLastPage + 1)
        tr.ClearRange(fdb.KeyRange{Begin: clearBegin, End: f.pageRange().End})
        // Update size
        b := make([]byte, 8)
        binary.BigEndian.PutUint64(b, uint64(size))
        tr.Set(f.sizeKey(), b)
        return nil, nil
    })
    return err
}

One ClearRange atomically removes all pages above the new size. This is O(1) in FDB key operations (one range-clear instruction), even if there are thousands of pages to clear. This is vastly more efficient than deleting pages one by one.

13. Production Considerations

13.1 Page Size Selection

The default SQLite page size is 4096 bytes. You can change it with PRAGMA page_size=N (must be set before the first write). Larger pages reduce the number of B-tree levels and improve scan performance on large tables, but increase read amplification for small point lookups. For an FDB-backed store:

• 4096 bytes (default): 4096-byte FDB values. Small key overhead. Good for mixed workloads.
• 8192 bytes: 8192-byte values. Better for large sequential scans.
• 65536 bytes (maximum SQLite page size): 64 KB values. Below FDB’s 100 KB limit. Excellent for large sequential scans, but poor for point lookups.

For most use cases, 4096 is the right choice. Match it to your workload’s read/write unit.

13.2 Hot Page Contention

Page 1 contains the database header and the root page of the main B-tree. Every query touches page 1. Every transaction increments the “file change counter” in the header. This means every write transaction does a read-modify-write on page 1 — even if the query doesn’t touch any user data on page 1 (e.g., inserting into a deep table touches only leaf pages and page 1 for the change counter).

In a high-concurrency write workload, this is a hot key. FDB’s conflict detection will serialize all transactions that write to page 1 — reducing concurrency.

Solution (mvsqlite’s approach): Move the change counter out of the page and into a separate FDB key. Use FDB’s atomic add to increment it without a read-modify-write. Since atomic add is commutative, it doesn’t cause conflicts. The page itself (minus the counter) is written only when actual B-tree structure changes.

13.3 Crash Recovery Testing

With FDB’s transactional guarantees, the traditional SQLite recovery path (re-applying the rollback journal) is never needed. But it’s worth testing:

1. Write some data.
2. Kill the process mid-write (use kill -9 or os.Exit(1) inside a WriteAt).
3. Restart the process and open the database.
4. Verify the database is in a consistent state (either the write happened or it didn’t, no partial state).

FDB’s transaction semantics make this trivially correct, but testing it gives you confidence that your implementation upholds the contract.

14. Interview Questions — SQLite, VFS, and Page Models

Q: What is a database page and why do databases use a fixed page size?

A page is the fundamental unit of I/O and storage in a database. Fixed page sizes allow: (1) simple address calculation — page N starts at byte (N-1) * pageSize; (2) alignment with OS page sizes (4 KiB for virtual memory, matching SQLite’s default); (3) buffer pool management — a page cache stores exactly one page per slot. Variable-size records are packed into pages; a B-tree node is exactly one page. The fixed size ensures that a B-tree node read is always one I/O operation, not an I/O per record.

Q: What is the partial-page write problem, and how does your FDB implementation solve it?

SQLite’s VFS xWrite can write any byte range — not necessarily page-aligned. If a 100-byte write starts at offset 0, it overwrites bytes 0–99 of page 1, but bytes 100–4095 are unchanged. An implementation that naively maps this to Set(page1Key, 100_bytes) would corrupt page 1 (losing bytes 100–4095). Our solution: read the existing page, overlay the new bytes, write the full 4096-byte page back — in one FDB transaction. The transaction atomicity ensures that no other writer sees a partial page and that a crash during the operation leaves the page unchanged.

Q: Why is xSync a no-op in your VFS implementation?

xSync requests durability: “flush all previous writes to stable storage.” Our WriteAt implementation uses fdb.Transact which does not return until the write is committed and replicated to FDB’s transaction log. The commit is synchronous and durable by the time WriteAt returns. There is nothing left for xSync to flush. The durability guarantee of xSync is already satisfied by the end of WriteAt.

Q: How does SQLite locking work, and how would you implement a multi-reader single-writer lock in FDB?

SQLite uses 5 lock levels. For multi-reader single-writer: maintain a SHARED lock counter (number of active readers) and an EXCLUSIVE lock flag. Acquire SHARED: check exclusive flag is unset, increment counter (in one FDB transaction). Acquire EXCLUSIVE: check counter == 0, set exclusive flag (in one FDB transaction). Release SHARED: decrement counter. Release EXCLUSIVE: clear flag. All check-and-set operations are atomic in FDB transactions, so no intermediate state is observable. A would-be exclusive lock holder that sees counter > 0 must retry (wait for readers to finish).

This is the NamespaceManager concept in mvsqlite, and it’s how a multi-tenant SQLite service would work.

Option C — Record Layer over FoundationDB

• What problem this solves
• Files
• Key layout
• Why it’s transactional by construction
• Mapping LookupByIndex to FDB ops
• Running
• What this layer omits compared to fdb-record-layer

Pattern: “structured records + secondary indexes, native FDB” — the same idea Apple’s fdb-record-layer is built around, distilled to ~250 lines of Go so you can read the whole thing in one sitting.

What problem this solves

FDB’s wire-level API is ordered KV. Most applications want records (a name, a city, an email) plus queries like “find users in Paris”. A record layer sits in between:

• Records are serialized blobs stored under a primary key.
• Indexes are extra KV entries keyed by (field, value, pk) so range scans on the index subspace return matching primary keys.
• Both updates happen in one FDB transaction, so the index is never out of sync with the records — even under concurrent writers.

Files

recordlayer/
  encoding.go   Key layouts for records (tag 0x00) and indexes (tag 0x01)
  msgpack.go    Wrapper around github.com/vmihailenco/msgpack
  store.go      Open / PutRecord / GetRecord / DeleteRecord / ScanRecords
  index.go      LookupByIndex (range-scan index → batch-read records)
demo/main.go    Users keyed by id, indexed by city

Key layout

records:   <ns> 0x00 <schemaName> 0x00 <pk>                               -> msgpack(record)
indexes:   <ns> 0x01 <schemaName> 0x00 <fieldName> 0x00 <value> 0x00 <pk> -> ""

Splitting records and indexes into two top-level “tags” keeps each subspace range-scannable without touching the other. Integers are written as sign-bit-flipped big-endian (see encodeIndexValue) so negative values sort before positive in the index — a tiny but useful property when you want range queries like “age >= 18”.

Why it’s transactional by construction

PutRecord does, inside one db.Transact:

1. Read the previous version of the record (if any).
2. For each indexed field that changed or was removed, Clear the old index entry.
3. Set the new record bytes.
4. Set fresh index entries for the new field values.

Because FDB transactions are serializable, no other client can observe a state where the record was updated but its indexes were not. This is the single biggest reason to build atop FDB rather than a non-transactional KV.

Mapping LookupByIndex to FDB ops

db.ReadTransact(rt -> {
    idxKVs := rt.GetRange(indexPrefix(schema, field, value))   // ordered scan
    for each idxKV:
        pendings[i] = rt.Get(recordKey(schema, pk))            // pipelined
    for each pending: collect record
})

The pipeline trick (issuing all rt.Get calls before awaiting any) lets FDB overlap network round-trips. Latency ≈ slowest single read instead of sum-of-reads.

Running

1. Bring up FDB and bootstrap (see top-level README).
2. cd option-c-record-layer && go run ./demo -cluster ../fdb.cluster

Expected output:

All users (PK order):
  u1 -> map[city:Paris name:Alice]
  u2 -> map[city:Tokyo name:Bob]
  u3 -> map[city:Paris name:Carol]

Lookup city=Paris (via secondary index):
  u1 -> map[city:Paris name:Alice]
  u3 -> map[city:Paris name:Carol]

After moving Alice to Tokyo:
Lookup city=Paris:
  u3 -> map[city:Paris name:Carol]
Lookup city=Tokyo:
  u1 -> map[city:Tokyo name:Alice]
  u2 -> map[city:Tokyo name:Bob]

What this layer omits compared to fdb-record-layer

• Schema evolution / Protobuf descriptors.
• Index definitions like COUNT, MAX, SUM aggregates.
• Query planner over multiple indexes.
• Versioned records and meta-data subspace.

But the storage shape and atomicity story are exactly the same.

Hitchhiker’s Guide — Option C: Record Layer with Secondary Indexes

• Table of Contents
• 1. What the Record Layer Is
• 2. Records vs Indexes: Two Subspaces, One Truth
• 3. The Index Key Format — Anatomy of a Lookup Key
• 4. PutRecord — Read-Modify-Write as a Pattern
• 5. LookupByIndex — Pipelining Explained
• 6. The encodeInt64 Sign-Bit Trick (Derived from First Principles)
• 7. Index Consistency: The Core Guarantee
• 8. The Apple Story: fdb-record-layer and CloudKit
• 9. Real-World Analogues
  • MySQL InnoDB Secondary Indexes
  • PostgreSQL GIN / B-Tree Indexes
  • MongoDB Indexes
• 10. Exercises
• 11. Source Code Deep Dive
  • encoding.go — The Key Construction Functions
  • encodeIndexValue — Dispatching on Type
  • store.go — ScanRecords with Cursor Logic
  • index.go — LookupByIndex Pipelining in Detail
• 12. Production Considerations
  • 12.1 Write Amplification Under Index Updates
  • 12.2 Online Index Building
  • 12.3 N+1 Read Problem
• 13. Interview Questions — Secondary Indexes and Record Layers

The question this answers: “How does a database maintain secondary indexes so they always agree with the primary data — even under concurrent writes?”

The deeper question: “What does Apple actually do inside iCloud, and why did they choose FoundationDB as the substrate?”

Table of Contents

1. What the Record Layer Is
2. Records vs Indexes: Two Subspaces, One Truth
3. The Index Key Format — Anatomy of a Lookup Key
4. PutRecord — Read-Modify-Write as a Pattern
5. LookupByIndex — Pipelining Explained
6. The encodeInt64 Sign-Bit Trick (Derived from First Principles)
7. Index Consistency: The Core Guarantee
8. The Apple Story: fdb-record-layer and CloudKit
9. Real-World Analogues: MySQL, PostgreSQL, MongoDB
10. Exercises

1. What the Record Layer Is

The FoundationDB Record Layer is a Java library open-sourced by Apple in 2019. It provides a record-oriented storage layer above FDB with:

• Typed records using Protobuf descriptors
• Declarative indexes (define once; maintenance is automatic)
• A query planner that compiles predicates into index range scans
• Schema evolution without downtime
• Change feeds via versionstamped keys

Our implementation is a Go reimagining: records are map[string]any (instead of Protobuf messages), indexes are field names (instead of declarative index definitions), and there is no query planner — you call LookupByIndex directly. But every core concept is the same.

Why a “record layer” and not just “a KV store”?

The moment you have more than one way to find a record, you need an index. The moment you have an index, you have two copies of the same data in two different shapes. The moment you have two copies, you need a protocol for keeping them consistent. The record layer is that protocol.

2. Records vs Indexes: Two Subspaces, One Truth

Our key space uses two tag bytes to separate the two subspaces:

Records subspace:
  <ns> + 0x00 + <schemaName> + 0x00 + <pk>
  → value: msgpack(record)

Indexes subspace:
  <ns> + 0x01 + <schemaName> + 0x00 + <fieldName> + 0x00 + <encodedFieldValue> + 0x00 + <pk>
  → value: "" (empty — the key IS the data)

Example with namespace "demo", schema "user", user Alice (pk="alice") with field city="Paris":

Records:
  64 65 6d 6f 00 75 73 65 72 00 61 6c 69 63 65
  "demo" 00 "user" 00 "alice"
  → msgpack({id:"alice", name:"Alice", city:"Paris"})

Indexes (city field):
  64 65 6d 6f 01 75 73 65 72 00 63 69 74 79 00 50 61 72 69 73 00 61 6c 69 63 65
  "demo" 01 "user" 00 "city" 00 "Paris" 00 "alice"
  → "" (empty value)

Why the value is empty for indexes:

The index key encodes everything you need to find the record: which schema, which field, what value, and what the PK is. The value at an index key carries no additional information. This is identical to how MySQL’s InnoDB secondary indexes work: the B-tree leaf stores just the clustered PK, not the full row.

Why the subspace separation matters:

• A full table scan (read all records) = GetRange(ns+0x00+schema, ns+0x00+schema+0x01) — hits only records, not indexes.
• An index scan (find all users where city=‘Paris’) = GetRange(ns+0x01+schema+0x00+city+0x00+Paris, ns+0x01+schema+0x00+city+0x00+Paris+0x01) — hits only that index column/value, not other indexes or records.
• Clearing all indexes for a schema = ClearRange(ns+0x01+schema, ns+0x01+schema+0x01).

With one tag byte, ranges are perfectly clean. Without it, you’d need carefully chosen sentinels and you’d still risk collisions.

3. The Index Key Format — Anatomy of a Lookup Key

Let’s trace exactly what key is written for each user:

Alice (pk="alice", city="Paris"):
  index key = ns + 0x01 + "user" + 0x00 + "city" + 0x00 + "Paris" + 0x00 + "alice"

Bob (pk="bob", city="Tokyo"):
  index key = ns + 0x01 + "user" + 0x00 + "city" + 0x00 + "Tokyo" + 0x00 + "bob"

Carol (pk="carol", city="Paris"):
  index key = ns + 0x01 + "user" + 0x00 + "city" + 0x00 + "Paris" + 0x00 + "carol"

In FDB’s sorted key space, these appear as:

(sorted by bytes)
...
ns+0x01+"user"+0x00+"city"+0x00+"Paris"+0x00+"alice"  → ""
ns+0x01+"user"+0x00+"city"+0x00+"Paris"+0x00+"carol"  → ""
ns+0x01+"user"+0x00+"city"+0x00+"Tokyo"+0x00+"bob"    → ""
...

A GetRange for city="Paris" scans from ...+"Paris"+0x00 to ...+"Paris"+0x01, returning exactly two keys (Alice and Carol). The PKs ("alice", "carol") are embedded in the key suffix. This is the index scan.

The extractPkFromIndexKey function:

Given the raw FDB index key, we need to extract just the PK suffix. The PK starts after the last 0x00 byte:

func extractPkFromIndexKey(key fdb.Key) []byte {
    for i := len(key) - 1; i >= 0; i-- {
        if key[i] == 0x00 {
            return key[i+1:]
        }
    }
    return nil
}

We scan backwards because the PK itself might contain 0x00 bytes if it was a binary key. Scanning backwards from the end finds the last 0x00, which is the separator between the encoded field value and the PK. This is safe because the encoded field value ends before that last 0x00, and the PK follows.

A subtle limitation: if the PK itself can contain 0x00, this parsing is ambiguous. A production implementation uses the Tuple encoding, which escapes 0x00 bytes within strings (\x00\xff) so the separator \x00\x00 is unambiguous.

4. PutRecord — Read-Modify-Write as a Pattern

This is the most important function in the record layer. Let’s walk through it completely:

func (s *Store) PutRecord(schema Schema, pk string, rec Record) error {
    _, err := s.fdb.Transact(func(tr fdb.Transaction) (interface{}, error) {
        // Step 1: Read the OLD record (if it exists)
        oldBytes, _ := tr.Get(recordKey(s.ns, schema.Name, pk)).Get()

        // Step 2: Decode the old record
        var oldRec Record
        if len(oldBytes) > 0 {
            msgpack.Unmarshal(oldBytes, &oldRec)
        }

        // Step 3: Remove OLD index entries
        for _, field := range schema.Indexes {
            if oldVal, ok := oldRec[field]; ok {
                tr.Clear(indexKey(s.ns, schema.Name, field, oldVal, pk))
            }
        }

        // Step 4: Write the NEW record
        encoded, _ := msgpack.Marshal(rec)
        tr.Set(recordKey(s.ns, schema.Name, pk), encoded)

        // Step 5: Write NEW index entries
        for _, field := range schema.Indexes {
            if newVal, ok := rec[field]; ok {
                tr.Set(indexKey(s.ns, schema.Name, field, newVal, pk), []byte{})
            }
        }
        return nil, nil
    })
    return err
}

Why must you read the old record before writing the new one?

Consider Alice moving from Paris to Tokyo:

Before: {id:"alice", name:"Alice", city:"Paris"}
After:  {id:"alice", name:"Alice", city:"Tokyo"}

We need to:

1. Remove the old index entry for city=Paris, pk=alice
2. Add a new index entry for city=Tokyo, pk=alice
3. Write the new record

We cannot do step 1 without knowing that Alice was previously in Paris. That information lives in the old record. So we read it first.

Why inside one transaction?

If we split this into two transactions:

Transaction 1: read old record, decide to clear Paris index
Transaction 2: clear Paris index, write new record, write Tokyo index

A crash between T1 and T2 leaves the Paris index entry pointing to a record that now says Tokyo. This is a phantom index entry — a stale pointer that silently returns wrong results.

With all steps in one transaction: either the entire operation commits (both index update and record update) or neither does. The index and the record always agree.

This is what database people mean when they say atomicity eliminates a class of bugs. It’s not just about performance; it’s about correctness.

The cost: one extra read per update.

Every PutRecord does one Get (the old record) before writing. This adds a read-version conflict key for the record’s current PK slot. If two writers try to update the same record concurrently, one will get a conflict and retry. This is acceptable and correct — the alternative (no-conflict update) risks interleaving index writes.

5. LookupByIndex — Pipelining Explained

This is where we use FDB’s pipelining to make multi-key lookups efficient.

The naive approach (one round-trip per record):

// BAD: N sequential round-trips for N results
pks := getIndexScan(...)  // one round-trip: get all PKs from index
for _, pk := range pks {
    rec := getRecord(pk)  // one round-trip each: N round-trips total
    results = append(results, rec)
}

For N=100, this is 101 network round-trips. At 1ms per round-trip, that’s ~100ms for a simple lookup. Unacceptable.

The pipelined approach (two round-trips total):

// GOOD: pipeline all Get calls, then collect all results
_, err = s.fdb.ReadTransact(func(rt fdb.ReadTransaction) (interface{}, error) {
    // Phase 1: issue ALL Get calls. None block yet.
    futures := make([]fdb.FutureByteSlice, len(pks))
    for i, pk := range pks {
        futures[i] = rt.Get(recordKey(s.ns, schema.Name, pk))
        // rt.Get returns a Future immediately — it has NOT waited for a response.
    }
    // Phase 2: now block on each future in sequence.
    for i, fut := range futures {
        b, _ := fut.Get()  // blocks until this specific response arrives
        // But by the time we block on futures[1], the response for futures[1]
        // through futures[N-1] may already be in flight or arrived.
        ...
    }
})

Phase 1 sends N requests in rapid succession without waiting for any response. The FDB client library batches these into its network buffer and sends them. The storage servers process all N in parallel. By the time we call Get() on the first future, many responses have already arrived.

Network diagram:

Without pipelining:        With pipelining:
  → Get(pk1)                → Get(pk1), Get(pk2), Get(pk3), ...  (burst)
  ← reply                   ← reply(pk1)
  → Get(pk2)                ← reply(pk2)
  ← reply                   ← reply(pk3)
  ...                       (collect all)
  Total: N round-trips      Total: ~1 round-trip worth of latency

Why this works in FDB’s model:

FDB transactions are snapshot reads. All Get calls within one ReadTransact closure see the same version of the database. There are no cross-key dependencies or ordering constraints on reads within a transaction. So the FDB client can send all reads in any order, and they will all return consistent (same read version) values.

This is the same principle as HTTP pipelining, REDIS pipelining, or database query batching — but with a guarantee that all reads are from the same snapshot.

6. The encodeInt64 Sign-Bit Trick (Derived from First Principles)

When indexing a numeric field (e.g., age: 25), we want the index to sort numerically. Bytes sort lexicographically. How do we make byte-sort match numeric sort for 64-bit signed integers?

Step 1: Start with what we have.

A 64-bit signed integer in two’s complement:

 0   = 0000 0000 0000 0000 0000 0000 0000 0000  (0x0000_0000_0000_0000)
 1   = 0000 0000 0000 0000 0000 0000 0000 0001  (0x0000_0000_0000_0001)
-1   = 1111 1111 1111 1111 1111 1111 1111 1111  (0xFFFF_FFFF_FFFF_FFFF)
-2   = 1111 1111 1111 1111 1111 1111 1111 1110  (0xFFFF_FFFF_FFFF_FFFE)
MIN  = 1000 0000 0000 0000 0000 0000 0000 0000  (0x8000_0000_0000_0000)
MAX  = 0111 1111 1111 1111 1111 1111 1111 1111  (0x7FFF_FFFF_FFFF_FFFF)

Big-endian byte-sort order: MIN, -2, -1, 0, 1, MAX (MIN starts with 0x80, which is 128, larger than 0x00 which is 0). So big-endian two’s complement does NOT sort correctly across the sign boundary.

Step 2: What would correct order look like?

We want the sorted sequence: …, -2, -1, 0, 1, …, MAX. In bytes, each element should be strictly less than the next. If we could assign distinct 64-bit unsigned values to each signed value such that the unsigned order matches the signed order, we’d be done.

Step 3: The offset encoding idea.

One option: add 2^63 to shift the range. Then:

-2^63 + 2^63 = 0
    -1 + 2^63 = 2^63 - 1
     0 + 2^63 = 2^63
     1 + 2^63 = 2^63 + 1
2^63-1 + 2^63 = 2^64 - 1

These are now unsigned values in the range [0, 2^64-1], in the same order as the original signed values. And they sort correctly in big-endian bytes!

Step 4: Addition vs. XOR.

Adding 2^63 is the same as flipping just the sign bit (bit 63). This is because adding 2^63 to a 64-bit value either sets bit 63 (if it was 0) or clears bit 63 and carries into bit 64 (which wraps around, but 2’s complement overflow is the same as XOR on individual bits).

More precisely: x + 2^63 = x XOR 2^63 for all x in the range [-2^63, 2^63-1]. This is because:

• If x ≥ 0: bit 63 of x is 0; adding 2^63 sets it to 1. XOR with 1<<63 also sets bit 63. Same result.
• If x < 0: bit 63 of x is 1; adding 2^63 causes a carry into bit 64, which is discarded in 64-bit arithmetic, leaving bit 63 as 0. XOR with 1<<63 also clears bit 63. Same result.

So the implementation is just:

func encodeInt64(x int64) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(x)^(1<<63))
    return b
}

Verification:

-2  → uint64(-2) = 0xFFFFFFFFFFFFFFFE → XOR 0x8000... → 0x7FFFFFFFFFFFFFFE
-1  → 0xFFFFFFFFFFFFFFFF → XOR → 0x7FFFFFFFFFFFFFFF
 0  → 0x0000000000000000 → XOR → 0x8000000000000000
 1  → 0x0000000000000001 → XOR → 0x8000000000000001
 2  → 0x0000000000000002 → XOR → 0x8000000000000002

Sorted by bytes: 0x7FFFFFFFFFFFFFE, 0x7FFFFFFFFFFFFFFF, 0x8000000000000000, 0x8000000000000001, 0x8000000000000002 — which is the same order as -2, -1, 0, 1, 2. Correct.

This is the standard encoding used by:

• FDB’s official Tuple layer (negative integers)
• Apache HBase row key encoding
• Google Cloud Bigtable ordered keys
• Apache Flink and Spark for sorted key encodings

7. Index Consistency: The Core Guarantee

Let’s state it precisely: at any point in time, for every record with field F=V and PK=P, there exists an index entry at (F, V, P). And there are no index entries without corresponding records.

Formally:

• forall records: indexEntries(record) ⊆ presentIndexKeys
• forall indexKey: correspondingRecord exists

Both directions. Our PutRecord maintains this invariant within one transaction:

1. Remove old index entries → breaks “no stale entries” for old value
2. Write new record → old record value is overwritten
3. Write new index entries → restores consistency

Between steps 1 and 3, within the transaction, the invariant is temporarily broken. But no other transaction sees this intermediate state — FDB’s isolation means only the final committed state is visible.

What happens if the invariant is violated?

A phantom index entry (index says Alice is in Paris, record says Tokyo):

• LookupByIndex(city="Paris") returns Alice
• Fetching Alice’s record shows city=Tokyo
• Application logic sees inconsistent data

This is precisely the class of bugs that a record layer is designed to prevent.

8. The Apple Story: fdb-record-layer and CloudKit

Apple’s CloudKit powers iCloud sync for iOS and macOS apps. At its peak it handles hundreds of millions of devices syncing calendar events, notes, photos metadata, and app-specific data. This is the production scale at which fdb-record-layer operates.

Why FDB?

Before FDB, Apple had a custom distributed database for CloudKit. It had the usual problems: hard to scale, hard to operate, hard to maintain consistency under failures. FDB was acquired in 2015. Over the next few years Apple rebuilt CloudKit’s storage engine on FDB.

Why a record layer on top?

FDB is a KV store. CloudKit stores typed records (calendar events have a start_date, end_date, title; notes have body, attachments). You cannot just say “find all events starting tomorrow” to an FDB cluster — you need to translate that query into a range scan over an encoded index.

fdb-record-layer provides:

1. Protobuf-typed records (schema validation, efficient encoding)
2. Value indexes, rank indexes, text indexes, aggregate indexes
3. A query planner that selects the best index for a given predicate
4. Change feed via versionstamped index keys (see GUIDE.md chapter on versionstamps)
5. Schema evolution (add fields with defaults, deprecate fields, change types safely)

Our implementation omits the Protobuf (replaced with map[string]any) and the query planner (you call LookupByIndex directly). But the index key layout and the PutRecord read-modify-write pattern are the same.

9. Real-World Analogues

MySQL InnoDB Secondary Indexes

In InnoDB, a secondary index B-tree contains:

(secondary_key_value, primary_key_value) → ""

Identical to our (field, encodedValue, pk) → "". To look up a row by a secondary key:

1. Range-scan the secondary index to get PKs.
2. Look up each PK in the clustered index (the primary B-tree).

This two-step process is called a “double-lookup” or “bookmark lookup” in MySQL docs. Our LookupByIndex does exactly this.

The reason InnoDB stores the PK (not the heap pointer) in secondary indexes: when a row is updated and moves in the clustered index, secondary indexes don’t need to be updated (they still point to the same PK value, which now lives in a different B-tree page but is found by the clustered index scan).

PostgreSQL GIN / B-Tree Indexes

PostgreSQL secondary indexes store (index_value → ctid) where ctid is the physical page+slot address. This is faster to look up (one step, not two) but requires updating ALL indexes when a row moves (e.g., after VACUUM).

PostgreSQL recently added “index-only scans” where the full row is stored in the index (similar to InnoDB’s covering indexes), trading space for read speed.

MongoDB Indexes

MongoDB maintains indexes as separate B-trees:

{ city: "Paris" } → [ ObjectId("alice"), ObjectId("carol") ]

MongoDB’s analog of LookupByIndex is called a “secondary index scan + fetch”. The MongoDB Document Layer (which stored MongoDB data in FDB) used exactly our two-subspace approach — one FDB range for documents, one for each index.

10. Exercises

Exercise 1 — COUNT Aggregate Index

Add a special “count” index that maintains a per-schema counter:

ns + 0x02 + schemaName + 0x00  →  msgpack(int64)

In PutRecord, if old record doesn’t exist, increment. In DeleteRecord, decrement. Both inside the same transaction as the record+index update.

This gives you COUNT(*) in O(1) instead of a full table scan.

Exercise 2 — Range Index Query

Add LookupByIndexRange(schema, field, minVal, maxVal any) ([]Record, error).

Since the index is sorted by encoded field value, you can do:

begin = indexPrefix(ns, schema, field) + encodeValue(minVal)
end   = indexPrefix(ns, schema, field) + encodeValue(maxVal) + 0x01
GetRange(begin, end)

This enables WHERE age >= 25 AND age < 35 as a single range scan. Make sure encodeValue produces sort-preserving bytes for your supported types (use encodeInt64 for integers).

Exercise 3 — Composite Indexes

Add support for multi-field indexes:

Schema{Name: "event", Indexes: [{"start", "end"}, {"city", "name"}]}