Performance — Latency, Throughput, Concurrency in Numbers

• 5.1 Latency — Where Every Millisecond Goes
  • Latency by operation, real measurements
• 5.2 Throughput — What a Cluster Can Sustain
  • Single-node (this repo’s setup)
  • Production-shape cluster (3 × c5.9xlarge, single AZ, double replication)
  • Scaling rules of thumb
• 5.3 Concurrency — How Many Transactions Can You Have In Flight?
  • Conflict rate as a function of concurrency
  • How to reason about concurrency in your layer
• 5.4 Transaction Volume — Sizing the Cluster
• 5.5 I/O — Where the Bytes Go
• 5.6 Reproducing the Numbers in This Repo
• 5.7 The Performance Mental Model
• Interview Questions

“FDB is fast” is not engineering. “A 3-node FDB cluster on c5.4xlarge with Redwood sustains 55,000 cross-shard commits/sec at p99 = 7.8 ms” is engineering.

This chapter gives you concrete numbers, the workloads that produced them, and the model that lets you predict performance for your own workload before you measure it.

All numbers below are from published benchmarks (FoundationDB Summit 2021, Snowflake’s 2021 Engineering Blog, the official 7.0 release notes, and our own reproduction on the labs in this repo). Treat them as starting points, not contracts.

5.1 Latency — Where Every Millisecond Goes

Anatomy of a single read in a healthy local-datacenter cluster:

Client process              GRV Proxy          Storage Server
     │                          │                     │
     │── GetReadVersion ───────►│                     │           0.3 ms (RTT/2)
     │                          │ batch with peers    │           0.1 ms
     │                          │ assign version v    │           0.0 ms
     │◄── version=v ────────────│                     │           0.3 ms
     │── readValue(key)@v ─────────────────────────►  │           0.3 ms
     │                                                │  routing  0.0 ms (cached)
     │                                                │  MVCC     0.05 ms
     │                                                │  storage  0.1–10 ms ← block cache vs disk
     │◄── value ─────────────────────────────────────  │          0.3 ms
     │
                                                       Total:  ≈ 1.5 ms (cache hit)
                                                       Total:  ≈ 6 ms   (disk read)

Anatomy of a single commit (small transaction, single shard):

Client → Commit Proxy            0.3 ms
Commit Proxy → Resolvers         0.3 ms          ↘
Resolver conflict check          0.05 ms          ├ done in parallel
Resolvers → Commit Proxy         0.3 ms          ↙
Commit Proxy → TLogs             0.3 ms
TLog fsync + ack                 1–4 ms     ← dominant cost; SSD/NVMe dependent
TLogs → Commit Proxy             0.3 ms
Commit Proxy → Client            0.3 ms

Total: ≈ 3–6 ms (p50)   ≈ 8–15 ms (p99 on commodity NVMe)

The TLog fsync is the dominant cost in a commit. This is why FDB clusters that need low write latency are run on machines with battery-backed write caches or Intel Optane, where fsync is sub-millisecond.

Latency by operation, real measurements

Measured on the cluster bootstrap’d by this repo’s docker-compose.yml (single-machine, single process, Redwood engine, on a 2023 MacBook Pro M2):

Multi-node production clusters add 1–3 ms per round trip (cross-rack) or 30–100 ms (cross-region). FDB is firmly a single-region product — cross- region replication is supported but adds the round-trip latency to every commit.

5.2 Throughput — What a Cluster Can Sustain

Single-node (this repo’s setup)

Production-shape cluster (3 × c5.9xlarge, single AZ, double replication)

Published by the FDB Summit 2021 talk “FoundationDB at Snowflake”:

Scaling rules of thumb

• Read throughput scales linearly with Storage Server count, until you saturate either the network or the GRV Proxy.
• Commit throughput scales sub-linearly with TLog count. Each TLog handles ~50–80 K commits/s. Adding TLogs helps until the Commit Proxy becomes the bottleneck (~250 K commits/s per Commit Proxy).
• Adding Commit Proxies shards commit traffic by key range and scales almost linearly until the Resolver becomes saturated. A Resolver handles ~500 K conflict-resolution ops/s.

If your workload exceeds these single-process limits, the answer is add more of the bottleneck process, not a bigger machine. FDB scales out, not up.

5.3 Concurrency — How Many Transactions Can You Have In Flight?

The fundamental concurrency parameter is the MVCC window — the number of versions kept readable. Default ≈ 5 seconds × ~1 M versions/sec = 5 M.

This sets:

• Maximum concurrent transactions. Each transaction holds a read version. With ≈ 1 M versions/sec issued by GRV Proxies, holding 5 M versions means you can have about 5 M in-flight transactions — though in practice the per-Resolver memory limit binds first.
• Maximum transaction duration. Same 5 seconds, end-to-end.
• Resolver memory. Holds recent write-key sets. With 250 K commits/s × 5 s = 1.25 M commits, each with ~10 keys, that’s 12.5 M keys × ~64 B per key = 800 MB. Hence the 1 GB RESOLVER_STATE_MEMORY_LIMIT knob.

Conflict rate as a function of concurrency

The classic OCC formula: if N transactions run concurrently, each touching k keys uniformly at random from a key space of size M, the per-transaction conflict probability is approximately:

$$ P_{\text{conflict}} \approx 1 - \left(1 - \frac{k}{M}\right)^{N \cdot k} $$

For k = 10, M = 1,000,000, the conflict rate by N:

This is why hot key contention dominates real-world FDB performance. A hot key effectively shrinks M to 1 for those transactions, and the conflict rate goes to ~100%. The fix is almost always sharding by client ID or time, transforming (counter) into (counter, clientID % 64) and summing on read.

How to reason about concurrency in your layer

Two questions to answer for every layer you build:

1. What is the largest natural key prefix shared by transactions? If two transactions both write user:42:*, they conflict on the implicit “user 42” read of the schema row. Split the schema reads from the data writes.
2. What is your atomic-op opportunity rate? Anywhere you do “read-modify-write” of a counter, replace it with Add(). This removes the read from the read-conflict set entirely — atomic ops never conflict.

5.4 Transaction Volume — Sizing the Cluster

Capacity planning starts with the bottleneck. For each FDB component:

Worked example. You need to sustain 100 K commits/s × 50 keys per commit (= 5 M sets/s), 500 K reads/s, with 10 TB total storage.

• Commit Proxies: 100 K / 250 K = 1; round up to 2 for HA. ✓
• TLogs: 100 K / 50 K = 2; round up to 4 for replication. ✓
• Resolvers: 100 K / 500 K = 1. ✓
• Storage Servers: 5 M sets/s ÷ (≈100 K writes/s per server) = 50. Cross-check: 10 TB ÷ (≈200 GB/server target) = 50. ✓ ✓
• Coordinators: 3 (or 5) for Paxos. Standard.

Total: 50 storage servers + 2 commit proxies + 1 GRV proxy + 1 resolver + 4 TLogs + 3 coordinators = 61 roles. Pack them onto ~20 machines (multiple roles per machine is fine, except don’t co-locate TLog and Storage Server unless you have separate disks for each — they will compete for fsync).

This calculation is exactly what the fdbcli> configure command lets you declare: configure new double ssd commit_proxies=2 grv_proxies=1 resolvers=1 logs=4.

5.5 I/O — Where the Bytes Go

For a single commit of B bytes of writes on a triple-replicated cluster with one Resolver:

This is much better than RocksDB’s 10–30× and slightly worse than naive in-place B-tree (which would be ≈ 4× but loses durability + concurrency). On a write- heavy workload pushing 100 MB/s of application writes, the cluster’s aggregate disk write rate is ≈ 600 MB/s — plan SSD endurance accordingly.

For reads with cold cache, every read is one B+tree path: 3–5 page reads of 4–32 KB each = 12–160 KB of disk reads per logical key, served from the Storage Server’s page cache when hot.

5.6 Reproducing the Numbers in This Repo

Use the lab clusters to measure your local equivalent:

# Spin up FDB
docker compose up -d
./scripts/bootstrap-fdb.sh

# Quick latency probe using fdbcli
docker exec -it $(docker ps -qf name=fdb) bash -lc \
  "echo -e 'option on TIMEOUT 5000\nget x' | fdbcli"

# Throughput: use option-a-leveldb's batch demo (writes 100k keys)
cd option-a-leveldb && time go run ./demo

To get production-shape numbers, run the official benchmark:

# Inside the FDB container:
fdbcli> configure new double ssd
mako --mode build --cluster_file /etc/foundationdb/fdb.cluster \
     --rows 1000000 --commits 50 --keys_per_xact 10
mako --mode run --cluster_file /etc/foundationdb/fdb.cluster \
     --rows 1000000 --threads 32 --duration 60 --tps 100000

mako is FoundationDB’s official load generator (in bindings/c/test/mako/). Its output gives per-operation latency histograms, conflict rate, retry rate, and aggregate throughput — the same toolchain the FDB team uses for releases.

5.7 The Performance Mental Model

Internalize these and you can ballpark almost any FDB workload:

1. Reads are cheap. ~1 ms p50, scale linearly with Storage Servers. If reads are your bottleneck, add Storage Servers.
2. Commits cost one fsync. ~3 ms p50, scale linearly with TLogs. If commits are your bottleneck, add TLogs (and check that you’re not hitting a Resolver / Commit Proxy ceiling first).
3. Conflicts are quadratic in concurrency. Halve hot-key access, halve conflict rate^2 (because both the writer and the reader’s odds drop).
4. Write amplification is ~5×. Plan SSD endurance and bandwidth from that, not from application MB/s.
5. The 5-second MVCC window bounds transaction duration. Long-running work must be chunked into bounded transactions.
6. There is no cross-region magic. Cross-region adds RTT to every commit. Use a single region with multi-DC zoning for HA, not for low latency.

Interview Questions

Q: A production FDB cluster has p50 commit latency of 4 ms but p99 of 80 ms. What are the likely causes?

The p99 outliers almost always trace to (a) TLog fsync stalls on a contended disk — check disk service times and whether TLog and Storage Server share disks; (b) conflict-retry cascades, where a contested hot key forces many transactions into multiple retries — check the cluster’s conflict_rate and retry_rate metrics; or (c) shard movement triggering brief read or commit queueing — check Data Distributor activity. The fact that p50 is good rules out a sustained capacity problem; it’s an episodic stall.

Q: How would you measure read amplification of a specific layer?

Run the layer with FDB’s per-transaction stats enabled (tr.GetEstimatedRangeSize(...) and tr.GetApproximateSize() give you read byte and write byte counts). Sum reads-per-user-request and compare to the size of the user-visible result. For an index-heavy schema the ratio is typically 2–4× (one read for the index, one for the row, plus schema lookups). If it’s > 10×, you have a missing index or schema-cache opportunity.

Q: A new shard split happens every few seconds during a bulk load. Why, and how do you mitigate?

Default shard target size is ~250 MB. A bulk load that writes contiguously fills shards quickly, and the Data Distributor splits them. Each split copies data to a new Storage Server, briefly degrading throughput. Mitigation: randomize the prefix of bulk-loaded keys (e.g., prefix with a hash) so writes spread across many existing shards instead of growing one shard linearly. Alternatively, pre-split: write sentinel keys to force the Data Distributor to pre-create shards at known boundaries before the bulk load begins.

Q: Your application sees consistent 500 K reads/s but commits cap at 40 K/s. Cluster has 1 Commit Proxy, 1 Resolver, 2 TLogs, 12 Storage Servers. What do you change?

40 K commits/s is roughly the per-TLog ceiling × 1 (the slower TLog is the bottleneck of replicated commits). Add 2 more TLogs (logs=4). If commits still don’t scale, the Commit Proxy is next (250 K/s ceiling, but it spends RPC bandwidth fanning out to TLogs, so the practical ceiling is lower); add a second Commit Proxy. Reads are fine because they bypass the commit path entirely and hit Storage Servers directly.

Q: You need to support a 30-second analytical scan over 100 GB of data. FDB’s MVCC window is 5 seconds. What’s the pattern?

Don’t. Break the scan into ~1-second chunks, each its own transaction, each resuming from the last key of the previous chunk (tr.GetRange(lastKey+\x00, end)). Accept that you no longer have a consistent snapshot across chunks — if that matters, snapshot the data to a side store (object storage) first and analyze the snapshot. For production-grade analytics over FDB data, the canonical pattern is to stream changes (via versionstamped change-feed keys) into a downstream columnar store like ClickHouse or BigQuery.