FDB Issue #4091 — Complete Contributor Guide

• Table of contents
• 1. Yes, FoundationDB is a C++ project — so why is the fix in Go?
  • The complete fix has two distinct layers
• 2. FDB Architecture crash course
  • 2a. The FDB transaction lifecycle
  • 2b. How the C layer actually stores transaction options
  • 2c. The Actor Framework (Flow)
• 3. The three persistent options and what “persistent” means
  • SetTimeout(ms) — option code 500
  • SetRetryLimit(n) — option code 501
  • SetMaxRetryDelay(ms) — option code 502
• 4. The bug — in precise terms
  • Why doesn’t the standard Transact helper have this problem?
• 5. Why would anyone use Reset() instead of OnError()?
• 6. Our Go-layer fix — how it works
  • PersistentOptions — a value type
  • SoftReset(tr, opts) — explicit intent
  • TransactWithPersistentOptions(db, opts, f) — the proper helper
• 7. The deeper C++ fix (Phase 2)
• 8. Files in this workspace
  • Running the demo
• 9. How to submit this as a PR
• 10. Concepts you learn from this issue
• 11. Environment setup
  • Quick-pick table
  • Option A — macOS, bare metal
  • Option B — Linux, bare metal
  • Option C — Docker, single node
  • Option D — Docker Compose, single node (this repo)
  • Option E — 3-node cluster, bare metal (macOS / Linux)
  • Option F — 3-node cluster, Docker Compose
  • Confirming your setup before proceeding
  • Clone the FDB repo (needed for PR work only)
• 12. Reproduce the bug
  • 12a. Minimal standalone repro
  • 12b. Full workspace demo
• 13. Apply the fix
  • 13a. Verify the fix compiles
  • 13b. Run with the fix active
  • 13c. What the fix does (summary)
• 14. Validate and verify
  • 14a. Unit tests — no FDB cluster needed
  • 14b. Integration tests — FDB cluster required
  • 14c. Behavioral verification with fdbcli
  • 14d. Run with the race detector
• 15. Pre-PR checklist
• 16. Building from source (optional — for the C++ phase)
  • 16a. Prerequisites
  • 16b. Configure and build
  • 16c. Running FDB’s own test suite (simulation)
• 17. Step-by-step PR guide
  • Step 1 — Fork and branch
  • Step 2 — Place the new file
  • Step 3 — Add tests
  • Step 4 — Run existing tests
  • Step 5 — Run the stack tester
  • Step 6 — Commit style
  • Step 7 — Open the PR
  • Step 8 — Respond to review
• 18. Understanding the FDB repository layout
  • The generated.go pipeline
  • Actor Framework (Flow) quick reference
• 19. Communication and community
• 20. Quick reference
  • FDB cluster management
  • Reproduce → apply fix → validate → PR

Issue: Support persistent transaction options for custom retry loops
Label: good first issue
Opened: Nov 2020 · Still unassigned as of May 2026

How to use this document
If you are completely new to FDB, read §1–5 first (architecture, the bug).
To reproduce the bug, jump to §12.
To run and verify the fix, go to §13–14.
To submit the PR, follow §15–18.
§20 is a quick-reference command card.

Table of contents

• 1. Yes, FoundationDB is a C++ project — so why is the fix in Go?
• 2. FDB Architecture crash course
• 3. The three persistent options and what “persistent” means
• 4. The bug — in precise terms
• 5. Why would anyone use Reset() instead of OnError()?
• 6. Our Go-layer fix — how it works
• 7. The deeper C++ fix (Phase 2)
• 8. Files in this workspace
• 9. How to submit this as a PR
• 10. Concepts you learn from this issue
• 11. Environment setup
  • Quick-pick table
  • Option A — macOS, bare metal
  • Option B — Linux, bare metal
  • Option C — Docker, single node
  • Option D — Docker Compose, single node (this repo)
  • Option E — 3-node cluster, bare metal
  • Option F — 3-node cluster, Docker Compose
• 12. Reproduce the bug
  • 12a. Minimal standalone repro
  • 12b. Full workspace demo
• 13. Apply the fix
• 14. Validate and verify
  • 14a. Unit tests — no FDB cluster needed
  • 14b. Integration tests — FDB cluster required
  • 14c. Behavioral verification with fdbcli
  • 14d. Run with the race detector
• 15. Pre-PR checklist
• 16. Building from source (optional — for the C++ phase)
• 17. Step-by-step PR guide
• 18. Understanding the FDB repository layout
• 19. Communication and community
• 20. Quick reference

1. Yes, FoundationDB is a C++ project — so why is the fix in Go?

FoundationDB’s core is C++. The storage engine, transaction coordinator, commit protocol, fault-tolerance logic, the simulation framework — all of it lives in C++ under fdbserver/, fdbclient/, and flow/.

But that C++ core exposes a thin C ABI (fdb_c.h / libfdb_c.dylib) that every language binding uses as its foundation:

┌──────────────────────────────────────────────────────┐
│                 FDB C++ Core                         │
│  fdbserver   fdbclient   flow   fdbrpc               │
└──────────────────────┬───────────────────────────────┘
                       │ C ABI  (libfdb_c.dylib)
                       │ fdb_transaction_set_option()
                       │ fdb_transaction_reset()
                       │ fdb_transaction_on_error()
                       │ ...
         ┌─────────────┼──────────────┐
         ▼             ▼              ▼
    Go bindings   Python bindings   Java bindings
    (CGo wrapper) (ctypes)          (JNI)

Issue #4091 lives at the boundary between the C ABI and the Go bindings. The issue was filed in the context of the Go API specifically — the words “custom retry loops” and “soft reset” are Go-user problems — but the same gap exists in every language binding because they all call the same C functions.

The complete fix has two distinct layers

This workspace implements the Go layer fix. The C layer fix is the next step once this is merged and the design is agreed upon with maintainers.

Starting with Go is correct for a first contribution because:

1. You can test it without re-compiling the C++ cluster — use any running FDB instance, including the pre-built package installed at /usr/local.
2. Binding-layer changes are reviewed by a larger set of maintainers (they affect the developer API, not internal correctness).
3. It unblocks users immediately while the deeper C change is designed.

2. FDB Architecture crash course

Before understanding the bug, you need to know how FDB handles transactions from a client’s point of view.

2a. The FDB transaction lifecycle

db.CreateTransaction()
        │
        ▼
  ┌───────────┐
  │  PENDING  │  ← reads, writes, Set options
  └─────┬─────┘
        │ tr.Commit()
        ▼
  ┌───────────┐
  │ COMMITTED │  ← success, done
  └───────────┘

  or on conflict / retryable error:

        │ tr.OnError(err)   OR   tr.Reset()
        ▼
  ┌───────────┐
  │  PENDING  │  ← back to start, try again
  └───────────┘

Two paths lead back to PENDING after a failure:

OnError(err)

• Calls fdb_transaction_on_error() in the C layer.
• The C layer decides if the error is retryable (e.g. not_committed, future_version) and sleeps for a back-off interval.
• The transaction’s read version, write set, and read/write conflict ranges are cleared — you get a fresh start on the next attempt.
• Critically: certain options are NOT cleared (see §3 below).

Reset()

• Calls fdb_transaction_reset() in the C layer.
• Equivalent to destroying the transaction object and creating a new one.
• All state is discarded — options, counters, everything.

2b. How the C layer actually stores transaction options

Inside NativeAPI.actor.cpp, every Transaction object holds a TransactionOptions struct. When you call fdb_transaction_set_option(), the C layer stores the value in that struct.

When fdb_transaction_on_error() is called, the implementation (simplified):

void Transaction::reset() {
    // Copies the *persistent* options to a temporary, clears everything,
    // then re-applies those options.
    savedOptions = tr->options.persistentOptions();  // timeout, retry_limit, max_retry_delay
    tr->fullReset();
    tr->options.applyPersistent(savedOptions);
}

When fdb_transaction_reset() is called:

void Transaction::fullReset() {
    // Clears absolutely everything, including options.
    tr->options = TransactionOptions();
}

This is the asymmetry at the heart of the bug.

2c. The Actor Framework (Flow)

You will see .actor.cpp files throughout the FDB codebase. FDB uses its own C++ coroutine system called Flow (predates C++20 coroutines). An ACTOR is a function that can wait() on futures without blocking a thread:

ACTOR Future<Void> myActor(Database db) {
    state Transaction tr(db);
    loop {
        try {
            wait(tr.commit());
            return Void();
        } catch (Error& e) {
            wait(tr.onError(e));  // back-off, then retry
        }
    }
}

When you see fdb_transaction_on_error() in the C header, it maps to a generated wrapper around an Actor like this. The wait() call is what produces the back-off delay (exponential back-off capped at max_retry_delay).

3. The three persistent options and what “persistent” means

Since FDB API version 610, three transaction options survive an OnError() call without being re-applied by the caller:

SetTimeout(ms) — option code 500

Sets a wall-clock budget for the entire logical operation (all attempts combined, not per attempt). The C layer records start_time when the option is first set; each OnError() call checks whether now - start_time > timeout and, if so, throws transaction_timed_out instead of retrying.

Because OnError does not clear this option, the clock keeps running across retries. A 10-second timeout means the whole thing (first attempt + all retries) must finish within 10 seconds.

SetRetryLimit(n) — option code 501

Sets a maximum number of calls to OnError(). The C layer keeps a counter that increments every time OnError() is called and succeeds (i.e., the error was retryable). When the counter reaches n, the next OnError() re-throws the most-recently-seen error instead of sleeping and continuing.

Because OnError does not clear this option OR its counter, a retry limit of 5 means at most 5 retries total, not 5 per-attempt.

SetMaxRetryDelay(ms) — option code 502

Caps the exponential back-off sleep inside OnError(). Not as critical as the other two, but it’s included in the persistence group for consistency.

4. The bug — in precise terms

Here is what goes wrong when you write a custom retry loop with Reset():

tr, _ := db.CreateTransaction()

for attempt := 0; ; attempt++ {
    // We want:  total timeout = 10 s, total retries ≤ 5
    // We get:   timeout = 10 s PER ATTEMPT, retries = 5 PER ATTEMPT
    tr.Options().SetTimeout(10_000)
    tr.Options().SetRetryLimit(5)

    doWork(tr)
    err := tr.Commit().Get()
    if err == nil { break }

    // fdb_transaction_reset() — ALL state discarded
    tr.Reset()
    // The loop goes back to the top, re-sets the options (fresh counters),
    // and the budget resets as if nothing happened.
}

On attempt 0: timeout clock starts, retry counter = 0. Reset() is called. Clock and counter are gone. On attempt 1: timeout clock starts fresh, retry counter = 0 again. …

The user wrote SetRetryLimit(5) intending “give up after 5 retries total”. They actually got “retry up to 5 times per attempt, indefinitely”.

Why doesn’t the standard Transact helper have this problem?

// From bindings/go/src/fdb/database.go
func (d Database) Transact(f func(Transaction) (interface{}, error)) (interface{}, error) {
    tr, _ := d.CreateTransaction()
    for {
        ret, e = f(tr)
        if e == nil {
            e = tr.Commit().Get()
        }
        if e == nil { return ret, nil }

        // Uses OnError, NOT Reset.
        // OnError preserves timeout + retry_limit in the C layer.
        e = tr.OnError(ferr).Get()
        if e != nil { return nil, e }
        // Loop continues — same transaction object, counters intact.
    }
}

Transact uses OnError which goes into the C layer, which specifically carries the persistent options forward. Reset() bypasses that logic entirely.

5. Why would anyone use Reset() instead of OnError()?

OnError() is the right tool for standard retry loops. But there are real scenarios where you want manual control:

Scenario A — Conditional retry logic

for {
    result, err := doWork(tr)
    if err != nil {
        if isRetryable(err) && shouldRetry(result) {
            tr.Reset()  // custom decision, not delegated to FDB
            continue
        }
        return err
    }
    ...
}

Scenario B — Two-phase reads before a write

for {
    snapshot := tr.Snapshot()   // cheaper reads, no conflict tracking
    data, _ := snapshot.GetRange(...).GetSliceWithError()

    // Decide what to write based on what we read.
    // If the computation is expensive, we may want to re-read from scratch
    // rather than propagating a stale read version through OnError.
    tr.Reset()
    doWrite(tr, data)
    err := tr.Commit().Get()
    if err == nil { break }
    ...
}

Scenario C — Integration with external retry libraries Some Go libraries (e.g. generic retry helpers) manage their own back-off and just want to re-use the transaction object without going through FDB’s back-off logic. They call Reset() and re-run the function.

In all these cases, the user loses their timeout / retry-limit semantics without any warning.

6. Our Go-layer fix — how it works

PersistentOptions — a value type

type PersistentOptions struct {
    timeout       *int64
    retryLimit    *int64
    maxRetryDelay *int64
}

Using pointer fields (rather than, say, -1 as a sentinel) means you can distinguish “not set” from “explicitly set to 0” (which disables the feature).

The builder pattern (WithTimeout, WithRetryLimit, WithMaxRetryDelay) returns a copy on each call, making PersistentOptions safe to pass by value and share across goroutines.

SoftReset(tr, opts) — explicit intent

func SoftReset(tr fdb.Transaction, opts PersistentOptions) error {
    tr.Reset()          // calls fdb_transaction_reset() in C
    return opts.Apply(tr)  // calls fdb_transaction_set_option() for each set field
}

This does NOT solve the “accumulated counter” problem — after Reset(), the C layer has no memory of elapsed time or previous retry count. Calling SetTimeout(10_000) after Reset() starts a fresh 10-second clock.

What it does solve:

• Prevents accidentally forgetting to re-apply options.
• Makes the intent explicit and auditable in code review.
• A single call site (SoftReset) is easier to audit than scattered SetTimeout / SetRetryLimit calls inside every retry loop.

TransactWithPersistentOptions(db, opts, f) — the proper helper

func TransactWithPersistentOptions(db fdb.Database, opts PersistentOptions,
    f func(fdb.Transaction) (interface{}, error)) (interface{}, error) {

    tr, _ := db.CreateTransaction()
    opts.Apply(tr)                    // set options before first attempt

    for {
        ret, e = f(tr)
        if e == nil { e = tr.Commit().Get() }
        if e == nil { return ret, nil }

        e = tr.OnError(ferr).Get()   // ← uses OnError, not Reset
        if e != nil { return nil, e }

        opts.Apply(tr)               // re-apply after OnError (redundant for
                                     // timeout/retry_limit at API >= 610, but
                                     // correct for max_retry_delay and future
                                     // options that may not yet be persistent)
    }
}

This uses OnError internally, so it gets the true C-layer persistence for free. The opts.Apply after OnError is technically redundant for timeout and retry_limit (the C layer already preserved them), but it is harmless and ensures the helper works correctly even if the user calls it at an older API version.

7. The deeper C++ fix (Phase 2)

The Go helper SoftReset resets the elapsed-time and retry counters because it calls fdb_transaction_reset() followed by fdb_transaction_set_option(). To preserve those counters, we need a new C API function:

// Proposed addition to fdb_c.h
DLLEXPORT void fdb_transaction_soft_reset(FDBTransaction* tr);

And its implementation in fdbclient/NativeAPI.actor.cpp:

void Transaction::softReset() {
    // Step 1: save the persistent options AND their accumulated state
    auto saved = options.snapshotPersistentWithCounters();
    // Step 2: full reset (clears read version, writes, conflicts, options)
    fullReset();
    // Step 3: restore the snapshot
    options.restorePersistent(saved);
}

This would let the Go binding expose:

func (t Transaction) SoftReset() {
    C.fdb_transaction_soft_reset(t.ptr)
}

And then SoftReset() would have the same semantics as OnError() for option persistence — accumulated elapsed time and retry count continue across the reset, exactly as users expect.

This is Phase 2 of the contribution:

1. Open a PR with the Go binding layer changes from this workspace. ✅ (this workspace)
2. After that merges and the API is agreed, implement fdb_transaction_soft_reset in C++ and update the Go binding to call it.

8. Files in this workspace

issue-4091-persistent-tx-options/
│
├── go.mod                   Go module, depends on the FDB Go bindings
│
├── fdb_retry/
│   └── retry.go             The actual proposal — PersistentOptions,
│                            SoftReset(), TransactWithPersistentOptions().
│                            In a real PR, this code would live in
│                            bindings/go/src/fdb/persistent.go
│
└── demo/
    └── main.go              Runnable experiment. Shows three retry styles
                             side-by-side with a contention-driven counter
                             increment so you can observe the difference.

Running the demo

# Make sure FDB_CLUSTER_FILE is exported (see §11 for all setup options)
fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"

cd issue-4091-persistent-tx-options
go run ./demo

Expected output shape:

FDB issue #4091 – persistent transaction options demo
=====================================================

BROKEN   (bare Reset, options silently dropped)           elapsed=12ms  counter=2
FIXED    (SoftReset, options explicitly reapplied)        elapsed=11ms  counter=2
STANDARD (TransactWithPersistentOptions helper)           elapsed=9ms   counter=2

All three produce the correct counter value (2 — one increment per goroutine). The timing difference between BROKEN/FIXED and STANDARD becomes visible when you lower the timeout to force a timeout error — STANDARD’s budget is total across all attempts while BROKEN’s resets per-attempt.

9. How to submit this as a PR

The FDB Go bindings live in a sub-module:

apple/foundationdb
└── bindings/
    └── go/
        ├── go.mod            ← separate Go module
        └── src/
            └── fdb/
                ├── transaction.go
                ├── database.go
                ├── generated.go
                └── persistent.go  ← new file (our retry.go content)

Steps:

1. Fork apple/foundationdb on GitHub.
2. Clone your fork.
3. Copy fdb_retry/retry.go content into bindings/go/src/fdb/persistent.go, change package fdb_retry to package fdb, and remove the fdb. prefix from all FDB type references.
4. Add a test file bindings/go/src/fdb/persistent_test.go.
5. Run the existing binding tests: cd bindings/go && go test ./src/fdb/...
6. Open the PR, reference #4091 in the description, and note that Phase 2 (the C++ fdb_transaction_soft_reset) is a follow-up.

10. Concepts you learn from this issue

11. Environment setup

Topology note for this issue: the bug lives entirely in the Go client library — Reset() discards options before they reach the server. A single FDB process is sufficient to reproduce it, apply the fix, and run all tests. Contention is generated by concurrent goroutines in the demo, not by cluster topology. A 3-node cluster gives you more realistic conflict rates but is not required.

Quick-pick table

All options end with the same environment variable pointing to a cluster file, so every command in §12–15 works identically regardless of which you chose.

Option A — macOS, bare metal

Install FDB (skip if already installed):

# Verify existing install
fdbcli --version   # FoundationDB CLI 7.x.x
ls /usr/local/lib/libfdb_c.dylib   # CGo link target

If not installed, download the macOS package from github.com/apple/foundationdb/releases and run the .pkg installer. The installer places binaries in /usr/local/bin/ and the library in /usr/local/lib/.

Install Go 1.21+:

brew install go    # or download from go.dev
go version         # go version go1.21+

Start a single-node cluster:

mkdir -p ~/fdb-data ~/.fdb

# Generate a cluster file (description:randomID@host:port)
echo "local:$(od -An -tx8 -N8 /dev/urandom | tr -d ' \n')@127.0.0.1:4500" \
  > ~/.fdb/fdb.cluster

fdbserver \
  -p 127.0.0.1:4500 \
  -C ~/.fdb/fdb.cluster \
  -d ~/fdb-data \
  -L /tmp/fdb-logs &

sleep 2
fdbcli -C ~/.fdb/fdb.cluster --exec "configure new single memory"

Export cluster file path (used by all subsequent steps):

export FDB_CLUSTER_FILE=~/.fdb/fdb.cluster

Verify:

fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"
# Output: The database is available.

Stop:

pkill fdbserver

Option B — Linux, bare metal

Install FDB:

# Debian / Ubuntu
FDB_VER=7.3.27
curl -LO "https://github.com/apple/foundationdb/releases/download/${FDB_VER}/foundationdb-clients_${FDB_VER}-1_amd64.deb"
curl -LO "https://github.com/apple/foundationdb/releases/download/${FDB_VER}/foundationdb-server_${FDB_VER}-1_amd64.deb"
sudo dpkg -i foundationdb-clients_*.deb foundationdb-server_*.deb

# RHEL / Fedora / Amazon Linux
FDB_VER=7.3.27
curl -LO "https://github.com/apple/foundationdb/releases/download/${FDB_VER}/foundationdb-clients-${FDB_VER}-1.el7.x86_64.rpm"
curl -LO "https://github.com/apple/foundationdb/releases/download/${FDB_VER}/foundationdb-server-${FDB_VER}-1.el7.x86_64.rpm"
sudo rpm -i foundationdb-clients-*.rpm foundationdb-server-*.rpm

The .deb/.rpm packages install and auto-start fdbserver via systemd, and write a cluster file to /etc/foundationdb/fdb.cluster. Check status:

sudo systemctl status foundationdb
fdbcli --exec "status minimal"   # uses /etc/foundationdb/fdb.cluster by default

Install Go 1.21+:

# Using the official tarball (adjust version as needed)
curl -LO https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Export cluster file path:

export FDB_CLUSTER_FILE=/etc/foundationdb/fdb.cluster

Stop:

sudo systemctl stop foundationdb

Option C — Docker, single node

No local FDB install needed. Requires Docker Desktop (macOS/Windows) or docker CLI (Linux).

# Pull and run a single FDB container
docker run -d \
  --name fdb \
  --network host \
  -e FDB_NETWORKING_MODE=host \
  -e FDB_COORDINATOR_PORT=4500 \
  -e FDB_PORT=4500 \
  -v "$PWD/fdb-data:/var/fdb/data" \
  foundationdb/foundationdb:7.3.27

# Wait for it to start, then initialise the database
sleep 3
docker exec fdb fdbcli --exec "configure new single memory"

# Copy the cluster file out so the host Go process can use it
docker exec fdb cat /etc/foundationdb/fdb.cluster > ./fdb.cluster

macOS / Docker Desktop: Docker --network host does not work on macOS (host networking is Linux-only). Replace --network host with -p 4500:4500 and update the cluster file’s address to 127.0.0.1:

# macOS variant
docker run -d --name fdb \
  -p 4500:4500 \
  -e FDB_NETWORKING_MODE=host \
  -v "$PWD/fdb-data:/var/fdb/data" \
  foundationdb/foundationdb:7.3.27
sleep 3
docker exec fdb fdbcli --exec "configure new single memory"
docker exec fdb cat /etc/foundationdb/fdb.cluster \
  | sed 's/@.*:/@127.0.0.1:/' > ./fdb.cluster

Export cluster file path:

export FDB_CLUSTER_FILE="$PWD/fdb.cluster"

Install the FDB client library on the host (CGo needs it to compile):

# macOS
brew install foundationdb   # or use the .pkg from the releases page

# Linux — client-only package (no server binary needed)
sudo dpkg -i foundationdb-clients_7.3.27-1_amd64.deb

Verify:

fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"

Stop:

docker stop fdb && docker rm fdb

Option D — Docker Compose, single node (this repo)

This is the fastest zero-config path. The repo already has a docker-compose.yml and a bootstrap script.

# From the repo root
docker compose up -d
bash scripts/bootstrap-fdb.sh   # waits for healthy, writes ./fdb.cluster

export FDB_CLUSTER_FILE="$PWD/fdb.cluster"

Verify:

docker exec fdb-layers fdbcli --exec "status minimal"
# The database is available.

Stop:

docker compose down

Option E — 3-node cluster, bare metal (macOS / Linux)

Run three fdbserver processes on the same machine, each on a different port. Use a shared cluster file that lists the coordinator.

mkdir -p ~/fdb-data/{n1,n2,n3} ~/.fdb /tmp/fdb-logs

# Write a cluster file pointing to node 1 as coordinator
echo "dev:$(od -An -tx8 -N8 /dev/urandom | tr -d ' \n')@127.0.0.1:4500" \
  > ~/.fdb/fdb.cluster

CFILE=~/.fdb/fdb.cluster

# Start three server processes
fdbserver -p 127.0.0.1:4500 -C "$CFILE" -d ~/fdb-data/n1 -L /tmp/fdb-logs &
fdbserver -p 127.0.0.1:4501 -C "$CFILE" -d ~/fdb-data/n2 -L /tmp/fdb-logs &
fdbserver -p 127.0.0.1:4502 -C "$CFILE" -d ~/fdb-data/n3 -L /tmp/fdb-logs &

sleep 3

# Configure as a triple-redundancy cluster (requires ≥ 3 processes)
fdbcli -C "$CFILE" --exec "configure new triple memory"

# Verify all three processes are visible
fdbcli -C "$CFILE" --exec "status"
# Look for: FoundationDB processes - 3

Export cluster file path:

export FDB_CLUSTER_FILE=~/.fdb/fdb.cluster

Stop:

pkill fdbserver

Option F — 3-node cluster, Docker Compose

Save this as docker-compose-cluster.yml in the repo root:

services:
  fdb1:
    image: foundationdb/foundationdb:7.3.27
    container_name: fdb-node1
    hostname: fdb1
    environment:
      FDB_NETWORKING_MODE: container
      FDB_COORDINATOR: fdb1
      FDB_COORDINATOR_PORT: "4500"
      FDB_PORT: "4500"
    ports:
      - "4500:4500"
    volumes:
      - ./fdb-data/n1:/var/fdb/data
      - ./fdb-config:/etc/foundationdb

  fdb2:
    image: foundationdb/foundationdb:7.3.27
    container_name: fdb-node2
    hostname: fdb2
    environment:
      FDB_NETWORKING_MODE: container
      FDB_COORDINATOR: fdb1
      FDB_COORDINATOR_PORT: "4500"
      FDB_PORT: "4500"
    volumes:
      - ./fdb-data/n2:/var/fdb/data
      - ./fdb-config:/etc/foundationdb
    depends_on: [fdb1]

  fdb3:
    image: foundationdb/foundationdb:7.3.27
    container_name: fdb-node3
    hostname: fdb3
    environment:
      FDB_NETWORKING_MODE: container
      FDB_COORDINATOR: fdb1
      FDB_COORDINATOR_PORT: "4500"
      FDB_PORT: "4500"
    volumes:
      - ./fdb-data/n3:/var/fdb/data
      - ./fdb-config:/etc/foundationdb
    depends_on: [fdb1]

mkdir -p fdb-data/{n1,n2,n3} fdb-config

docker compose -f docker-compose-cluster.yml up -d
sleep 5

# Init the database as a triple-redundancy cluster
docker exec fdb-node1 fdbcli --exec "configure new triple memory"

# Copy cluster file to the host (coordinator address is fdb1:4500 inside the
# Docker network; expose it via the mapped port 4500 on 127.0.0.1)
docker exec fdb-node1 cat /etc/foundationdb/fdb.cluster \
  | sed 's/fdb1/127.0.0.1/' > ./fdb.cluster

export FDB_CLUSTER_FILE="$PWD/fdb.cluster"

Verify:

fdbcli -C "$FDB_CLUSTER_FILE" --exec "status"
# FoundationDB processes - 3
# Database - available

Stop:

docker compose -f docker-compose-cluster.yml down

Confirming your setup before proceeding

Regardless of which option you used, run this check — every step in §12–15 depends on it passing:

# 1. Cluster is reachable
fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"
# Expected: The database is available.

# 2. Go toolchain works
go version
# Expected: go version go1.21+

# 3. CGo can link against libfdb_c
cd issue-4091-persistent-tx-options
go build ./fdb_retry/
# Expected: no output (success)

Clone the FDB repo (needed for PR work only)

The steps below do not require this. Clone only when you are ready for §15.

# Fork apple/foundationdb on GitHub first, then:
git clone git@github.com:YOUR_USERNAME/foundationdb.git ~/src/foundationdb
cd ~/src/foundationdb
git remote add upstream https://github.com/apple/foundationdb.git
git fetch upstream
cd bindings/go && go mod download

12. Reproduce the bug

These commands work with any option from §11. FDB_CLUSTER_FILE must be exported (or ~/.fdb/fdb.cluster / /etc/foundationdb/fdb.cluster must exist as the default).

12a. Minimal standalone repro

No workspace clone needed — paste and run in any temporary directory.

mkdir /tmp/fdb-repro && cd /tmp/fdb-repro
cat > go.mod << 'EOF'
module repro
go 1.21
require github.com/apple/foundationdb/bindings/go v0.0.0-20231107151356-57ccdb8fee6d
EOF
go mod download

cat > main.go << 'EOF'
package main

import (
    "fmt"
    "time"
    "github.com/apple/foundationdb/bindings/go/src/fdb"
)

func main() {
    fdb.MustAPIVersion(730)
    db, _ := fdb.OpenDefault()
    key := fdb.Key("repro:counter")

    // --- BROKEN: Reset() silently drops the retry_limit counter ---
    tr, _ := db.CreateTransaction()
    attempts := 0
    start := time.Now()
    for {
        attempts++
        tr.Options().SetRetryLimit(3) // intent: stop after 3 retries total
        tr.Get(key)
        tr.Set(key, []byte{byte(attempts)})
        err := tr.Commit().Get()
        if err == nil { break }
        if attempts > 20 {
            fmt.Printf("BROKEN:  gave up manually after %d attempts (%v) — SetRetryLimit ignored\n",
                attempts, time.Since(start))
            break
        }
        tr.Reset() // nukes the retry_limit counter; loop re-sets it to 0
    }

    // --- CORRECT: OnError preserves the counter in the C layer ---
    tr2, _ := db.CreateTransaction()
    tr2.Options().SetRetryLimit(3)
    attempts2 := 0
    start2 := time.Now()
    for {
        attempts2++
        tr2.Get(key)
        tr2.Set(key, []byte{byte(attempts2)})
        err := tr2.Commit().Get()
        if err == nil {
            fmt.Printf("CORRECT: committed after %d attempt(s) (%v)\n",
                attempts2, time.Since(start2))
            break
        }
        ferr, _ := err.(fdb.Error)
        if retryErr := tr2.OnError(ferr).Get(); retryErr != nil {
            fmt.Printf("CORRECT: stopped after %d attempt(s) as intended (%v): %v\n",
                attempts2, time.Since(start2), retryErr)
            break
        }
    }
}
EOF
go run .

Expected output:

BROKEN:  gave up manually after 21 attempts (38ms) — SetRetryLimit ignored
CORRECT: stopped after 4 attempt(s) as intended (5ms): transaction_too_old

The BROKEN path hits the safety valve (21 attempts) because every Reset() resets the retry counter to zero. The CORRECT path stops at 4 attempts (1 initial + 3 retries) as SetRetryLimit(3) intended.

Why transaction_too_old? In a low-contention single-node cluster, the first commit usually succeeds. Forced retries come from the read version expiring while the loop runs, not from write conflicts. The full workspace demo (§12b) generates real write conflicts using concurrent goroutines.

12b. Full workspace demo

cd issue-4091-persistent-tx-options

# Confirm FDB is reachable
fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"

# Run the three-way comparison
go run ./demo

The demo spawns two goroutines per experiment, both writing to the same key, so they generate real write conflicts. Expected output:

FDB issue #4091 – persistent transaction options demo
=====================================================

BROKEN   (bare Reset, options silently dropped)           elapsed=14ms  counter=2
FIXED    (SoftReset, options explicitly reapplied)        elapsed=12ms  counter=2
STANDARD (TransactWithPersistentOptions helper)           elapsed=10ms  counter=2

All three show counter=2 — both goroutines incremented exactly once (correct final value). The elapsed times look similar at default settings.

To make the semantic difference obvious, edit demo/main.go and lower the timeout to force a timeout error on the BROKEN path:

# In demo/main.go, set:  const timeoutMs = 50
go run ./demo

BROKEN   ...  elapsed=312ms  counter=2  ← budget kept resetting; ran 300+ ms
STANDARD ...  elapsed=52ms   counter=2  ← hard-stopped at 50 ms total budget

With a 3-node cluster (Options E or F): the demo works identically. The cluster topology does not affect the result — the conflict is goroutine-driven, not cluster-driven. The 3-node setup is useful if you want to observe real not_committed conflict errors rather than transaction_too_old expiry errors, which you can do by monitoring fdbcli --exec "status json" while the demo runs.

13. Apply the fix

The fix is already implemented in fdb_retry/retry.go. This section shows how to verify it compiles, run it, and understand what it changes.

13a. Verify the fix compiles

cd issue-4091-persistent-tx-options
go build ./...
go vet ./...

No output means success.

13b. Run with the fix active

The FIXED row in the demo output is the SoftReset path. The STANDARD row is TransactWithPersistentOptions. Both use the code in fdb_retry/retry.go.

go run ./demo

To confirm the fix is being used, check fdb_retry/retry.go:

grep -n "func SoftReset\|func TransactWith" fdb_retry/retry.go
# fdb_retry/retry.go:NN: func SoftReset(tr fdb.Transaction, opts PersistentOptions) error {
# fdb_retry/retry.go:NN: func TransactWithPersistentOptions(...) (interface{}, error) {

13c. What the fix does (summary)

SoftReset is a pragmatic fix for code that genuinely needs Reset(). The fully correct solution for preserving counters is TransactWithPersistentOptions (which uses OnError), or the Phase 2 C++ change described in §7.

14. Validate and verify

14a. Unit tests — no FDB cluster needed

These test the PersistentOptions builder logic: value semantics, no mutation, correct field access. They run anywhere, no server required.

cd issue-4091-persistent-tx-options
go test ./fdb_retry/... -v -count=1

Expected:

=== RUN   TestPersistentOptionsBuilder
--- PASS: TestPersistentOptionsBuilder (0.00s)
PASS
ok  github.com/10xdev/fdb-issue-4091/fdb_retry

If the test file does not exist yet, create fdb_retry/retry_test.go:

package fdb_retry_test

import (
    "testing"
    fdb_retry "github.com/10xdev/fdb-issue-4091/fdb_retry"
)

func TestPersistentOptionsBuilder(t *testing.T) {
    var opts fdb_retry.PersistentOptions
    if opts.HasTimeout() {
        t.Error("zero value should have no timeout")
    }
    opts = opts.WithTimeout(5000)
    if !opts.HasTimeout() || opts.TimeoutMs() != 5000 {
        t.Errorf("WithTimeout: HasTimeout=%v TimeoutMs=%d", opts.HasTimeout(), opts.TimeoutMs())
    }
    var zero fdb_retry.PersistentOptions
    if zero.HasTimeout() {
        t.Error("original was mutated")
    }
    opts3 := fdb_retry.PersistentOptions{}.
        WithTimeout(10_000).WithRetryLimit(5).WithMaxRetryDelay(500)
    if !opts3.HasTimeout() || !opts3.HasRetryLimit() || !opts3.HasMaxRetryDelay() {
        t.Error("chained builder missing a field")
    }
}

HasTimeout(), TimeoutMs(), etc. must be exported methods on PersistentOptions for this test. Add them when writing the PR version of persistent.go. The integration tests below exercise the full path through the private fields.

14b. Integration tests — FDB cluster required

These test the runtime behavior: retry limits are enforced, SoftReset re-applies options, write sets are cleared.

First, confirm your cluster is reachable:

fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal"

Create fdb_retry/integration_test.go:

//go:build integration

package fdb_retry_test

import (
    "testing"
    fdb_retry "github.com/10xdev/fdb-issue-4091/fdb_retry"
    "github.com/apple/foundationdb/bindings/go/src/fdb"
)

func setupDB(t *testing.T) fdb.Database {
    t.Helper()
    fdb.MustAPIVersion(730)
    db, err := fdb.OpenDefault()
    if err != nil {
        t.Skipf("FDB not available: %v", err)
    }
    return db
}

// TestRetryLimitIsRespected proves the helper stops after the configured limit.
func TestRetryLimitIsRespected(t *testing.T) {
    db := setupDB(t)
    attempts := 0
    opts := fdb_retry.PersistentOptions{}.WithRetryLimit(2)

    _, err := fdb_retry.TransactWithPersistentOptions(db, opts,
        func(tr fdb.Transaction) (interface{}, error) {
            attempts++
            tr.Get(fdb.Key("test:retry_limit"))
            tr.Set(fdb.Key("test:retry_limit"), []byte("x"))
            return nil, fdb.Error{Code: 1020} // not_committed — always retryable
        },
    )

    if err == nil {
        t.Fatal("expected error after retry limit")
    }
    // RetryLimit(2) → 1 initial + 2 retries = 3 total attempts maximum
    if attempts > 3 {
        t.Errorf("retry limit not respected: %d attempts, want ≤ 3", attempts)
    }
    t.Logf("stopped after %d attempts: %v", attempts, err)
}

// TestSoftResetClearsWriteSetButKeepsOptions proves write set is gone after
// SoftReset but the re-applied options allow a successful commit.
func TestSoftResetClearsWriteSetButKeepsOptions(t *testing.T) {
    db := setupDB(t)
    key := fdb.Key("test:soft_reset")
    opts := fdb_retry.PersistentOptions{}.WithTimeout(30_000).WithRetryLimit(10)

    tr, _ := db.CreateTransaction()
    opts.Apply(tr)
    tr.Set(key, []byte("before"))

    if err := fdb_retry.SoftReset(tr, opts); err != nil {
        t.Fatalf("SoftReset: %v", err)
    }

    // "before" write is gone; options are active; new write should commit
    tr.Set(key, []byte("after"))
    if err := tr.Commit().Get(); err != nil {
        t.Fatalf("Commit after SoftReset: %v", err)
    }

    val, _ := db.Transact(func(tr2 fdb.Transaction) (interface{}, error) {
        return tr2.Get(key).Get()
    })
    if string(val.([]byte)) != "after" {
        t.Errorf("expected 'after', got %q — Reset did not clear write set", val)
    }
}

Run:

# Single node or 3-node — same command
go test ./fdb_retry/... -tags integration -v -count=1

Expected:

=== RUN   TestRetryLimitIsRespected
    retry_test.go:NN: stopped after 3 attempts: transaction_too_old
--- PASS: TestRetryLimitIsRespected (0.12s)
=== RUN   TestSoftResetClearsWriteSetButKeepsOptions
--- PASS: TestSoftResetClearsWriteSetButKeepsOptions (0.03s)
PASS

14c. Behavioral verification with fdbcli

After running the full demo, check the final key value:

fdbcli -C "$FDB_CLUSTER_FILE" --exec "get issue4091:counter"
# `issue4091:counter' is `\x00\x00\x00\x00\x00\x00\x00\x02'

\x02 as a big-endian int64 means 2 — both goroutines incremented exactly once.

Additional spot-checks:

Clean up test keys before opening the PR:

fdbcli -C "$FDB_CLUSTER_FILE" \
  --exec "writemode on; clearrange issue4091 issue4092; clearrange test: test;"

14d. Run with the race detector

FDB operations are concurrent. Run with -race before opening the PR:

go test ./fdb_retry/... -tags integration -race -count=1
go run -race ./demo

Both should complete with no DATA RACE warnings.

15. Pre-PR checklist

Work through this in order. Every item must be green before opening the PR.

SETUP (§11)
  [ ] fdbcli status minimal  →  "The database is available."
  [ ] go build ./...         →  no errors
  [ ] go vet ./...           →  no warnings

REPRODUCE THE BUG (§12)
  [ ] Minimal repro (§12a): BROKEN path hits safety valve (21 attempts)
  [ ] Full demo (§12b):     BROKEN row shows inflated elapsed time with
                             timeoutMs = 50

APPLY THE FIX (§13)
  [ ] go build ./...         →  no errors after any edits to retry.go
  [ ] Full demo:             FIXED and STANDARD rows show correct counter=2
                             and elapsed time ≤ timeoutMs + small overhead

VALIDATE (§14)
  [ ] Unit tests             go test ./fdb_retry/... -v -count=1
  [ ] Integration tests      go test ./fdb_retry/... -tags integration -v -count=1
  [ ] Race detector          go test ./fdb_retry/... -tags integration -race -count=1
  [ ] fdbcli key check       issue4091:counter = \x02
  [ ] Test data cleaned      clearrange wiped issue4091:* and test:* keys

PR PREPARATION (§16)
  [ ] Copy retry.go → bindings/go/src/fdb/persistent.go
  [ ] Change package fdb_retry → package fdb
  [ ] Remove fdb. prefix from all FDB type references
  [ ] Make TransactWithPersistentOptions a method on Database
  [ ] cd bindings/go && go test ./src/fdb/... -race -timeout 120s  →  all pass
  [ ] Stack tester still passes (§16 step 5)
  [ ] Commit message references #4091 (§16 step 6)

16. Building from source (optional — for the C++ phase)

You do not need to build FDB from source for the Go layer fix. The pre-installed /usr/local/lib/libfdb_c.dylib is sufficient. This section documents the build process for when you later implement the C++ phase.

16a. Prerequisites

# Xcode command-line tools
xcode-select --install

# CMake 3.24+
brew install cmake

# Ninja (faster builds than make)
brew install ninja

# Mono (required for the flow code generator)
brew install mono

# Boost (FDB uses specific Boost headers)
brew install boost

# Go 1.21+ (already installed if you are reading this)

16b. Configure and build

cd ~/src/foundationdb

# Create a build directory (out-of-source build)
cmake -G Ninja -S . -B build \
  -DCMAKE_BUILD_TYPE=Debug \
  -DFDB_RELEASE=OFF \
  -DOPEN_FOR_IDE=OFF

# Build only what you need for C++ work.
# fdbserver + fdbcli takes ~20 min on M1; libfdb_c takes ~5 min.
cd build
ninja fdbserver fdbcli fdb_c  # or just: ninja (builds everything ~40 min)

Build artifacts land in build/:

build/bin/fdbserver
build/bin/fdbcli
build/lib/libfdb_c.dylib

To test your custom-built binaries instead of the system ones:

# Point the Go binding to your local libfdb_c
export CGO_CFLAGS="-I$HOME/src/foundationdb/fdbrpc/include"
export CGO_LDFLAGS="-L$HOME/src/foundationdb/build/lib"
export DYLD_LIBRARY_PATH="$HOME/src/foundationdb/build/lib:$DYLD_LIBRARY_PATH"

# Run the demo against your custom-built server
~/src/foundationdb/build/bin/fdbserver \
  -p 127.0.0.1:4501 \
  -C ~/.fdb/fdb.cluster \
  -d ~/fdb-data2 &

go run ./demo

16c. Running FDB’s own test suite (simulation)

FDB’s killer feature is its deterministic simulation framework. Bugs that take months to appear in production can be reproduced in seconds under simulation. This is what you run for C++ changes:

cd build

# Joshua is FDB's simulation runner — this runs 100 simulation seeds:
bin/fdbserver -r simulation -f tests/fast/WriteTagThrottling.toml -s 0
# -s <seed> sets the random seed; different seeds exercise different paths

# Or use the CMake test target (runs the full suite, takes a long time):
ctest -j4 --output-on-failure

You do not need to touch simulation for the Go binding change (Phase 1). Simulation is relevant for Phase 2 (the C++ fdb_transaction_soft_reset).

17. Step-by-step PR guide

Step 1 — Fork and branch

# On GitHub: click "Fork" on https://github.com/apple/foundationdb
# Then locally:
cd ~/src/foundationdb
git checkout -b fix/go-persistent-tx-options main

Step 2 — Place the new file

# The Go bindings sub-module
cd bindings/go/src/fdb

Create persistent.go. The content is the same as fdb_retry/retry.go in this workspace, with two changes:

1. Change package fdb_retry → package fdb
2. Remove the fdb. qualifier from all FDB type references (they are now in the same package). For example:
   • fdb.Transaction → Transaction
   • fdb.Database → Database
   • fdb.Error → Error

The function signatures become:

// In package fdb — no fdb. prefix needed

func (o PersistentOptions) Apply(tr Transaction) error { ... }

func SoftReset(tr Transaction, opts PersistentOptions) error { ... }

func (d Database) TransactWithPersistentOptions(
    opts PersistentOptions,
    f func(Transaction) (interface{}, error),
) (interface{}, error) { ... }

Note that TransactWithPersistentOptions becomes a method on Database (consistent with Transact and ReadTransact).

Step 3 — Add tests

Create bindings/go/src/fdb/persistent_test.go with the integration tests from §13c, adapted to the fdb package:

package fdb_test    // external test package, consistent with existing tests

import (
    "testing"
    "github.com/apple/foundationdb/bindings/go/src/fdb"
)

Check how the existing test file fdb_test.go sets up its cluster connection and mirror that pattern exactly.

Step 4 — Run existing tests

cd bindings/go

# Run all Go binding tests (requires fdbserver running)
go test ./src/fdb/... -v -timeout 120s

# Run with the race detector (FDB's CI always does this)
go test ./src/fdb/... -race -timeout 120s

All pre-existing tests must still pass. If any fail, something in your new file has a name collision or an import cycle.

Step 5 — Run the stack tester

FDB’s binding correctness is verified by a deterministic “stack tester” that replays a sequence of API calls and compares output across all language bindings. It lives at bindings/go/src/_stacktester/.

# Build the stack tester binary
cd bindings/go
go build -o /tmp/go_stacktester ./src/_stacktester/

# Run it against a live cluster (the Python reference binding runs in parallel)
/tmp/go_stacktester --cluster-file ~/.fdb/fdb.cluster --test-name api --num-ops 1000

Your new PersistentOptions and SoftReset APIs are not exercised by the stack tester (it only tests core read/write/commit operations), but you must confirm the stack tester still passes to show no regressions.

Step 6 — Commit style

FDB uses a conventional commit format with a component prefix:

bindings/go: add PersistentOptions and SoftReset for custom retry loops

Closes #4091

Transaction options timeout, retry_limit, and max_retry_delay are
preserved by the C layer across OnError() calls (API >= 610), but are
silently discarded when a custom retry loop calls Reset() instead.

This adds:
- PersistentOptions: a builder-style value type for the three options.
- PersistentOptions.Apply(tr): sets all stored options on a transaction.
- SoftReset(tr, opts): calls Reset() then Apply(), making the option
  reapplication explicit and auditable.
- Database.TransactWithPersistentOptions(opts, f): a drop-in for
  Database.Transact that uses OnError internally (so the C layer's own
  persistence is honoured) and also reapplies opts after each retry.

A deeper fix (fdb_transaction_soft_reset in NativeAPI.actor.cpp) that
preserves accumulated elapsed-time and retry counters across Reset() is
tracked as a follow-up to this PR.

Step 7 — Open the PR

1. Push the branch: git push origin fix/go-persistent-tx-options
2. Open a PR against apple/foundationdb:main on GitHub.
3. In the PR description:
   • Link the issue: Closes #4091
   • Explain what Phase 2 would look like (the C++ change)
   • Mention which tests you ran and on which FDB version
4. Request review from @sfc-gh-abeamon (who filed the issue) or @jzhou77 (who labelled it).

Step 8 — Respond to review

FDB maintainers are thorough. Common review points for binding PRs:

• API naming consistency (match the style of Transact, ReadTransact)
• Whether TransactWithPersistentOptions should be a free function or a method on Database
• Whether SoftReset needs to be exported vs. internal
• Test coverage for the zero-value case (no options set)

18. Understanding the FDB repository layout

When you open the repo for the first time, the directory count is overwhelming. Here is a map of the directories you actually touch for this issue and related work:

foundationdb/
│
├── bindings/                   ← language bindings
│   ├── go/                     ← Go sub-module (separate go.mod)
│   │   └── src/fdb/
│   │       ├── transaction.go  ← Transaction type, Cancel/Reset/OnError/Commit
│   │       ├── database.go     ← Database type, Transact/ReadTransact
│   │       ├── generated.go    ← SetTimeout/SetRetryLimit etc (auto-generated)
│   │       └── persistent.go   ← ← ← YOUR NEW FILE
│   ├── python/                 ← Python binding (same gap exists here)
│   └── java/                   ← Java binding
│
├── fdbclient/
│   ├── NativeAPI.actor.cpp     ← C++ Transaction implementation
│   │                             fdb_transaction_reset() lives here
│   │                             This is where Phase 2 changes go
│   ├── vexillographer/
│   │   └── fdb.options         ← Source of truth for all option codes
│   │                             Option 500=timeout, 501=retry_limit, 502=max_retry_delay
│   │                             generated.go is built FROM this file
│   └── include/foundationdb/
│       └── fdb_c.h             ← C ABI header; all bindings #include this
│
├── fdbserver/
│   └── workloads/              ← simulation workloads (C++ integration tests)
│
├── flow/
│   ├── actorcompiler/          ← transforms ACTOR keyword into plain C++
│   └── flow.h                  ← Future<T>, ACTOR macros
│
└── tests/
    └── rare/                   ← simulation test scenarios

The generated.go pipeline

generated.go is not hand-written. It is produced by a script that reads fdbclient/vexillographer/fdb.options. If you ever need to add a new option you would edit fdb.options, run the generator, and commit the output. For this issue we only add Go types and helpers on top of existing option codes, so there is no need to re-run the generator.

# How to regenerate bindings/go/src/fdb/generated.go (for reference only)
cd bindings/go/src/_util
go run translate_fdb_options.go \
  ../../../../fdbclient/vexillographer/fdb.options \
  > ../fdb/generated.go

Actor Framework (Flow) quick reference

You will see this pattern throughout fdbclient/:

ACTOR Future<Void> someOperation(Database db) {
    state Transaction tr(db);
    loop {
        try {
            wait(tr.someAsyncOp());
            return Void();
        } catch (Error& e) {
            wait(tr.onError(e));
        }
    }
}

Key terms:

19. Communication and community

Before opening a PR, comment on issue #4091 to claim it:

“I’d like to work on this. My approach: add PersistentOptions, SoftReset, and Database.TransactWithPersistentOptions to the Go bindings as Phase 1, with the C++ fdb_transaction_soft_reset as a follow-up PR.”

Channels:

• GitHub Issues / PR comments — primary channel for technical discussion
• FDB Community Forums — higher-level questions
• Discord — real-time chat with maintainers

What maintainers care about:

1. Correctness — the fix must not introduce a regression in any binding
2. Consistency — match the naming and style of existing Transact/ReadTransact
3. Test coverage — both unit and integration tests
4. Documentation — update doc.go or add godoc to new exported symbols
5. Backward compatibility — the change must be additive (no existing signatures change)

20. Quick reference

FDB cluster management

# ── Option A/B: bare metal ─────────────────────────────────────────────────
export FDB_CLUSTER_FILE=~/.fdb/fdb.cluster           # macOS
export FDB_CLUSTER_FILE=/etc/foundationdb/fdb.cluster # Linux systemd install

fdbcli -C "$FDB_CLUSTER_FILE" --exec "status minimal" # health check
pkill fdbserver                                        # stop all processes

# ── Option C: Docker single node ───────────────────────────────────────────
docker run -d --name fdb -p 4500:4500 \
  foundationdb/foundationdb:7.3.27
docker exec fdb fdbcli --exec "configure new single memory"
docker exec fdb cat /etc/foundationdb/fdb.cluster \
  | sed 's/@.*:/@127.0.0.1:/' > ./fdb.cluster
export FDB_CLUSTER_FILE="$PWD/fdb.cluster"
docker stop fdb && docker rm fdb                       # stop

# ── Option D: Docker Compose single node (this repo) ───────────────────────
docker compose up -d && bash scripts/bootstrap-fdb.sh
export FDB_CLUSTER_FILE="$PWD/fdb.cluster"
docker compose down                                    # stop

# ── Option E: 3-node bare metal ────────────────────────────────────────────
fdbserver -p 127.0.0.1:4500 -C ~/.fdb/fdb.cluster -d ~/fdb-data/n1 &
fdbserver -p 127.0.0.1:4501 -C ~/.fdb/fdb.cluster -d ~/fdb-data/n2 &
fdbserver -p 127.0.0.1:4502 -C ~/.fdb/fdb.cluster -d ~/fdb-data/n3 &
sleep 3 && fdbcli -C ~/.fdb/fdb.cluster --exec "configure new triple memory"
pkill fdbserver                                        # stop

# ── Option F: 3-node Docker Compose ────────────────────────────────────────
docker compose -f docker-compose-cluster.yml up -d
sleep 5
docker exec fdb-node1 fdbcli --exec "configure new triple memory"
docker exec fdb-node1 cat /etc/foundationdb/fdb.cluster \
  | sed 's/fdb1/127.0.0.1/' > ./fdb.cluster
export FDB_CLUSTER_FILE="$PWD/fdb.cluster"
docker compose -f docker-compose-cluster.yml down      # stop

Reproduce → apply fix → validate → PR

# ── REPRODUCE THE BUG ──────────────────────────────────────────────────────
cd /tmp/fdb-repro && go run .        # minimal standalone (§12a)
                                     # expect: BROKEN hits 21 attempts

cd issue-4091-persistent-tx-options
go run ./demo                        # full three-way comparison (§12b)
                                     # expect: BROKEN/FIXED/STANDARD all counter=2
# Edit demo/main.go: set timeoutMs = 50, then re-run to see elapsed diverge

# ── APPLY THE FIX ──────────────────────────────────────────────────────────
go build ./...                       # must compile cleanly
go vet ./...                         # must pass

# ── VALIDATE ───────────────────────────────────────────────────────────────
go test ./fdb_retry/... -v -count=1                        # unit tests, no FDB
go test ./fdb_retry/... -tags integration -v -count=1      # integration, FDB needed
go test ./fdb_retry/... -tags integration -race -count=1   # race detector
go run -race ./demo                                        # race detector on demo

# Spot check final key value
fdbcli -C "$FDB_CLUSTER_FILE" --exec "get issue4091:counter"
# → `issue4091:counter' is `\x00\x00\x00\x00\x00\x00\x00\x02'

# Clean up test keys
fdbcli -C "$FDB_CLUSTER_FILE" \
  --exec "writemode on; clearrange issue4091 issue4092; clearrange test: test;"

# ── PR PREPARATION ─────────────────────────────────────────────────────────
# 1. cp fdb_retry/retry.go → bindings/go/src/fdb/persistent.go
# 2. Change package + remove fdb. prefixes + make TransactWith... a method
# 3. cd bindings/go && go test ./src/fdb/... -race -timeout 120s
# 4. Run stack tester (§16 step 5)
# 5. git push && open PR referencing #4091