Option C — Record Layer over FoundationDB

Pattern: “structured records + secondary indexes, native FDB” — the same idea Apple’s fdb-record-layer is built around, distilled to ~250 lines of Go so you can read the whole thing in one sitting.

What problem this solves

FDB’s wire-level API is ordered KV. Most applications want records (a name, a city, an email) plus queries like “find users in Paris”. A record layer sits in between:

• Records are serialized blobs stored under a primary key.
• Indexes are extra KV entries keyed by (field, value, pk) so range scans on the index subspace return matching primary keys.
• Both updates happen in one FDB transaction, so the index is never out of sync with the records — even under concurrent writers.

Files

recordlayer/
  encoding.go   Key layouts for records (tag 0x00) and indexes (tag 0x01)
  msgpack.go    Wrapper around github.com/vmihailenco/msgpack
  store.go      Open / PutRecord / GetRecord / DeleteRecord / ScanRecords
  index.go      LookupByIndex (range-scan index → batch-read records)
demo/main.go    Users keyed by id, indexed by city

Key layout

records:   <ns> 0x00 <schemaName> 0x00 <pk>                               -> msgpack(record)
indexes:   <ns> 0x01 <schemaName> 0x00 <fieldName> 0x00 <value> 0x00 <pk> -> ""

Splitting records and indexes into two top-level “tags” keeps each subspace range-scannable without touching the other. Integers are written as sign-bit-flipped big-endian (see encodeIndexValue) so negative values sort before positive in the index — a tiny but useful property when you want range queries like “age >= 18”.

Why it’s transactional by construction

PutRecord does, inside one db.Transact:

1. Read the previous version of the record (if any).
2. For each indexed field that changed or was removed, Clear the old index entry.
3. Set the new record bytes.
4. Set fresh index entries for the new field values.

Because FDB transactions are serializable, no other client can observe a state where the record was updated but its indexes were not. This is the single biggest reason to build atop FDB rather than a non-transactional KV.

Mapping LookupByIndex to FDB ops

db.ReadTransact(rt -> {
    idxKVs := rt.GetRange(indexPrefix(schema, field, value))   // ordered scan
    for each idxKV:
        pendings[i] = rt.Get(recordKey(schema, pk))            // pipelined
    for each pending: collect record
})

The pipeline trick (issuing all rt.Get calls before awaiting any) lets FDB overlap network round-trips. Latency ≈ slowest single read instead of sum-of-reads.

Running

1. Bring up FDB and bootstrap (see top-level README).
2. cd option-c-record-layer && go run ./demo -cluster ../fdb.cluster

Expected output:

All users (PK order):
  u1 -> map[city:Paris name:Alice]
  u2 -> map[city:Tokyo name:Bob]
  u3 -> map[city:Paris name:Carol]

Lookup city=Paris (via secondary index):
  u1 -> map[city:Paris name:Alice]
  u3 -> map[city:Paris name:Carol]

After moving Alice to Tokyo:
Lookup city=Paris:
  u3 -> map[city:Paris name:Carol]
Lookup city=Tokyo:
  u1 -> map[city:Tokyo name:Alice]
  u2 -> map[city:Tokyo name:Bob]

What this layer omits compared to fdb-record-layer

• Schema evolution / Protobuf descriptors.
• Index definitions like COUNT, MAX, SUM aggregates.
• Query planner over multiple indexes.
• Versioned records and meta-data subspace.

But the storage shape and atomicity story are exactly the same.