Lab 01 — Write-Ahead Log (WAL)

• Concept: Why a write-ahead log?
  • Why append-only?
    • HDD mechanics
    • SSD / NVMe mechanics
    • O_APPEND atomicity guarantee (POSIX)
    • Filesystem journaling vs application WAL
  • What Sync means
    • The complete write() path: userspace → persistent storage
    • fsync vs fdatasync vs sync_file_range
    • Dirty page writeback: the kernel’s async path
    • O_DIRECT: bypassing the page cache
    • O_DSYNC / O_SYNC: per-write durability
    • Drive write cache: the last hiding place
    • macOS: F_FULLFSYNC
    • sync_file_range: pipelined WAL writes
    • io_uring: truly async file I/O without blocking syscalls
    • epoll / select / poll — why they do not apply to file I/O
  • Partial writes and tail corruption
    • Why only the tail can be corrupt
    • What “silent discard” means for the caller
    • CRC32 vs CRC32C vs stronger checksums
    • Sparse files and zero-padding: an edge case
• File format
• Implementation notes
  • Why two separate Write calls in Append?
    • Reason 1: zero allocation
    • Reason 2: no zero-copy join for stack array + slice
    • The proper fix: scatter-gather I/O (writev)
    • The tradeoff: dangling header on crash
    • Why production engines still use two separate writes
  • Self-framing: no delimiters needed
    • Why length-prefix beats delimiters
    • Who else uses this framing
    • LevelDB’s block-based WAL framing
• Running the tests
• Running the demo
• FoundationDB parallel

Concept: Why a write-ahead log?

Any storage engine faces one fundamental problem: memory is volatile and disk writes are not atomic. If the process crashes mid-write, the on-disk state could be partially updated — half a key written, a stale value left in place, or an index that disagrees with the data it points to.

The classic solution is a write-ahead log (WAL): a simple append-only file where every mutation is recorded before it modifies any other data structure. On crash, the WAL is replayed from beginning to end, reconstructing the in-memory state exactly as it was at the moment of the crash.

  Normal write:           Crash at ①:            Crash at ②:

  1. append to WAL  ①     WAL record written       WAL record written
  2. update memory  ②     memory update lost       memory updated
  3. acknowledge          caller never got OK      replay restores memory

Because the write is only acknowledged to the caller after the WAL record is durably on disk, the promise is: any acknowledged write survives a crash.

Why append-only?

Random writes on spinning disk require a physical seek — moving the read head to the right track and waiting for the right sector to rotate under it. A sequential append never seeks; it always writes to the end. Even on SSDs, sequential writes are preferred: the FTL (Flash Translation Layer) batches and aligns sequential writes more efficiently, reducing write amplification inside the drive. An append-only log extracts the maximum possible throughput from the underlying storage hardware.

HDD mechanics

IOPS (Input/Output Operations Per Second) measures how many discrete read-or-write operations a storage device can sustain per second. One IOPS covers the full round-trip from when the OS submits the request until the drive acknowledges completion. Storage benchmarks always quote a 4 KB block size because 4 KB is the smallest addressable unit on most filesystems and matches the CPU MMU’s page size.

DMA (Direct Memory Access) is a hardware capability that lets a storage controller transfer data directly between its internal buffers and host RAM without occupying the CPU. The CPU writes a DMA descriptor (source address, destination address, byte count) to the controller’s register, then goes on to other work. The controller’s DMA engine moves the bytes autonomously and raises a CPU interrupt when done. Without DMA, every byte would require a CPU load + CPU store — hundreds of millions of wasted cycles per second at modern storage speeds.

A spinning disk at 7,200 RPM completes one revolution in 8.3 ms. Every random write requires three sequential phases before the first byte hits magnetic media:

Random 4 KB write: ~9 ms → ~110 IOPS. Sequential 4 KB write (head already at end): ~0.03 ms → ~33,000 IOPS.

The ratio is roughly 300×. A WAL that forces sequential access converts a seek-bound device into a throughput-bound device for the write path.

SSD / NVMe mechanics

SSDs have no mechanical actuator arm, so seek time is near zero (~50 µs vs. ~5 ms for HDD). But flash storage introduces fundamentally different constraints — rooted in the physics of how NAND cells store charge.

NAND flash: floating-gate transistors

A NAND flash cell is a floating-gate transistor — a standard MOSFET with an extra layer of polysilicon (the “floating gate”) sandwiched between two thin insulating oxide layers. The floating gate is electrically isolated: charge placed on it has nowhere to go and stays there for years.

       Control gate  (word line — driven by the SSD controller)
            │
    ──────────────────────────────  ← interpoly dielectric
    ──────── ■ ──────────────────   ← floating gate (stores charge)
    ──────────────────────────────  ← tunnel oxide (~8 nm — key constraint)
            │
    ══ Source ══╬══ Drain ══════   ← silicon channel / bit line

• Programming (writing a 0): the controller applies ~20 V to the control gate. Quantum tunneling forces electrons from the channel through the thin tunnel oxide onto the floating gate. The trapped electrons lower the transistor’s threshold voltage — the sense amplifier reads the cell as logic 0.
• Erasing (resetting to 1): the controller applies a large negative voltage to the control gate (or positive to the substrate). Electrons tunnel back off the floating gate. The threshold voltage rises — the cell reads as 1.
• Reading: the controller applies a moderate voltage between 0 V and the programmed threshold. If the cell conducts (floating gate uncharged) → 1. If it does not (floating gate charged, threshold lowered below sense voltage) → 0.

The critical asymmetry that drives all SSD design decisions: programming is fine-grained (one page at a time), but erasing is coarse-grained (one entire block at once) — because the erase circuit applies voltage uniformly across all cells in a block via their shared substrate connection.

SLC, MLC, TLC, QLC: bits per cell

The floating gate’s charge exists on a continuous voltage spectrum. By dividing that spectrum into more discrete voltage windows, you can store more bits per cell:

SLC (1 bit):   |──── 1 ─────|──── 0 ─────|
                 uncharged      charged
                 (wide margin — tolerates oxide degradation)

MLC (2 bits):  |── 11 ──|── 10 ──|── 01 ──|── 00 ──|
                 4 voltage windows

TLC (3 bits):  |─111─|─110─|─101─|─100─|─011─|─010─|─001─|─000─|
                 8 voltage windows (~100 mV margin between each)

QLC (4 bits):  16 voltage windows (~30 mV margin — extreme sensitivity)

More bits per cell = smaller voltage margins. Each P/E cycle slightly degrades the tunnel oxide by trapping a residual charge, shifting all threshold voltages slightly. With 8 windows spaced ~100 mV apart (TLC), a shift of just 50 mV can move a cell from window 3 into window 4 — a read error. SLC’s two wide windows tolerate far more oxide degradation before read failures occur. Selecting drive type is not optional for production WAL servers: a database WAL on a QLC SSD may wear out in months under heavy write load.

P/E cycles: program and erase at the physics level

A P/E cycle (Program/Erase cycle) is one complete round of:

1. Program: force electrons onto the floating gate via Fowler-Nordheim quantum tunneling — electrons pass through the ~8 nm tunnel oxide under a strong electric field (~10 MV/cm).
2. Erase: remove those electrons by reversing the field direction.

Each cycle injects a small amount of charge into the tunnel oxide itself (not the floating gate). This oxide-trapped charge gradually raises the tunnel barrier, requiring ever-higher programming voltages and eventually making cells permanently stuck in one state. After TLC’s rated ~1,000–3,000 P/E cycles, blocks are retired — the FTL marks them bad and stops allocating to them.

The physical hierarchy: Die → Plane → Block → Page

NAND chips are organised in a strict four-level hierarchy that determines which operations can run in parallel and at what granularity:

┌──────────────────────────────────────────────────────────────────┐
│  Die  (one physical silicon chip; an M.2 drive has 4–16 dies)   │
│                                                                   │
│  ┌────────────────────────────┐  ┌────────────────────────────┐  │
│  │  Plane 0                   │  │  Plane 1                   │  │
│  │  ┌──────────────────────┐  │  │  ┌──────────────────────┐  │  │
│  │  │ Block 0  (256 KB–4 MB│  │  │  │ Block 0              │  │  │
│  │  │  page 0  (4–16 KB)   │  │  │  │  page 0              │  │  │
│  │  │  page 1              │  │  │  │  page 1              │  │  │
│  │  │  page 2  …           │  │  │  │  …                   │  │  │
│  │  ├──────────────────────┤  │  │  ├──────────────────────┤  │  │
│  │  │ Block 1              │  │  │  │ Block 1              │  │  │
│  │  │  …                   │  │  │  │  …                   │  │  │
│  │  └──────────────────────┘  │  │  └──────────────────────┘  │  │
│  └────────────────────────────┘  └────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘

• Die: one physical silicon chip. A modern M.2 NVMe drive contains 4–16 dies. Multiple dies can operate in parallel (die interleaving) — the controller submits a page write to die 0 while die 1 is still programming a previous request (~100–300 µs), hiding the programming latency behind parallelism.
• Plane: each die has 2–4 planes, each with its own page register and sense amplifiers. Planes within one die can execute one operation simultaneously — a 2-die × 2-plane drive has 4 independent NAND operation slots.
• Block: the erase unit — 256 KB to 4 MB of pages that share an erase circuit. Think of a block as a hotel floor: you can furnish individual rooms (program pages), but to reset, you must vacate the entire floor at once (erase the whole block). Pages within a block can only be programmed once per erase cycle, in order from page 0 upward.
• Page: the read and write unit — 4–16 KB. The smallest granularity at which data can be programmed into NAND or read out.

The fundamental constraint that follows: you cannot overwrite an existing page in place. To update data at logical address X, the controller must write it to a fresh page elsewhere and update a mapping table. The old page becomes stale.

FTL: the Flash Translation Layer

The FTL is firmware running on the SSD’s controller chip — typically an ARM Cortex-R processor with 256 MB–4 GB of its own DRAM for bookkeeping. The host OS presents read/write requests using Logical Block Addresses (LBAs) — a flat numbering of 512-byte or 4096-byte sectors from 0 to N. The FTL translates LBAs to physical NAND locations and manages three critical jobs:

1. Logical-to-Physical Mapping (L2P)

The FTL maintains an L2P mapping table in DRAM:

L2P table (one entry per LBA):
  LBA 42  →  Die 0, Plane 1, Block 17, Page 3   ← current live copy
  LBA 99  →  Die 1, Plane 0, Block 29, Page 0
  …

When the host overwrites LBA 42, the FTL:

1. Allocates a fresh page (e.g., Die 0, Plane 1, Block 29, Page 1).
2. Programs the new data there.
3. Updates the L2P table: LBA 42 → Block 29, Page 1.
4. Marks the old page (Block 17, Page 3) as stale — it holds outdated data and cannot be reused until its block is erased.

The L2P table is periodically checkpointed to reserved NAND blocks. If power is lost before a checkpoint, the FTL replays its own internal log on next power-on — the SSD controller runs a WAL for exactly the same reason our application does.

2. Garbage Collection (GC)

Over time, blocks accumulate stale pages. A block with only stale pages wastes space but can’t be reused until it’s erased. GC reclaims these blocks:

Victim block (Block 17):       Step 1 — copy live pages out:
  page 0  [stale]                → write to fresh clean block
  page 1  [live, LBA 5]   ──→   page in Block 31
  page 2  [stale]
  page 3  [live, LBA 8]   ──→   page in Block 31
  page 4  [stale]

Step 2 — erase Block 17:   Step 3 — Block 17 is now clean and reusable.
  all 256 KB erased

This copy-then-erase is write amplification: the host wrote one new 4 KB page, but GC physically moved dozens of live pages internally to free up space.

WAF (Write Amplification Factor) measures how many bytes the drive actually writes per byte the host requested. WAF 10× means 10 physical writes per 1 logical write — consuming drive endurance 10× faster than necessary.

Under heavy random write load, GC runs continuously, competing with host I/O for NAND bandwidth. When all blocks hold at least some live data and GC must run before every host write, throughput collapses — the “write cliff”. Sequential workloads that fill entire blocks in order leave no stale pages and impose almost zero GC pressure.

3. Wear Leveling

Without intervention, the FTL’s allocator would repeatedly reuse the same recently-emptied blocks while cold blocks (holding rarely-updated data) remain pristine. Those hot blocks exhaust their P/E cycles while most of the drive stays barely worn.

Wear leveling redistributes writes across all blocks:

• Dynamic wear leveling: new writes go to the least-worn block with free pages.
• Static wear leveling: periodically evict cold data off low-wear-count (pristine) blocks and relocate it to higher-wear blocks — freeing fresh cells for hot sequential writes.

A WAL — which writes sequentially and fills blocks in order — is the ideal workload for the FTL: GC pressure is minimal, pages fill blocks cleanly, and wear distributes naturally across the drive.

Consequences for WAL design:

• Drive endurance: TLC NAND has ~1,000–3,000 P/E cycles per cell. WAF 10× means the drive wears 10× faster than necessary. Choose SLC or MLC for write-intensive WAL workloads.
• Sustained throughput: GC competing with host writes causes the write cliff. Sequential workloads delay this dramatically.
• Wear leveling: the FTL’s algorithm works best when writes fill whole blocks in order, simplifying its bookkeeping.

Typical peak NVMe figures (consumer):

• Random 4 KB write: 300k–700k IOPS.
• Sequential write: 3–7 GB/s.
• Sustained random write under GC load: 100k–200k IOPS.
• Sustained sequential write: close to peak (GC almost absent).

O_APPEND atomicity guarantee (POSIX)

Our file is opened with O_APPEND. POSIX specifies:

The file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.

The kernel holds the inode lock between the implicit lseek(SEEK_END) and the write(). Two concurrent processes both calling write() on the same O_APPEND fd will not interleave within a single call — one atomically claims its extent at EOF before the other begins.

// Both processes open the same WAL with O_APPEND.
int fd = open("wal.log", O_CREAT|O_RDWR|O_APPEND, 0644);

// Safe from two concurrent processes — the kernel serialises offset assignment.
write(fd, record_a, len_a);  // Process A
write(fd, record_b, len_b);  // Process B — appended after A, never overlapping

Limit for our design: O_APPEND atomicity covers a single write() call. Our Append does two writes (header, then payload). If two goroutines shared the WAL and called Append concurrently, the header of one could appear between the header and payload of the other. Our WAL is single-writer, so this is not an issue. Multi-writer WALs use flock / fcntl locks or a per-writer fd with a merge step. (flock(2) grants advisory exclusive or shared locks on an entire file; fcntl(2) with F_SETLK / F_SETLKW provides POSIX record locks on byte ranges — both are advisory, meaning only co-operating processes that explicitly call the lock API are serialised.)

Filesystem journaling vs application WAL

Modern filesystems (ext4, XFS, APFS, ZFS, btrfs) maintain their own internal journals. These protect filesystem metadata — inode tables, directory entries, extent maps — not the application’s payload bytes.

data=ordered prevents the “file contains garbage” bug (a new file’s blocks are committed before the directory entry that exposes them), but it provides no guarantee about which application-level writes are logically durable. The application still needs its own WAL to define that boundary.

ZFS is a special case: its copy-on-write B-tree model provides atomic block writes and its ZIL (ZFS Intent Log) acts as a per-filesystem WAL. ZFS never overwrites data in place — every write creates a new tree node and atomically redirects the tree’s root pointer. The ZIL is a separate append-only log device (ideally a low-latency SLC SSD or NVRAM SLOG — Separate LOG Device) where synchronous writes are committed before ZFS applies them to the main pool. When fsync arrives, ZFS flushes the ZIL. On recovery, the ZIL is replayed to reconstruct the pool’s logical state. Most databases still keep their own WAL for portability across filesystems.

What Sync means

Calling file.Write() on any OS copies bytes into the kernel’s page cache — a region of RAM. The OS will flush dirty pages to disk eventually, but “eventually” is seconds later, not microseconds. If the power is cut before the flush, those pages are lost.

fdatasync (or file.Sync() in Go) forces the kernel to flush all dirty pages for that file to the physical storage medium before returning. It blocks until the drive confirms the data is persistent. Every WAL append in our implementation calls Sync, so each acknowledged write is truly durable.

The complete write() path: userspace → persistent storage

Understanding exactly where data lives at each stage is the key to knowing what can be lost in a crash.

Key abstractions in the Linux I/O stack

VFS (Virtual File System)

The VFS is the kernel’s abstraction layer over all filesystems. Every filesystem (ext4, XFS, APFS, NFS, tmpfs) registers a file_operations struct with the VFS containing function pointers for read, write, fsync, mmap, and so on. When your Go program calls os.File.Write(), the kernel dispatches to the registered handler:

// Simplified from fs/read_write.c
ssize_t vfs_write(struct file *file, const char __user *buf,
                  size_t count, loff_t *pos) {
    // dispatches via the filesystem's registered .write_iter pointer:
    return file->f_op->write_iter(&kiocb, &iter);
}

The VFS exists so the same write(2) syscall works correctly regardless of which filesystem the file lives on — the caller never needs to know.

struct folio / struct page

The kernel page cache stores file data in 4 KB pages (matching the CPU MMU’s page size). The newer struct folio groups multiple physically-contiguous pages into one logical unit, reducing bookkeeping overhead for large I/Os. Each page or folio tracks:

• The physical RAM frame it occupies.
• PG_dirty: set when userspace has written to the page but it has not yet been flushed to disk.
• PG_uptodate: set when the page content reflects the on-disk state.

write() calls copy_from_user() to copy bytes from your process’s virtual address space into the appropriate folio in the page cache, then sets PG_dirty. The disk write happens later — asynchronously by the writeback thread, or immediately if fdatasync is called.

struct bio and blk_mq

Once the filesystem decides to flush dirty pages, it does not call the device driver directly. It constructs a struct bio and submits it to the block layer:

• struct bio (Block I/O): the fundamental I/O request unit. It describes one logical transfer: which device, starting Logical Block Address (LBA — the flat numbering of 512-byte or 4096-byte sectors that the OS uses), and a scatter-gather list of (page, offset, length) tuples. Multiple small I/Os can be merged into one bio by the scheduler.

• blk_mq (Block Multi-Queue): the Linux block I/O scheduler (introduced in Linux 3.13 to replace the single-queue elevator). It maintains per-CPU software staging queues and per-hardware-queue dispatch queues, enabling parallel I/O submission on many-core machines. Available schedulers:

  • mq-deadline: latency-bounded; prevents starvation. Good for HDDs.
  • kyber: low-overhead; tuned for fast NVMe.
  • bfq (Budget Fair Queueing): proportional I/O bandwidth fairness.
  • none: no reordering; submit in arrival order. Best for NVMe.

SATA/AHCI: FIS and NCQ

SATA (Serial ATA) is the physical and electrical interface between the host and the drive. AHCI (Advanced Host Controller Interface) is the software interface — a standardised register layout that the kernel’s AHCI driver programs to submit commands over SATA.

A FIS (Frame Information Structure) is the packet format SATA uses to carry commands and data between host and drive — analogous to an Ethernet frame:

FIS type 0x27 — Register (Host to Device):
  byte 0:   FIS type (0x27)
  byte 1:   C=1 (command), port multiplier field
  byte 2:   ATA command register
              0x35 = WRITE DMA EXT  (write sectors)
              0xEA = FLUSH CACHE EXT  ← issued by fdatasync
  byte 3:   Features
  bytes 4–9: LBA (48-bit logical block address)
  bytes 10–11: sector count
  …

NCQ (Native Command Queuing): SATA allows up to 32 outstanding FIS commands in the drive’s internal queue simultaneously. The drive firmware reorders them for optimal access patterns (minimal seek distance for HDD; optimal NAND plane interleaving for SSD) and processes them out-of-submission order. The host submits up to 32 commands without waiting for each to complete — dramatically increasing throughput compared to single-command polling.

NVMe: SQE, CQE, doorbell registers, and MMIO

NVMe (Non-Volatile Memory Express) is the interface specification for PCIe- attached SSDs, designed from scratch to exploit parallelism at every level. Instead of AHCI’s single command register, NVMe uses ring buffers in host RAM that both the CPU and the SSD controller access via DMA:

Host RAM                              NVMe Controller (on the SSD PCB)
┌───────────────────────┐            ┌──────────────────────────────────┐
│  Submission Queue (SQ) │            │  ARM DMA engine                  │
│  [SQE₀][SQE₁][SQE₂]… │─ PCIe DMA ►│  reads SQEs from host RAM        │
│       head    tail ↑  │            │  programs NAND cells             │
│  (CPU writes SQEs here)│            │                                  │
└───────────────────────┘            │  writes CQEs to host RAM         │
┌───────────────────────┐            │  raises MSI-X interrupt          │
│  Completion Queue (CQ) │◄ PCIe DMA─│  (MSI-X: per-vector interrupts   │
│  [CQE₀][CQE₁][CQE₂]… │            │   without sharing an IRQ line)   │
│  (CPU reads CQEs here) │            └──────────────────────────────────┘
└───────────────────────┘

• SQE (Submission Queue Entry): a 64-byte command descriptor the CPU places in the SQ ring buffer. Fields include: opcode (0x01 = write, 0x02 = read, 0x00 = flush), nsid (namespace ID), slba (starting LBA), nlb (number of logical blocks), and prp / sgl (host memory buffer address for DMA).

• CQE (Completion Queue Entry): a 16-byte result the controller places in the CQ after executing an SQE. Fields: sq_head (how many SQEs the controller has consumed), status (0 = success, non-zero = error code), phase bit (toggles each ring wrap so software can tell new CQEs from old ones), and cid (Command ID, matching the original SQE).

• Doorbell register: after writing SQEs, the CPU signals the controller by writing the new SQ tail index to a doorbell register on the device.

• MMIO (Memory-Mapped I/O): the NVMe controller exposes its registers (including doorbell registers) as a memory region via the PCIe BAR (Base Address Register) — a physical address range that the BIOS maps into the CPU’s address space. The CPU writes to the doorbell by executing a plain store instruction (MOV [doorbell_addr], tail) — the PCIe interconnect routes the transaction to the controller’s register file. No special IN/OUT port instructions are needed; the address space itself is the communication channel.

NVMe supports up to 65,535 SQ/CQ queue pairs — one per CPU core — enabling fully lock-free parallel I/O. AHCI’s single-queue design is exactly the bottleneck NVMe was designed to replace.

User process
  │  write(fd, buf, n)
  │  ← enters kernel via SYSCALL (x86-64) or SVC (ARM64) instruction
  ▼
VFS layer  (Linux: fs/read_write.c → vfs_write → file->f_op->write_iter)
  │  dispatches to the filesystem's write implementation
  ▼
Filesystem  (e.g. ext4_file_write_iter → generic_file_write_iter)
  │  copy_from_user() — copies bytes from user VA space into kernel page cache
  │  marks affected struct page / folio as PG_dirty
  ▼
Page cache  (struct address_space, backed by XArray of struct folio)
  │
  │  ◄── write() RETURNS HERE ─────────────────────────────────────────────────
  │      bytes are in kernel DRAM; a power cut here loses them
  │
  │  (from this point, writeback is ASYNCHRONOUS and unordered with writes)
  ▼
Writeback thread  (kworker/flush-X:Y  /  bdi_writeback / wb_workfn)
  │  woken by: dirty_ratio threshold, dirty_expire_centisecs timer,
  │            or an explicit sync call
  │  iterates dirty folios → calls ->writepages() on the address_space
  ▼
Block layer  (submit_bio → blk_mq)
  │  I/O scheduler (mq-deadline / kyber / bfq / none)
  │  merges adjacent requests, reorders for rotational efficiency
  ▼
Device driver
  │  SATA/AHCI: builds FIS (Frame Information Structure),
  │             queues into NCQ (Native Command Queuing, up to 32 slots)
  │  NVMe:      writes SQE to Submission Queue,
  │             rings doorbell MMIO register → CQE arrives on completion
  ▼
Drive controller  (volatile DRAM write buffer inside the drive)
  │  drive sends completion interrupt → DMA transfer complete
  │  host OS marks bio as completed
  │
  │  ◄── WITHOUT fdatasync, the OS considers the write "done" HERE ───────────
  │      drive DRAM is still volatile; power cut = lost data
  │
  ▼
Persistent storage  (NAND flash / magnetic platter)
  │  drive programs NAND page (~100–300 µs) or writes to platter sector
  │
  │  ◄── fdatasync guarantees we are at least HERE before returning ──────────

fdatasync issues a FLUSH CACHE command that forces the drive to commit its DRAM buffer to persistent media before responding.

• SATA: ATA FLUSH CACHE EXT (opcode 0xEA)
• NVMe: FLUSH admin command (opcode 0x00)
• SCSI: SYNCHRONIZE CACHE(10)

fsync vs fdatasync vs sync_file_range

fdatasync is faster than fsync because it skips persisting the inode’s mtime and ctime. An inode (index node) is the kernel’s in-memory record of a file’s metadata — permissions, owner, size in bytes, and timestamps — stored as a struct inode in kernel memory and persisted in a reserved area of the filesystem. Every file has exactly one inode; directory entries hold the filename and a pointer to the inode number. On a journalled filesystem (ext4, XFS), flushing metadata requires an extra journal commit. For a WAL, you care that payload bytes are durable — not that the file’s modification timestamp is persisted.

In C:

int fd = open("wal.log", O_CREAT|O_RDWR|O_APPEND, 0644);
write(fd, hdr, 8);
write(fd, data, data_len);
fdatasync(fd);  // data-only flush — sufficient and faster than fsync
// fsync(fd);  // use if you also need the file size update to survive a crash

In Rust:

#![allow(unused)]
fn main() {
use std::fs::File;
file.sync_data()?;  // fdatasync — data only
file.sync_all()?;   // fsync    — data + metadata
}

In C++:

#include <unistd.h>
::fdatasync(fileno(fp));   // FILE* → fd
::fdatasync(raw_fd);       // raw POSIX fd

Dirty page writeback: the kernel’s async path

Between write() and an explicit sync, dirty pages live in the page cache and are flushed asynchronously by kernel worker threads, governed by sysctl tunables:

# Block writes once this % of RAM is dirty (hard throttle — write() stalls)
sysctl vm.dirty_ratio               # default: 20

# Start background writeback at this % of RAM (soft limit — no blocking)
sysctl vm.dirty_background_ratio    # default: 10

# Maximum age of a dirty page before forced writeback (centiseconds)
sysctl vm.dirty_expire_centisecs    # default: 3000  (30 seconds)

# How often the background writeback thread wakes up
sysctl vm.dirty_writeback_centisecs # default: 500   (5 seconds)

A storage engine under heavy writes can hit vm.dirty_ratio and find write() stalling — the kernel throttles the writing process until the writeback thread catches up. This manifests as irregular latency spikes.

RocksDB’s production tuning guide recommends absolute byte values instead of ratios, so behaviour is independent of total RAM:

sysctl -w vm.dirty_bytes=209715200            # 200 MB hard throttle
sysctl -w vm.dirty_background_bytes=104857600 # 100 MB background start

Absolute limits prevent a 512 GB server with a 20% ratio from accumulating 100+ GB of dirty pages before stalling.

O_DIRECT: bypassing the page cache

O_DIRECT directs the kernel to DMA data between the user-space buffer and the storage device, skipping the page cache entirely:

// Buffer, offset, and transfer size must ALL be aligned to the logical block
// size — query it with: ioctl(fd, BLKSSZGET, &lbs)  (usually 512 B or 4096 B)
void *buf;
posix_memalign(&buf, 4096, ALIGN_UP(8 + data_len, 4096));

int fd = open("wal.log", O_CREAT|O_RDWR|O_DIRECT, 0644);
memcpy(buf, hdr, 8);
memcpy((char *)buf + 8, data, data_len);

// Transfer size and file offset must also be block-aligned
write(fd, buf, ALIGN_UP(8 + data_len, 4096));

// O_DIRECT skips the page cache but NOT the drive's volatile DRAM cache.
// fdatasync is still required for full durability.
fdatasync(fd);

Advantages:

• No double-copy: user memory → page cache → device becomes user memory → device.
• No page cache pollution: WAL bytes are write-once; caching them wastes RAM that the B-tree read path could use.
• Predictable latency: no interaction with dirty_ratio throttling.

PostgreSQL uses O_DIRECT for WAL on Linux. RocksDB has use_direct_io_for_flush_and_compaction and use_direct_reads options.

macOS pitfall: O_DIRECT is not supported on macOS. Use fcntl(fd, F_NOCACHE, 1) instead. Portable code needs an #ifdef __APPLE__ guard.

O_DSYNC / O_SYNC: per-write durability

These flags make every write() behave as if immediately followed by fdatasync (O_DSYNC) or fsync (O_SYNC):

// Every write() blocks until data is durable — no separate fdatasync needed
int fd = open("wal.log", O_CREAT|O_RDWR|O_APPEND|O_DSYNC, 0644);
write(fd, hdr, 8);    // blocks until drive confirms persistence
write(fd, data, n);   // blocks until drive confirms persistence
// no fdatasync call needed

Tradeoff: each write() now includes a full storage round-trip (~100 µs for NVMe, ~10 ms for HDD). You cannot pipeline writes. For a WAL that calls fdatasync once per record anyway, O_DSYNC with two writes is equivalent but less efficient (two sync waits vs. one).

Drive write cache: the last hiding place

Even after write() returns and the kernel considers the I/O complete, bytes may still be in the drive’s volatile DRAM write cache — a 32–256 MB buffer inside the drive housing. The drive firmware uses it to:

1. Acknowledge I/O to the host immediately (reducing host-visible latency).
2. Coalesce and reorder writes internally for NAND wear leveling.

A power failure while bytes are in this buffer loses them permanently. fdatasync / fsync issue a FLUSH CACHE command before returning (see above), forcing the drive to drain its cache to persistent media.

Check whether your drive has a write cache:

hdparm -I /dev/sda | grep -i 'write cache'
# "Write cache" present → fdatasync issues FLUSH; critical for durability

Enterprise NVMe with Power Loss Protection (PLP) uses an on-board capacitor to flush the DRAM cache to NAND on power loss — making fdatasync’s FLUSH optional. Consumer drives almost never have PLP.

macOS: F_FULLFSYNC

macOS fsync(2) is explicitly documented as advisory:

Note that while fsync() will flush all data from the host to the drive (i.e. the “permanent storage device”), the drive itself may not physically write the data to the platters for quite some time.

F_FULLFSYNC is the only macOS API that guarantees a drive-level FLUSH:

#include <fcntl.h>
// macOS-only: issues drive FLUSH CACHE; blocks until drive confirms persistence
if (fcntl(fd, F_FULLFSYNC) == -1) {
    // F_FULLFSYNC fails on network filesystems (NFS, SMB) — fall back
    fsync(fd);
}

Go’s os.File.Sync() calls F_FULLFSYNC on Darwin since Go 1.12 — our implementation is correctly durable on macOS. SQLite has an explicit #if defined(__APPLE__) block; PostgreSQL checks for F_FULLFSYNC at compile time.

sync_file_range: pipelined WAL writes

sync_file_range lets you initiate writeback of a byte range asynchronously, enabling overlap of disk I/O with CPU work — the core of pipelined WAL design:

// Write batch N
write(fd, batch_n, n_len);
off_t off_n = batch_n_offset;

// Initiate async writeback of batch N (returns immediately, no durability yet)
sync_file_range(fd, off_n, n_len, SYNC_FILE_RANGE_WRITE);

// While batch N flushes, build and write batch N+1
write(fd, batch_n1, n1_len);

// Block until batch N's data has actually reached the device
sync_file_range(fd, off_n, n_len,
    SYNC_FILE_RANGE_WAIT_BEFORE |
    SYNC_FILE_RANGE_WRITE       |
    SYNC_FILE_RANGE_WAIT_AFTER);

// fdatasync flushes remaining dirty pages + commits the file size to the inode
fdatasync(fd);

MySQL InnoDB’s redo log writer and PostgreSQL’s WAL writer both use this pipelining pattern. The async initiation overlaps disk I/O with CPU work, cutting effective latency by up to 50% on write-heavy workloads.

Linux-only: sync_file_range is not in POSIX. macOS/BSD have no equivalent. Portable code falls back to fdatasync.

io_uring: truly async file I/O without blocking syscalls

The traditional fdatasync blocks the calling OS thread for ~100 µs (NVMe) or ~10 ms (HDD). Linux 5.1 introduced io_uring — a shared ring-buffer interface where userspace submits batches of operations and reaps completions without blocking or entering the kernel per operation.

The ring-buffer model

io_uring allocates two lock-free ring buffers in a memory region shared between userspace and the kernel (mapped into both address spaces via mmap). No data is copied through the syscall boundary — both sides read and write the same physical memory pages:

Userspace process                    Linux kernel
┌─────────────────────────────┐
│  SQE array (fixed-size)     │      ← CPU writes command descriptors here
│  [SQE₀][SQE₁][SQE₂]…       │
├─────────────────────────────┤
│  Submission Queue (SQ ring) │      ← CPU writes indices into SQE array,
│  indices: [0][1][2]…        │        advances SQ tail
│  head (kernel) tail (CPU)↑  │      ← kernel advances SQ head as it consumes
└─────────────────────────────┘

                                       kernel executes I/O asynchronously
                                       (DMA, NAND programming, etc.)

┌─────────────────────────────┐
│  Completion Queue (CQ ring) │      ← kernel writes CQE results here,
│  [CQE₀][CQE₁]…             │        advances CQ tail
│  head (CPU)↑  tail (kernel) │      ← CPU advances CQ head as it reads
└─────────────────────────────┘

• SQ (Submission Queue): a ring of indices into the SQE array. The CPU writes an SQE into the array and then pushes its index onto the SQ ring by advancing the tail. No syscall — just a memory write.
• CQ (Completion Queue): a ring of CQEs that the kernel fills when I/O completes. The CPU polls the CQ tail or waits via io_uring_enter() with IORING_ENTER_GETEVENTS.
• SQE (Submission Queue Entry): a 64-byte descriptor encoding one I/O operation. Key fields: opcode (read, write, fsync, …), fd, addr (userspace buffer address), len, off (file offset), user_data (an opaque 64-bit value you choose — echoed back in the matching CQE).
• CQE (Completion Queue Entry): a 16-byte result. Fields: user_data (matches the originating SQE), res (return value — bytes transferred, or -errno on error), flags.
• IORING_SETUP_SQPOLL: a dedicated kernel thread polls the SQ tail. The CPU can write SQEs and advance the tail — the kernel thread picks them up without a syscall, reducing hot-path overhead from ~1 µs per io_uring_enter() to ~10 ns per SQE.
• IOSQE_IO_LINK: chains SQEs — SQE N+1 is not started until SQE N completes successfully. Guarantees ordering: header bytes before payload bytes before the fsync.
• IOSQE_IO_DRAIN: a write barrier — the marked SQE is not dispatched until all prior SQEs in the ring have completed. Used on the fsync SQE to ensure data writes are durable before the sync is issued.

#include <liburing.h>

struct io_uring ring;
// IORING_SETUP_SQPOLL: kernel thread polls the SQ — zero syscalls for submission
io_uring_queue_init(256, &ring, IORING_SETUP_SQPOLL);

// Pre-register buffers to avoid per-operation VA mapping overhead
struct iovec iov[2] = {
    { .iov_base = hdr,  .iov_len = 8 },
    { .iov_base = data, .iov_len = data_len },
};
io_uring_register_buffers(&ring, iov, 2);

// SQE 1: write header (linked — SQE 2 starts only after SQE 1 completes)
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_write_fixed(sqe, fd, hdr, 8, file_offset, 0 /* buf index */);
sqe->flags |= IOSQE_IO_LINK;
sqe->user_data = TAG_WRITE_HDR;

// SQE 2: write payload (linked — SQE 3 starts only after SQE 2 completes)
sqe = io_uring_get_sqe(&ring);
io_uring_prep_write_fixed(sqe, fd, data, data_len, file_offset + 8, 1);
sqe->flags |= IOSQE_IO_LINK;
sqe->user_data = TAG_WRITE_DATA;

// SQE 3: fdatasync — drained (waits for ALL prior SQEs in the ring to finish)
sqe = io_uring_get_sqe(&ring);
io_uring_prep_fsync(sqe, fd, IORING_FSYNC_DATASYNC);
sqe->flags |= IOSQE_IO_DRAIN;   // write barrier
sqe->user_data = TAG_SYNC;

// Submit all three in one syscall (zero syscalls with SQPOLL if kernel is polling)
io_uring_submit(&ring);

// Thread is free to prepare the next record while I/O is in flight
// ... build next batch ...

// Reap completions when you actually need the result
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
if (cqe->res < 0) { /* handle error: -errno */ }
io_uring_cqe_seen(&ring, cqe);

IOSQE_IO_LINK creates a dependency chain — each SQE starts only after the previous one completes successfully. IOSQE_IO_DRAIN is a write barrier: the fsync is not dispatched until all prior SQEs in the ring have completed.

With IORING_SETUP_SQPOLL, a kernel thread spins on the submission queue so io_uring_submit requires no syscall at all on the hot path.

ScyllaDB’s entire storage engine is built on io_uring. RocksDB has experimental io_uring support (use_io_uring = true).

epoll / select / poll — why they do not apply to file I/O

This confusion comes up often. epoll, select, and poll are I/O readiness multiplexers: they tell a thread when a file descriptor has transitioned from “not ready” to “ready to read/write without blocking”. They work on objects backed by kernel queues that can genuinely stall — sockets, pipes, eventfd, timerfd, signalfd.

Regular files are always ready from the VFS perspective. write() to a regular file always succeeds immediately by copying into the page cache — it never blocks waiting for “space to become available” the way a TCP socket does. Therefore:

• epoll_ctl(epfd, EPOLL_CTL_ADD, regular_file_fd, ...) returns EPERM.
• select() / poll() on a regular file always report it as both readable and writable, regardless of actual disk state.

For true async file I/O, the options are:

POSIX AIO (<aio.h>): the standard async I/O API. You submit an aio_write() with a struct aiocb describing the operation, then poll with aio_error() or wait with aio_suspend(). On Linux, glibc implements POSIX AIO using an internal thread pool — it is not truly kernel-level async. On FreeBSD and macOS, kqueue notifications enable genuine kernel-level async completion.

libaio (Linux kernel AIO, <libaio.h>): a Linux-specific low-level API that submits I/O directly to the kernel’s async submission path. Unlike POSIX AIO, it works correctly with O_DIRECT (bypassing the page cache) and uses no hidden threads. PostgreSQL’s WAL writer and MySQL InnoDB’s redo log use libaio when O_DIRECT is enabled. It has been largely superseded by io_uring, which is safer and more featureful.

kqueue (BSD/macOS): a general-purpose kernel event notification interface — the BSD equivalent of Linux epoll. Combined with the EVFILT_AIO filter it delivers a notification when a struct aiocb AIO operation completes, enabling event-loop-style async file I/O on macOS without hidden threads.

Go’s os.File.Write issues a blocking write() syscall. The Go runtime detects the block, parks the goroutine, and hands the OS thread to another goroutine. This is not zero-copy async I/O — one OS thread is consumed per concurrently-blocked file write. For a single-writer WAL this is fine; for high-concurrency workloads, io_uring via cgo or a purpose-built library is the right tool.

Partial writes and tail corruption

Even with Sync, a crash can leave a partial record at the end of the WAL. Consider: the OS writes the length field (4 bytes), then crashes before writing the payload. On restart, the WAL contains a valid header that promises N bytes of payload, but the file ends after 4 bytes.

Our recovery logic detects this by checking that the file contains at least headerSize + length bytes at every record boundary. A truncated record at the end is treated as if the write never happened — the partially-written record is silently discarded and recovery stops there. Records before the truncation are still valid and are replayed normally.

CRC32 catches a different class of corruption: a record whose bytes were written completely but with a storage error (bit flip, bad sector). If the checksum does not match, recovery stops at that record — same policy as truncation.

Why only the tail can be corrupt

Every Append call ends with Sync() before returning. A successful return means all bytes of that record are on persistent media. Therefore:

• Records 0 … N−2: fully written and individually synced → permanently durable.
• Record N−1 (the last): may be truncated if the crash occurred during Append — after some bytes were written but before Sync returned.
• Records N and beyond: do not exist in the file.

The only ambiguous region is the tail. This property justifies the “stop at first corrupt record” recovery policy — it is not conservative; it is exact.

Group commit exception: engines that batch multiple records per fdatasync (PostgreSQL, MySQL, RocksDB) trade per-record durability for throughput. If a crash occurs during a batched sync, the entire batch may be absent — not just the last record. Recovery tracks a “known-good LSN” written at the end of each batch to know exactly how far to replay.

LSN (Log Sequence Number): a monotonically increasing integer assigned to each WAL record or batch, acting as a cursor into the log. “Replay from LSN 47182” means skip all records with LSN ≤ 47182 and apply all records with LSN > 47182. PostgreSQL encodes the LSN as a (segment_file_number, byte_offset) pair (e.g., 0/3000028); MySQL InnoDB uses a single 64-bit byte offset from the start of the redo log. RocksDB calls the equivalent a sequence number embedded in every key.

What “silent discard” means for the caller

When the last record is partially written, the Append call either:

• Returned an error (process killed cleanly mid-write), or
• Never returned (power loss).

In neither case did the caller receive nil. The WAL’s contract is:

Any Append that returned nil is guaranteed to be recovered. Any Append that did not return nil may or may not be present — callers must retry.

This is at-least-once delivery. Exactly-once semantics require sequence numbers at a higher layer (the memtable / MVCC layer deduplicates replays).

MVCC (Multi-Version Concurrency Control): a concurrency strategy where writers never overwrite existing data. Instead, they append a new version alongside the old one, stamped with a sequence number or transaction timestamp. Readers see the snapshot that was current at the start of their read, without blocking writers. For WAL recovery, MVCC matters because replaying the same record twice is safe: the second application of sequence number N is a no-op — the memtable already holds that version and ignores the duplicate.

CRC32 vs CRC32C vs stronger checksums

crc32.ChecksumIEEE uses the IEEE 802.3 polynomial. The probability of an undetected random error is 1/(2^32) ≈ 2.3 × 10⁻¹⁰ per record — negligible for a local WAL.

SIMD, SSE4.2, and AVX2: hardware-accelerated data processing

SIMD (Single Instruction, Multiple Data) is a CPU instruction class that applies one operation to multiple data elements in parallel using wide registers:

Scalar (one value per instruction):
  ADD r1, r2          →  one 32-bit addition

SSE2 (128-bit XMM registers — 4 × 32-bit lanes):
  PADDD xmm0, xmm1   →  four 32-bit additions in one clock cycle

AVX2 (256-bit YMM registers — 8 × 32-bit lanes):
  VPADDD ymm0, ymm1  →  eight 32-bit additions in one clock cycle

Modern x86 CPUs have 128-bit XMM registers (SSE family, all x86-64 since 2003) and 256-bit YMM registers (AVX/AVX2, Intel Haswell 2013+). ARM has 128-bit Neon registers plus a hardware CRC extension.

SSE4.2 (Intel Nehalem, 2008 — effectively all x86-64 hardware since ~2012) added one purpose-built instruction: CRC32. It computes the CRC32C (Castagnoli polynomial) checksum in hardware, one byte, two, four, or eight bytes at a time:

uint32_t crc = 0xFFFFFFFF;
// _mm_crc32_u64: one instruction that processes 8 bytes, ~1 byte/cycle throughput
crc = _mm_crc32_u64(crc, *(uint64_t*)data);
// At 4 GHz: ~32 GB/s checksum throughput — 5–10× faster than a software table

ARM processors with the CRC extension (ARMv8.1-A, all Apple Silicon) provide __crc32cd with equivalent throughput.

AVX2 (Intel Haswell, 2013) provides 256-bit integer SIMD. It does not directly accelerate CRC32 but enables vectorized hash functions like xxHash3 and BLAKE3 to process 32 bytes per instruction cycle — relevant when the checksum algorithm is hash-based rather than polynomial-based.

CRC32C detects more error patterns and has native CPU instructions on all modern x86 (SSE4.2) and ARM (CRC extension) processors.

In C with SSE4.2:

#include <nmmintrin.h>  // SSE4.2

// Process 8 bytes at a time for throughput; 1 byte for simplicity:
uint32_t crc32c(const uint8_t *data, size_t len) {
    uint32_t crc = 0xFFFFFFFF;
    while (len >= 8) {
        crc = _mm_crc32_u64(crc, *(uint64_t *)data);
        data += 8; len -= 8;
    }
    while (len--) crc = _mm_crc32_u8(crc, *data++);
    return crc ^ 0xFFFFFFFF;
}

In Rust (auto-selects SSE4.2 or software fallback via crc32fast crate):

#![allow(unused)]
fn main() {
use crc32fast::Hasher;
let mut h = Hasher::new();
h.update(&data);
let checksum = h.finalize();
}

LevelDB and RocksDB switched from CRC32 (IEEE) to CRC32C for exactly these reasons. A production WAL should use CRC32C.

Sparse files and zero-padding: an edge case

On filesystems that support sparse files (ext4, XFS, APFS), a failed write may leave a hole — a region that reads as zeros but consumes no disk space. The recovery scanner would interpret those zeros as valid payload bytes, potentially computing a CRC over all-zero data.

If crc32(zeros, n) == stored_checksum (astronomically unlikely, but theoretically possible), a corrupt or missing record could pass the CRC check.

Production WALs protect against this with a RecordType byte in the header:

┌────────────┬──────────┬────────────┬─────────────────────┐
│ type (1 B) │ len (4 B)│ crc32 (4 B)│ payload (len bytes) │
└────────────┴──────────┴────────────┴─────────────────────┘

A zero type byte (0x00) is immediately recognisable as a hole rather than a valid record, stopping recovery without any CRC computation. LevelDB’s log format uses RecordType values kFullType, kFirstType, kMiddleType, kLastType to support records that span 32 KB blocks.

WAL file after partial crash:

  ┌──────────────────────────────────┬────────────────────────┬────────────┐
  │  record 0  (complete, CRC ok)    │  record 1  (complete)  │  trunc...  │
  └──────────────────────────────────┴────────────────────────┴────────────┘
                                                                ↑
                                                  recovery stops here,
                                                  records 0 and 1 replayed

File format

Each record occupies a contiguous run of bytes:

┌────────────────┬────────────────┬──────────────────────┐
│  length (4 B)  │  CRC32  (4 B)  │  payload  (length B) │
│   uint32 LE    │  IEEE/LE       │  raw bytes           │
└────────────────┴────────────────┴──────────────────────┘

• length — byte count of the payload.
• CRC32 — IEEE checksum of the payload only (not the header). Detects silent data corruption during recovery.
• payload — arbitrary bytes; in later labs this encodes a sequence number, key type, key, and value.

Recovery reads records one at a time. If a header or payload is truncated, or the CRC32 does not match, recovery stops silently — exactly modelling a crash where the last write was incomplete.

Implementation notes

Why two separate Write calls in Append?

Append issues two write() syscalls — one for the 8-byte header and one for the payload — rather than concatenating them into a single buffer first:

w.f.Write(hdr[:])   // fixed-size [8]byte allocated on the stack
w.f.Write(data)     // caller-supplied []byte

Reason 1: zero allocation

Combining into a single slice requires:

buf := make([]byte, 8+len(data))  // heap allocation + GC pressure
copy(buf[:8], hdr[:])
copy(buf[8:], data)
w.f.Write(buf)

For a WAL called on every mutation this is a heap allocation on the critical write path. Two separate writes eliminate it entirely. The Go escape analyser confirms hdr stays on the stack when used as [8]byte — zero heap traffic.

Reason 2: no zero-copy join for stack array + slice

hdr is a [8]byte — a fixed-size array on the stack. data is a []byte — a heap-backed slice. The stack and heap regions are not adjacent in memory. There is no way to present them as a single contiguous buffer to the OS without allocating a new region and copying both into it.

The proper fix: scatter-gather I/O (writev)

POSIX provides writev(2) — a single syscall that atomically writes from multiple non-contiguous buffers:

#include <sys/uio.h>

struct iovec iov[2];
iov[0].iov_base = hdr;
iov[0].iov_len  = 8;
iov[1].iov_base = data;
iov[1].iov_len  = data_len;

// One syscall; OS appends both buffers atomically w.r.t. O_APPEND offset
ssize_t written = writev(fd, iov, 2);

This achieves zero allocation (no copy needed) and halves the syscall count. On a WAL writing millions of records per second, two syscalls vs. one is measurable — each syscall involves a SYSCALL instruction (x86-64) or SVC (ARM64), a ring-0 privilege-level switch, TLB management, and a return.

TLB (Translation Lookaside Buffer): a small hardware cache inside the CPU’s MMU (Memory Management Unit) that stores recently-used virtual→physical address translations. Every syscall that passes a buffer pointer to the kernel requires the kernel to translate that virtual address to a physical one. If the mapping is not in the TLB, the MMU walks the multi-level page tables in RAM — ~20–100 extra cycles. Pre-registering buffers with io_uring (io_uring_register_buffers) pins their physical addresses in the kernel’s DMA mapping, eliminating this per-submission TLB overhead.

Go exposes writev indirectly through net.Buffers:

bufs := net.Buffers{hdr[:], data}
bufs.WriteTo(w.f)  // calls writev(2) on Linux/macOS — zero alloc, one syscall

The tradeoff: dangling header on crash

A crash between the two writes leaves a valid 8-byte header followed by no payload bytes. Recovery encounters this in Recover:

data := make([]byte, length)
if _, err := io.ReadFull(f, data); err != nil {
    // io.ErrUnexpectedEOF — truncated payload
    break  // silently discard and stop
}

The partial record is discarded. This is the correct, intended behaviour.

Importantly, writev does not eliminate this risk. A single writev call is not crash-atomic — the kernel can flush the first iov (the header) to disk before the second iov (the payload), leaving the same dangling-header state. The only true atomic record boundary is the fdatasync at the end: once it returns, the complete header+payload record is durable.

The two-write vs. one-write vs. writev distinction affects syscall overhead only — not crash safety.

Why production engines still use two separate writes

RocksDB, LevelDB, and SQLite all use separate writes for header and payload:

• writev requires building an iovec array — a small but real cost per call. An iovec is a { void *iov_base; size_t iov_len; } pair describing one buffer segment; writev accepts an array of them and transfers all segments atomically in one syscall.
• The fdatasync that follows dominates latency by 3–4 orders of magnitude (~100 µs NVMe, ~10 ms HDD vs. ~100 ns for an extra write() syscall that hits the page cache).
• The dangling-header case is handled gracefully by recovery in all these engines anyway — the two-write approach has no correctness cost.

io_uring with linked SQEs (shown in the Sync section above) is the modern answer for truly zero-overhead, batched, async record writes.

Self-framing: no delimiters needed

There are no separator bytes between records. The file is a dense stream:

[hdr₀][payload₀][hdr₁][payload₁][hdr₂][payload₂]...

Recover can parse it because the format is length-prefixed (also called TLV — Tag/Length/Value):

1. Read 8 bytes → parse length and checksum.
2. Read exactly length bytes → that is the complete payload.
3. Verify CRC32; if mismatch, stop.
4. File cursor is now exactly at the start of the next record.
5. Repeat from step 1.

The length field encodes the exact byte count of the payload that follows it — the reader always knows where one record ends and the next begins. No sentinel or delimiter byte is needed.

Why length-prefix beats delimiters

Delimiter-based framing (newlines, \0, magic sequences) has a fundamental flaw: the delimiter must not appear in the payload. Either you forbid certain byte values (restricting what can be stored), or you escape them (adding complexity and variable overhead).

Who else uses this framing

Every major binary protocol and serialization format uses length-prefix framing:

LevelDB’s block-based WAL framing

LevelDB adds a refinement on top of simple length-prefix: it divides the file into fixed 32 KB blocks. A record that spans a block boundary is split into kFirstType, kMiddleType, and kLastType fragments:

Block 0 (32 KB):                    Block 1 (32 KB):
┌──────────────────────────────┐    ┌──────────────────────────────┐
│ hdr(kFullType)  │ record A   │    │ hdr(kLastType) │ record B... │
│ hdr(kFirstType) │ record B   │    │ hdr(kFullType) │ record C   │
│ ...continues... │            │    │                │            │
└──────────────────────────────┘    └──────────────────────────────┘

Advantages of the block layout:

• Recovery can seek to any 32 KB boundary and resume scanning — critical for log files that have grown to GB scale.
• A corrupt block can be skipped without losing records in adjacent blocks.
• Block-aligned I/O maps naturally to filesystem block size and SSD page size, reducing partial-block writes and improving read performance.

Our simple variable-length format is correct and sufficient for a lab. A production WAL should consider block alignment for recovery efficiency on large files.

Running the tests

cd leveldb
go test ./lab01/... -v -count=1

Expected output:

=== RUN   TestAppendRecover
--- PASS: TestAppendRecover (0.00s)
=== RUN   TestCRCDetects
--- PASS: TestCRCDetects (0.00s)
=== RUN   TestEmptyFile
--- PASS: TestEmptyFile (0.00s)
PASS
ok      github.com/10xdev/leveldb/lab01

Running the demo

go run ./lab01/demo

Expected output:

wrote: "apple"
wrote: "banana"
wrote: "cherry"

--- simulated crash ---

recovered 3 records:
  [0] "apple"
  [1] "banana"
  [2] "cherry"

FoundationDB parallel

In FoundationDB, the transaction log plays this role automatically. Every committed transaction is durably written to multiple transaction log processes before the commit returns. You never interact with it directly — crash recovery is handled transparently by the cluster.

The key difference: FDB’s transaction log is replicated across at least two machines (configurable), so a single-machine failure does not lose any commits. Our WAL is a single-process approximation of the same guarantee.

The FDB storage server also maintains a WAL internally (the “mutation log”) to protect in-memory B-tree mutations. The layering is: FDB transaction log (cross-machine durability) → storage server mutation log (single-machine durability) → B-tree pages (the permanent on-disk state). Our single WAL collapses all three into one.