Deterministic Simulation: Reproducing Rare Failures in Distributed Systems

A cluster may occasionally lose a write after an improbable sequence of events: a node restarts, a packet is delayed, an election begins at the wrong moment and a disk responds in an unexpected order. The test fails once in several thousand runs, then passes as soon as an engineer enables diagnostic logging. How can a team fix a failure it cannot replay?

Deterministic simulation offers a more ambitious answer than simply rerunning the test with the same seed. It places every decision that could alter execution under the control of the test harness. Time, networking, storage, task scheduling and randomness become controllable inputs. A rare failure can then be replayed, slowed down and examined one event at a time. But reproducibility is useful only if the simulator can also explore meaningful scenarios and recognize a genuine violation. The entire architecture behind this capability therefore matters.

Why a Random Seed Is Not Enough

A seed is the initial value supplied to a pseudorandom number generator, or PRNG: an algorithm that produces a sequence of apparently random values that is fully determined by its initial state. Given the same seed and the same calls in the same order, the generator returns the same sequence. It can therefore select the same node to stop or the same packet to drop during a replay.

That contract breaks as soon as one decision escapes the generator. Two threads may be scheduled differently by the operating system. A timeout may expire a few microseconds earlier. A read from /dev/urandom may return a new value. The network, file system or an external service may respond in a different order. That single deviation then changes the number and order of subsequent calls to the PRNG. The seed remains the same, but the history diverges.

FoundationDB's client testing documentation states this limitation precisely: replay works when every behavior that influences execution is deterministic. A workload that creates ordinary threads can reintroduce uncontrolled concurrency and make a failure difficult to reproduce.

Two properties must therefore be distinguished. The first is replay: a particular execution can be reproduced. The second is exploration: the test harness can generate many different executions and reach unusual states. A seed may identify one path, but it proves neither that the path will remain stable in the presence of external inputs nor that the explored state space contains the most dangerous situations.

Deterministic simulation must consequently be treated as an execution-control architecture. The seed is only its entry key.

Controlling Five Sources of Nondeterminism

FoundationDB illustrates the most integrated version of this approach. Its actual database code can run with synthetic workloads and injected faults inside a discrete-event simulation. In this model, time does not advance continuously. The engine selects the next scheduled event, moves the clock to that event's time, executes it and repeats the process. The FoundationDB technical paper describes abstractions for networking, disk, time and the pseudorandom generator, with several simulated servers running in a single process.

Virtualizing Time

Reading the machine clock directly makes results dependent on processor speed, system load and unexpected pauses. A virtual clock replaces that dependency. Timeouts, retransmission delays, leases and deadlines must all consult this controlled clock.

The simulator can then accelerate periods with no activity. If the only remaining timer is scheduled for one virtual hour in the future, there is no reason to wait for a real hour: the clock jumps to the next event. This makes it possible to test mechanisms whose normal delays would be prohibitively long in a conventional environment.

Control must also extend into business logic whenever time can affect its result. The TigerBeetle architecture explains that its state machine does not read system time directly. Instead, it receives a timestamp determined by the protocol, ensuring that the same input retains the same logical result and execution path.

Intercepting Networking and Storage

A simulated network does more than deliver messages in memory. It must be able to delay, drop, duplicate or reorder communications according to the model under test. It can also represent a partition, meaning that some groups of nodes are temporarily unable to communicate with one another.

Storage requires a comparable abstraction. Input/output operations, commonly abbreviated as I/O, must pass through an interface that the simulator can delay or cause to fail. Depending on the guarantees claimed by the software, the model may also expose incomplete writes or corrupted data. The objective is not to reproduce all the physical behavior of a disk, but to make explicit the behaviors the algorithm is supposed to withstand.

This leads to a design rule: no meaningful network or storage access should bypass the controlled layer. A single direct call introduced during a refactoring can create a new source of divergence.

Controlling Scheduling and Entropy

Scheduling is the choice of which task is allowed to make progress next. In a multithreaded program, that choice normally depends on the operating system and hardware. FoundationDB avoids this difficulty in its simulator by using single-threaded execution and Flow's actor model. An actor is a logical unit that processes events and communicates through messages, allowing the engine to choose explicitly the order in which actors advance.

Another option is to control the interleaving of existing threads. This is one of the roles of the environment described by Antithesis. Whatever mechanism is selected, the platform must be able to observe the points at which a task can yield control.

Finally, all entropy, meaning any data used as a source of randomness, must be centralized. The Antithesis environment replaces the Linux devices /dev/random and /dev/urandom with entropy supplied by the platform, isolates containers from the external network and can accelerate idle periods. Its documentation advises against caching that entropy in an application-level PRNG, because doing so would give the platform fewer opportunities to branch the execution history and explore alternative paths.

Exploring Failures Instead of Replaying the Same History

A perfectly deterministic simulation may be perfectly useless if it keeps repeating trivial scenarios. The harness must vary topology, load, configuration and the timing of faults. This is where fuzzing becomes relevant: the technique automatically generates or transforms inputs to trigger unexpected behavior.

In a distributed system, many dimensions can be combined: client requests, cluster size, latency, restarts, message loss, I/O errors and role changes. FoundationDB randomizes configurations, workloads and fault parameters. Its approach also includes “buggification,” which deliberately activates branches that simulate errors, and swarm testing, in which different features or faults are enabled in varying combinations. These mechanisms are described in its architecture and testing paper.

Generation must nevertheless follow an understandable model. Stopping every node forever demonstrates only that a cluster with no available machines cannot progress. An interesting failure brings the system close to the boundary of its guarantees: a majority remains available, a node restarts during recovery, corruption is limited to one operation, or a partition heals after a series of concurrent decisions.

One practical strategy is to separate three layers. The first produces normal client actions. The second selects faults and their timing. The third controls the environment, including message delivery, the advancement of time and task wake-ups. All three consume a recordable source of decisions. When a failure occurs, the report should retain at least the seed, configuration, scenario and type of oracle that was violated.

Replay can then be enhanced through scenario reduction. This process searches for a shorter sequence that preserves the failure: fewer requests, fewer affected nodes or fewer faults. Even when reduction is not fully automated, determinism makes it possible to remove an event and verify unambiguously whether the defect remains.

Detecting Errors with Safety and Liveness Oracles

Crashing processes does not constitute a complete test. The harness must determine what counts as a correct outcome. An oracle is the mechanism that makes this judgment, often by relying on an invariant: a property that must remain true in every relevant state.

Safety properties express that something forbidden never occurs. Depending on the product, an oracle might verify that two contradictory values are not both committed, that a transaction honors its stated guarantees or that an accounting quantity is conserved. FoundationDB combines transactional invariants, local assertions and cluster recovery checks, according to its reference paper.

Liveness properties ask a different question: does the system eventually make progress once the required conditions are present? A cluster can preserve all its data and still remain stuck. It can also enter a livelock, in which components continue performing actions, but those actions cancel one another out or repeat without producing useful progress.

Liveness cannot be tested properly by injecting unlimited maximum disruption. If every attempt is systematically interrupted, a lack of progress is expected. The oracle's assumptions must be stated explicitly. For example, sufficient connectivity has been restored and a quorum, meaning the minimum number of members required to make a collective decision, remains stable for a specified period of virtual time.

TigerBeetle's VOPR simulator separates these concerns. Its safety mode can randomly inject crashes, packet loss and I/O corruption. For liveness testing, the scenario stabilizes a quorum while retaining some peripheral failures, then verifies that the core of the cluster continues to progress. TigerBeetle's account of simulation testing for liveness shows why this controlled phase can reveal stalls that permanent chaos may conceal.

The distinction also improves diagnosis. A safety violation should be reported as soon as the forbidden state appears. A liveness violation instead requires an observation window and explicit stability conditions. Combining the two leads either to misleading alerts or to tests that wait forever.

Two Strategies: Designing for Simulation or Wrapping Existing Software

The first strategy is to make software simulatable by design. Nondeterministic dependencies are placed behind interchangeable interfaces. Production code calls an abstraction for time, transport, storage or randomness; the real environment provides the normal implementation, while the simulator supplies a controlled version.

FoundationDB takes this model particularly far: its real code runs with multiple simulated servers inside a deterministic process. TigerBeetle likewise treats determinism as an architectural principle. Its VOPR simulator can run a cluster on a single thread, accelerate time, replace I/O with faulty models and reproduce a failure from its seed, as explained in its architecture documentation.

This approach provides fine-grained control and makes system boundaries visible. It also demands lasting discipline. Libraries used by the tested code must not quietly start a thread, consult the real clock or produce their own randomness. Code reviews must treat simulability as an architectural property, not as a convenience reserved for testing.

The second strategy places more conventional software inside a deterministic environment. A hypervisor is a layer that controls the execution of machines or virtualized environments. In the approach presented by Antithesis, it can control elements such as clocks, thread interleavings, faults and sources of randomness without requiring every component to be rewritten around simulation interfaces.

The advantage is clear for an existing stack made up of several services or dependent on Linux behavior. The nature of control is different, however. Instead of injecting a disk interface directly into business logic, the platform controls the environment as seen by the software. This method does not remove the need to instrument the properties being checked. A hypervisor can explore and replay an execution, but it cannot infer on its own that a balance, index or protocol has violated its semantics.

The choice is therefore not purely technical. A team building a new distributed engine can make determinism a structural constraint. For a collection of services already in production, a virtualized wrapper may reduce adoption costs. Hybrid approaches are also possible: an algorithmic core designed for simulation, followed by integration testing in a broader controlled environment.

Building a Replayable Harness Without Recreating the Entire Internet

The first task is to inventory every nondeterministic input. Teams should look for clock reads, random generators, threads, timers, network calls, file-system access, child processes and external dependencies. Each element must either be controlled, replaced or explicitly excluded from the scope of replay.

A minimal architecture can then be built in stages:

1. Create a central event loop. It owns the virtual clock and an ordered queue of actions to execute. Ties must be resolved using a deterministic rule.
2. Inject dependencies. Code receives interfaces for time, networking, storage and randomness instead of calling the operating system directly.
3. Centralize decisions. Workload, fault and scheduling choices must derive from a controlled source and be associated with a seed.
4. Define failure models. Each fault should correspond to a useful assumption: a lost message, stopped node, rejected write or delayed response. The model must remain readable and configurable.
5. Write invariants before multiplying scenarios. Without an oracle, increasing the number of executions mainly increases the amount of behavior that goes unanalyzed.
6. Produce a replay artifact. A seed is not always enough to communicate an incident. The configuration, enabled options and failed scenario must also be retained.

It is preferable to begin with one narrow property and a simple model. For example, use three nodes, one request type, network delays and controlled crashes. Once replay is stable, the scope can expand to include partitions, storage errors and richer workloads. Introducing every possible fault at the outset makes it difficult to determine whether a divergence comes from the product or from the simulator.

The harness itself must also be tested. A small error in the storage model can create an unrealistic guarantee, while a scheduler that overlooks some events can fabricate a false deadlock. Focused tests of the abstractions, along with internal assertions about the consistency of the event queue and clock, reduce this risk.

Conclusion: Complementing Simulation with Real Interfaces

Deterministic simulation provides deep visibility into the algorithmic paths it controls, but it does not automatically reproduce every behavior of the real world. A simulated network interface does not necessarily validate the actual transport stack. A disk model tests neither the driver nor the file system nor the exact behavior of the hardware. A single-threaded execution does not expose every data race in a multithreaded client.

TigerBeetle explicitly acknowledges this boundary. Its deterministic test replaces network transport and the storage interface with stubs, meaning simplified implementations used in place of real components. The project complements it with Vörtex, a nondeterministic harness that exercises compiled binaries, client drivers and language bindings from the outside. Its article about Vörtex presents this combination as a response to the simulator's blind spots.

FoundationDB follows a similar logic. Its client testing documentation explains that some multithreaded behaviors are not suitable for single-threaded simulation and are instead covered by end-to-end API tests against a real cluster.

The right strategy is therefore defense in depth. Deterministic simulation rapidly explores rare sequences, accelerates time and turns a fleeting failure into a replayable scenario. Tests on real binaries and infrastructure validate the boundaries replaced by the model. Performance testing must remain grounded in representative environments because virtual time cannot measure physical latency.

The decisive benefit is not merely the ability to rerun a test with the same seed. It is the ability to answer three precise questions: which decisions led to the failure, which property was violated and which parts of the result still depend on a simplified model. When a test harness can provide those answers, rare distributed failures stop being irreproducible anecdotes. They become durable test cases.

Daymain Team