A cluster can lose a write after a sequence of events that is almost impossible to trigger deliberately: one node restarts, a packet is delayed, an election begins at exactly the wrong moment and the disk responds in an unexpected order. The test fails once in several thousand runs. Then it passes as soon as diagnostic logging is enabled.
For the team, the problem is not only finding the defect. It must be possible to see it again. Without that, every fix rests on a hypothesis, and every new run may tell a different story.
Deterministic simulation addresses this problem by controlling the decisions that can change execution. The clock, network, storage, task scheduler and sources of randomness become inputs to the test harness. A rare failure can then be slowed down, inspected event by event and replayed from the same decisions.
Reproducibility, however, is only half the subject. A simulator that always replays trivial scenarios will not find interesting defects. It must also explore varied combinations and know how to recognize a real violation. That is what separates controlled repetition from a genuine distributed-systems testing tool.
A Seed Only Replays What Is Controlled
A seed is the initial value supplied to a pseudo-random number generator, or PRNG. The generator produces a sequence of values that appears random but is fully determined by its initial state. Given the same seed and the same calls in the same order, it returns the same sequence. The test harness can therefore choose the same node to stop, the same packet to drop or the same configuration to apply.
That promise ends as soon as an important decision escapes the generator. Two threads may be scheduled differently by the operating system. A timer may expire a few microseconds earlier. A read from /dev/urandom may return a different value. The network, file system or an external service may also respond in another order.
The first divergence is enough. It changes the sequence of events, and then the number and order of calls to the PRNG. The seed has not changed, but the history has.
The FoundationDB documentation on client testing makes this condition explicit: replay works when every behavior capable of influencing execution remains deterministic. A workload that creates ordinary threads can reintroduce uncontrolled concurrency and make a failure difficult to reproduce.
Two properties that are often confused therefore need to be separated:
- Replay: a given execution can be reproduced identically.
- Exploration: the harness can produce many different executions and reach unusual states.
A seed can identify one precise path. It does not guarantee that the path will remain stable when external inputs change, nor that the explored paths include the most dangerous situations. A seed is an access key to a controlled execution, not a complete testing strategy.
Deterministic simulation must therefore be designed as a control architecture. The random generator is only the starting point.
In plain language
A simulation is truly replayable only when every decision capable of influencing execution passes through deterministic mechanisms.
The Five Frontiers of Nondeterminism
FoundationDB shows what deep integration looks like. Its real database code can run with synthetic workloads and injected faults inside a discrete-event simulation. Time does not move continuously. The engine selects the next scheduled event, advances the clock to its timestamp, executes the event and repeats.
The technical paper on FoundationDB describes abstractions for the network, disk, time and pseudo-random generator, with several simulated servers running in one process. The FoundationDB testing documentation also presents deterministic single-threaded execution, simulated time, network and machine models, and the rare-failure scenarios used by the project.
Replace the Real Clock
A direct read from the machine clock makes results depend on processor speed, system load and unexpected pauses. A virtual clock removes that dependency. Timers, retransmission delays, leases and deadlines must all consult the controlled clock.
The simulator can then accelerate periods of inactivity. If the only remaining action is a timer scheduled for one virtual hour from now, it does not need to wait an actual hour. The clock jumps to the next event. Mechanisms whose delays would be prohibitive in a conventional environment become quick to test.
Control must reach business logic whenever time changes its result. The TigerBeetle architecture states that its state machine does not read the system clock directly. A timestamp determined by the protocol is injected instead. The same input therefore preserves the same logical result and execution path.
Model the Network and Storage
A simulated network should do more than deliver in-memory messages. It must be able to delay, drop, duplicate or reorder communications according to the model under study. It must also represent a partition: for a period of time, some groups of nodes can no longer communicate.
Storage requires an abstraction of the same kind. Input/output operations, often abbreviated as I/O, must pass through an interface that the simulator can delay or fail. Depending on the guarantees the software claims to provide, the model may expose incomplete writes or corrupted data.
The goal is not to reproduce every detail of disk physics. It is to make explicit the behaviors the algorithm claims to withstand. A direct access added during a refactoring can bypass this layer and create a new source of divergence.
Make Scheduling Visible
Scheduling is the choice of which task is allowed to make progress next. In a multithreaded program, that choice normally depends on the operating system and hardware. It can therefore vary from one execution to the next, even when application inputs are identical.
FoundationDB avoids this difficulty in its simulator through single-threaded execution and the Flow actor model. An actor is a logical unit that processes events and communicates through messages. The engine can thus choose explicitly the order in which actors advance.
Another approach is to control the interleaving of existing threads. That is one of the roles of the environment described by Antithesis. In either case, the points where a task can yield control must be visible to the platform.
Centralize Entropy
Entropy here means any data used as a source of randomness. It must be centralized, or at least intercepted. The Antithesis environment replaces the Linux devices /dev/random and /dev/urandom with entropy supplied by the platform. It also isolates containers from the outside network and can accelerate idle periods, as described in its environment documentation.
Antithesis advises against caching this entropy in an application-level PRNG. The platform would then have fewer opportunities to branch the history and explore other paths. This detail illustrates an important tension: the source of randomness must be replayable, but it must also remain accessible to the exploration engine.
Explore Failures That Matter
A perfectly deterministic simulator can be perfectly useless if it repeats obvious scenarios. The harness must vary topology, load, configuration and the timing of faults. Fuzzing works at this level: the technique automatically generates or transforms inputs to provoke unexpected behavior.
In a distributed system, the dimensions combine quickly: client requests, cluster size, latencies, restarts, message loss, I/O errors and role changes. FoundationDB randomizes configurations, workloads and fault parameters. Its approach also includes “buggification,” which deliberately enables branches that simulate errors, and swarm testing, where different features or faults are activated through varying combinations. These mechanisms are described in its architecture and testing paper.
Generation must nevertheless remain intelligible. Stopping every node forever only shows that a cluster without machines makes no progress. A useful failure places the system near the boundary of its guarantees:
- a majority remains available;
- a node restarts during recovery;
- corruption is limited to one operation;
- a partition heals after a sequence of concurrent decisions.
A practical strategy is to separate three layers. The first produces normal client actions. The second chooses faults and their timing. The third drives the environment: message delivery, the passage of time and the waking of tasks. All three layers must consume a source of recordable decisions.
When a failure occurs, the report should retain at least the seed, configuration, scenario and type of oracle that was violated. The seed alone does not always give another team enough context to reconstruct the run.
Replay can then help reduce the scenario. Reduction seeks a shorter sequence that preserves the failure: fewer requests, fewer affected nodes or fewer faults. Even when it is not fully automated, determinism makes it possible to remove one event and check unambiguously whether the defect remains. This turns a trace of several thousand events into a test case that people can use.
Time, networking, storage, concurrent scheduling and sources of entropy must be virtualized or intercepted.
Distinguish Safety from Liveness
Crashing processes is not a complete test. The harness must define what counts as a correct result. An oracle is the mechanism that makes that judgment. It often relies on an invariant: a property that must remain true in every relevant state.
A safety property says that something forbidden never happens. Depending on the product, the oracle may check that two contradictory values are never both accepted, that a transaction respects its stated guarantees or that an accounting quantity is preserved. According to its reference paper, FoundationDB combines transactional invariants, local assertions and checks on cluster recovery.
Liveness asks a different question: does the system eventually make progress when the necessary conditions are present? A cluster can preserve all its data and remain stuck. It can also enter a livelock: components continue taking actions, but those actions repeat or cancel one another without producing useful progress.
Liveness cannot be tested properly with maximum, permanent disruption. If every attempt is interrupted, the absence of progress is normal. The oracle must state its assumptions. Has sufficient connectivity been restored? Does a quorum—the minimum number of members required for a collective decision—remain stable for a given virtual duration?
TigerBeetle’s VOPR simulator separates these concerns. Its safety mode can randomly inject crashes, packet loss and I/O corruption. For liveness, the scenario stabilizes a quorum while maintaining certain peripheral failures, then checks that the core of the cluster continues to progress. The TigerBeetle account of liveness testing explains why this controlled phase reveals deadlocks that permanent chaos can hide.
The distinction directly improves diagnosis. A safety violation should be reported as soon as the forbidden state appears. A liveness violation instead requires an observation window and explicit stability conditions. Mixing the two leads either to misleading alerts or to expectations that never terminate.
Two Ways to Make a System Simulatable
The first strategy is to design the software for simulation. Nondeterministic dependencies are placed behind interchangeable interfaces. Production code calls an abstraction for time, transport, storage or randomness; the real environment supplies the normal implementation, while the simulator supplies a controlled one.
FoundationDB takes this model a long way: its real code runs with several simulated servers in one deterministic process. TigerBeetle likewise makes determinism an architectural principle. Its VOPR simulator can run a cluster on one thread, accelerate time, replace I/O with failure models and reproduce a failure from its seed, as explained in its architecture documentation.
The benefit is fine-grained control over the system’s boundaries. The cost is constant discipline. A library used by the tested code must not quietly start a thread, read the real clock or generate its own randomness. Code reviews therefore need to treat simulability as an architectural property, not as a testing convenience.
The second strategy places more conventional software in a deterministic environment. A hypervisor is a layer that controls the execution of virtualized machines or environments. In the approach presented by Antithesis, it can control the clock, thread interleaving, faults and random sources without requiring every component to be rewritten around simulation interfaces.
This is useful for an existing stack made up of several services or dependent on Linux behavior. The control operates at a different level, however. Instead of injecting a disk interface into business code, the platform controls the environment the software sees.
This method does not remove the need to instrument the properties being checked. A hypervisor can explore and replay an execution; it cannot know by itself that a balance, index or protocol has violated its semantics.
The right choice therefore depends on the system. A team building a new distributed engine can make determinism a structural constraint. For an existing set of services, a virtualized envelope may reduce the cost of adoption. A hybrid approach is also possible: design the algorithmic core for simulation, then test the larger assembly in a controlled environment.
What to keep in mind
A seed reproduces a controlled execution, but it does not guarantee useful state-space exploration or error detection.
Build a Replayable Harness Step by Step
The first task is to inventory nondeterministic inputs. Search for clock reads, random generators, threads, timers, network calls, file-system access, child processes and external dependencies. Each item must be controlled, replaced or explicitly excluded from the replay boundary.
A minimal architecture can then develop in stages:
- Create a central event loop. It owns the virtual clock and an ordered queue of actions. Ties must be resolved by a deterministic rule.
- Inject dependencies. Code receives interfaces for time, networking, storage and randomness instead of calling the operating system directly.
- Centralize decisions. Choices about load, faults and scheduling should derive from a controlled source and be associated with a seed.
- Define failure models. Each fault should correspond to a useful assumption: a lost message, a stopped node, a refused write or a delayed response. The model should remain readable and configurable.
- Write invariants before multiplying scenarios. Without an oracle, increasing the number of runs mainly increases the amount of unanalyzed behavior.
- Produce a replay artifact. The seed must be accompanied by the configuration, active options and scenario that failed.
It is better to begin with a narrow property and a simple model: three nodes, one request type, network delays and controlled crashes. Once replay is stable, the scope can include partitions, storage errors and richer workloads.
Adding every fault at the start makes it difficult to identify the cause of a divergence. Is the defect in the product or in the simulator? The same principle applies to the harness itself. An error in the storage model can create an unrealistic guarantee; a scheduler that forgets certain events can manufacture a false block.
Targeted tests of the abstractions, along with internal assertions about the consistency of the event queue and clock, reduce this risk. The simulator is critical software too.
Perspective: Simulation Does Not Replace Reality
Deterministic simulation examines the algorithmic paths it controls in depth. It does not automatically reproduce every behavior of the real world. A simulated network interface does not necessarily validate the actual transport stack. A disk model tests neither the driver, nor the file system, nor the exact behavior of the hardware. Single-threaded execution does not expose every data race in a multithreaded client.
TigerBeetle explicitly recognizes this boundary. Its deterministic tests replace the network transport and storage interface with stubs, simplified implementations used in place of the real components. The project complements them with Vörtex, a nondeterministic harness that exercises compiled binaries, client drivers and language bindings from the outside. Its article on Vörtex presents this complementarity as a response to the simulator’s blind spots.
FoundationDB follows a similar logic. Its client-testing documentation explains that some multithreaded behavior does not fit single-threaded simulation and is covered by end-to-end API tests on a real cluster.
A strong strategy therefore relies on several layers of defense:
- deterministic simulation rapidly explores rare sequences, accelerates time and turns a fleeting failure into a replayable scenario;
- tests on real binaries and infrastructure verify the boundaries that the model replaces;
- performance tests remain anchored in representative environments, because virtual time does not measure physical latency.
The decisive benefit is not simply rerunning a test with the same seed. It is being able to answer three precise questions: which decisions led to the failure, which property was violated and which parts of the result still depend on a simplified model.
When a harness can answer those questions, rare distributed failures stop being impossible-to-reproduce anecdotes. They become durable, understandable and verifiable test cases.
Sources
- FoundationDB: A Distributed Unbundled Transactional Key Value Store — ACM SIGMOD / authors: FoundationDB, Apple, Snowflake and Antithesis
- Simulation and Testing — FoundationDB ON documentation — Apple and the FoundationDB project
- Client Testing — FoundationDB ON documentation — Apple and the FoundationDB project
- Deterministic simulation testing - how it works and when to use it — Antithesis
- The Antithesis environment — Antithesis
- TigerBeetle Architecture — TigerBeetle
- Simulation Testing For Liveness — TigerBeetle
- A Descent Into the Vörtex — TigerBeetle
Daymain Team
