Observability >> Predictability

Tests are good. Especially well-written ones: layered tests that catch real bugs and give you the confidence to ship.

But why are they good? Because they make us feel safe? Because of the green checkmark?

A test written before we know all the cases is a prediction of behaviour. That works for closed algorithms, such as sorting functions, parsers and calculations, where the inputs can be enumerated and correctness argued on paper. Does it work for a complex distributed system? Can anyone foresee every outcome, every timing-dependent interaction, every way a dependency they don't control can fail?

Anyone honest with themselves answers no. Information theory suggests another starting point, if we are willing to change perspective.

Prediction has a ceiling

In Shannon's terms, an outcome you could forecast with certainty carries no information: watching it tells you nothing new. That suits a space rocket. Its trajectory is designed on paper and verified exhaustively before launch, because a failure in flight leaves no second attempt.

Most systems today are not rockets. They are more like organisms: they grow, take on new requirements, depend on third-party services they don't control, and run in environments that shift under them. They have to adapt quickly, and adapting takes new information from the running system.

Testing by prediction assumes the relevant behaviours can be listed before they happen. For one pure function they can. For a distributed system of n services, each able to be in one of s observable states, the system as a whole can be in up to sⁿ combinations.

Take a modest system, something like Google's Online Boutique demo: frontend, cart, checkout, product catalog, payment, shipping and recommendations. Seven services, each with ten meaningful states: healthy, degraded, overloaded, connection pool exhausted, disk full, DNS timeout, certificate expired, rate-limited, version mismatch, and the one you haven't thought of yet.

The count does not depend on how the services are wired: sⁿ counts combinations of states, not connections. A sceptic might object that their services aren't all connected to each other. True, and sparse wiring does mean fewer direct interactions. It doesn't make the system easier to test: the interactions that do exist tend to be less obvious, less documented and more surprising.

A test suite samples this space. Even an excellent integration suite might cover a few hundred scenarios, a few thousandths of a percent of it, and this is a small system. Add an external payment provider and a message broker, and the count reaches 10⁹.

This is not a limitation of your test framework. It is a combinatorial wall.

The graph inside the graph

And sⁿ is still optimistic: it treats each service as one node with s states. In reality each service is a graph of its own. The checkout service has a request handler, order logic, clients for payment, shipping and the cart, and a database connection, each with states of its own. The payment client can be healthy, timed out or rate-limited; the connection pool full, degraded or recovering; the handler backed up. These internal states compose.

If each service has m components, each with s states, one service alone can be in sᵐ states, and the whole system in:

The exponent is itself a product: the components inside each service times the services in the system. Seven services, each with six components, each with five states:

A test suite covers a few hundred of these. The ratio isn't worth computing.

This is why integration tests that mock a service's insides miss what fails in production. They tested the interaction between services, but the failure happened inside one: a connection pool exhausted under load, a cache that expired at the wrong moment, a retry loop that amplified a problem instead of damping it. The graph inside the graph is where the surprises live.

The black box problem

There is a dimension prediction cannot reach at all: dependencies you don't control. Nearly every non-trivial system mocks external services: Stripe, AWS, Auth0, PostgreSQL. But a mock is a model of the dependency, not the dependency itself. When the payment provider changes an error format or adds a rate limit you didn't know about, your mock still returns the old behaviour. The test passes. Production breaks.

You cannot fully predict the behaviour of a system you don't own. What works for a black box is observation: watch what the dependency actually does, and react to what you learn.

The verification recursion

If your tests are complex enough to verify a complex system, they are complex enough to contain bugs of their own. A buggy test can be worse than none: it gives false confidence while making your picture of the system less accurate.

So you verify the tests. But the verifier can have bugs too:

This is more than an engineering inconvenience. Rice's theorem (1953) shows that every non-trivial property of what a program does is undecidable in general. Whether a test correctly verifies a system is such a question: no algorithm answers it for every test and every system, so no general procedure ends the recursion. That doesn't make tests useless. It means more tests cannot settle, once and for all, whether the tests are right.

In practice it shows as tests that pass for months while the bug they were meant to catch is there all along. We saw it in our own codebase: when we generated tests at scale, roughly 19% turned out superficial, raising coverage without catching real bugs.

Observation breaks the recursion

What if we turn the direction around?

Instead of predicting what might go wrong and writing tests to prevent it, we observe what actually goes wrong, and write tests that keep it from coming back.

This is observation-driven testing, ODT:

In the drawing, each test settles where a request actually failed: red while it fails, light once the fix makes it pass, and then it stays. The tests written in advance are spread evenly, as a guess is. More than a third sit where no request ever passes.

Notice what this buys you:

No verification recursion. The error you observed is real: it happened in production or staging, not in your imagination. The regression test shows the error exists, because it fails before the fix; the fix makes it pass. The test is grounded in reality, not in a prediction about reality, so it needs no test of its own. The system itself was the oracle.

No state-space explosion. You don't enumerate every possible state. You cover the paths that actually run and fail. The running system samples the state space for you, and it samples the states that matter, weighted by how the system is really used.

No black-box blindness. When a dependency misbehaves, a payment provider changing an error format or a database behaving differently under load, observation catches it as soon as it shows in the running system. You don't need to predict what the provider might do. You see what it did.

Less false confidence. Every test in the suite corresponds to a failure that really happened. The suite becomes a fossil record of the system's actual failure modes, not a prediction of hypothetical ones.

Where TDD still wins

This is not a blanket rejection of test-driven development. There are domains where prediction works:

The dividing line is whether you can enumerate the relevant state space. If you can, predict. If you can't, and for most distributed systems you can't, observe.

The plant and the blueprint

Premature prediction holds back much of what is good in the real world.

A plant grows from a small seed. It doesn't plan where its roots will go. A root tip grows, meets a rock and turns; meets water and branches. The root system is shaped by the actual soil, not by a model of the soil that was wrong about the rock at thirty centimetres.

We already run this loop in many places. CI/CD is observation-driven delivery. Feature flags are observation-driven rollout. A/B tests are observation-driven product design. Canary releases are observation-driven reliability. Why not put the same loop into how we test?

Why observation scales and prediction doesn't

The argument rests on three observations. The states of concurrent components multiply: the state-space explosion that model checking has worked against since Clarke, Emerson and Sistla in 1986. By Rice's theorem, no general procedure decides whether a test verifies a system, so the recursion has no end. And a black box is opaque by definition: you cannot test inside what you cannot see inside.

Together:

• Testing by prediction grows with the system's potential complexity: compounding, never verified in full, blind to black boxes.
• Testing by observation grows with its realised complexity: the paths that actually run, including how dependencies really behave. Real workloads are heavy-tailed: a few paths carry most of the traffic.

AI changes the testing philosophy, not the maths

AI can generate tests faster than people can. A language model reads a function and produces dozens of test cases in seconds. That sounds as if it solves the state-space problem.

It doesn't. A model generates tests from patterns: what looks as if it should be tested, given the tests it has seen. That is still prediction, only faster. The combinatorial wall doesn't care how quickly you sample: a few thousandths of a percent, sampled in one second, is still a few thousandths of a percent.

Where AI really helps is inside the loop:

• Observing: AI can read traces, correlate logs and spot anomalies across services faster than a person can.
• Writing regression tests: given an observed failure, AI can draft the test in seconds.
• Seeing patterns: AI can spot failure modes that recur across incidents and suggest structural fixes.

AI speeds up the observation loop. It doesn't make prediction viable for complex systems.

Cookbook: ODT in practice

Here is how to start today: no rewrite, no migration, five steps.

1. Pick an observability stack

You need three things: metrics, logs and traces. Grafana with Prometheus, Loki and Tempo is a common open-source stack; Datadog, Honeycomb or any APM works too. The tool isn't the point. The point is that you can see what your system does while it runs.

If you have nothing yet, start on a hosted free tier. One service can be instrumented in an afternoon.

2. Instrument one critical flow

Don't instrument everything. Pick the flow that matters most: checkout, onboarding, the API your biggest customer calls. Add structured logs at decision points, traces that cross service boundaries, and a dashboard that shows the flow's health.

If the application already runs, decorate the existing flow. OpenTelemetry's automatic instrumentation gets you much of the way without touching business logic.

3. Build an agent harness

This is the step that changes the most. Instead of writing test cases by hand, let an AI agent explore your application the way a real user would.

Set up a pipeline that runs a language model with Playwright. Give the agent a persona, not a test script. A persona has intent, context and emotion:

Each persona explores a different region of the state space. The impatient user finds timeouts. The power user finds race conditions. The frustrated user finds broken navigation. A suite written by a developer who knows where every button is rarely finds any of them.

4. Correlate agent sessions with traces

This is where the observability stack earns its keep. Each Playwright session gets a trace ID, and the ID travels through your services. When the agent clicks and something breaks, a 500, a timeout, a page that hangs, you can see what happened inside the system: which service failed, which query was slow, which retry loop amplified the problem.

Add the agent sessions themselves to the stack. Now one dashboard shows this persona, this flow, this failure, this trace.

5. Write regression tests only for what broke

When the agent finds a real failure, not a flaky assertion or a predicted edge case but something that actually broke in the running system, write a regression test. It fails before the fix; the fix makes it pass. The test joins the suite as a fossil: a record that this failure happened, so that it doesn't return unnoticed.

Your suite grows from reality, not imagination. Every test has a story; none is speculative.

This is the ODT loop made concrete. The agent doesn't predict what might go wrong; it generates realistic traffic your system has to handle. The observability stack watches what actually happens, and you write tests only for what actually fails. The system teaches you what to test.

The rhythm, continued

The first article argued that every growing system gathers structural disorder, and that the answer is not to prevent it but to have a rhythm for resolving it: the refactoring cycle.

This article applies the same idea to correctness. You cannot predict every failure of a complex system. The answer is not to test harder, but to observe better and let the system teach you what to test.

Prediction assumes you know the answer before the question is asked. Observation assumes you don't, and builds what it takes to learn quickly when the question arrives.

The entropy cycle is breathing. Observation-driven testing is listening.

Both are rhythms that keep a system alive.

The foundations: Shannon (1948), Rice (1953), Clarke, Emerson and Sistla (1986). The approach builds on Charity Majors' observability-driven development, Netflix's testing in production and the principles of chaos engineering, and writes the feedback loop down as an algorithm rather than a mindset. The 19% comes from one production microservices system, where we judged that share of generated tests superficial: they raised coverage without catching real bugs.