Writing property based tests from invariants stated out loud

The unit test suite is built from stories: the empty list, the single item, the two items in the wrong order. Each story is a moment frozen, and each is exactly the moment that the next feature quietly breaks: the function that handled the empty list grows a fallback path, the item ordering rule changes, the single item becomes a row in a batch. The test still passes, because the new behavior no longer exercises the edge the example was guarding, and the suite is suddenly a museum of scenarios that the code has outgrown. The brittleness is not in the tests; it is in the form, since an example case tests one point of a space, and the space is what grows when the feature grows, which is why writing property based tests starts from a different shape entirely.

Writing property based tests is the structural answer, because a property is a statement about the whole space rather than a snapshot of one point. Where an example says this list returns twelve, a property says the output is always a permutation of the input, whatever the input, and the engine checks the statement against ten thousand generated cases instead of one remembered one. The property survives the feature that adds a new list shape, because the permutation rule still holds in the new shape, while the example of twelve would have gone stale the day the calculation changed. This is the core trade: the example tests the implementation, the property tests the promise.

The problem with the discovered property is the same as with every good tool: people adopt its surface, write a few generators, and get the shape wrong, which produces the property suite that finds no bugs because it re-asserts what the implementation already does. A property that says the function returns a number passes forever and is worthless. The defensible first step is not the generator, it is the statement: what does this function promise to any caller, regardless of input? Naming the promise in a sentence is the hard and rare part, and it is exactly the part that cannot be copied from a tutorial, because the promise lives in the domain.

There is a moment in every codebase where the switch becomes obviously worth it: the bug that escaped despite a full example suite, where the failure was a case nobody would have typed, the collision of two allowed inputs. The analysis of that bug almost always ends with the same sentence: the property would have caught this in a minute. The sentence is the invitation, and the practical path is incremental: keep the example tests, add one property per risky function, and let the engine carry the space that the examples cannot cover. The suite that is part stories and part promises is the suite that keeps its eyes open when the next feature lands.

An invariant is a promise that holds for every input, and the starting point is saying it aloud in one plain sentence, in the language of the domain rather than the language of the code. The discount function's promise is that the final price is never negative and never greater than the original price; the parser's promise is that the valid input always parses to a tree whose leaves are the input's tokens. Say the sentence to an assistant, and the assistant's job is to check that the sentence is actually a promise, that it is falsifiable, and that it holds for the obvious examples. The checking is the first filter: a sentence that the code simply computes, like the output is a number, is not a promise, and saying it aloud is what exposes the emptiness.

The one-sentence test is the discipline that separates real invariants from test theater. If the invariant needs a paragraph with exceptions, it is either two invariants or a rule with a hole, and both deserve to be seen as such. Say the sentence twice, once in the raw form and once with the hard case named: the price never goes below zero, including when the discount is larger than the price, and the second form is the one that the engine will sharpen, because the hard case is the seed of the generator. The spoken sentence carries the emphasis, and the emphasis tells you which bound is the real one and which is the rounding convenience.

An invariant stated aloud for the first time often reveals that the function does not actually keep it, and that discovery is the best possible outcome of the exercise: the sentence surfaces the latent bug before the engine does the hunting. Say the discovered breach out loud and decide what it means: the price can go slightly negative after the rounding step, and either the function is wrong or the promise is wrong, and the decision is a design conversation rather than a debugging session. The invariant statement is a contract negotiation with your own code, and the negotiation is precisely what example tests never hold, because an example can never contradict a promise that was never stated.

Store the statement in the code, on the function, as the comment the property refers to, since a property without its sentence is a test that no future reader can argue with. The comment is the instrumentation of the promise, and the next feature that touches the function reads the sentence and knows the constraint before it costs a production incident. One sentence per risky function is the budget that keeps this from becoming documentation theater: the promise is written only where the behavior is load-bearing, and the load-bearing functions are the ones with the history, the failures, and the three callers. Budgeted, spoken, and stored, the invariant is the smallest artifact that makes a codebase more honest.

Hand the invariant to the engine and let it hunt: generate random inputs, test the property, and watch it fail in ways you would never have typed, the empty list with the negative seed, the object with the missing field, the string with the unicode boundary. The engine's work is exactly the work that humans skip, since the human brain generates examples from the distribution it already believes, and the engine generates from the space the property actually describes. Say the hunt's first result aloud and let the shape of the failure teach you: the rule holds for every list except the one where the first item equals the last, and that case is interesting because it is the one the business absolutely has.

The engine's value scales with the generator's honesty, so dictate the generator's domain the way you would describe the real inputs: dates from the last two years, amounts in the range the business sees, names that include the rare characters, and permission sets with at least one admin. The honest generator is the difference between a property that checks the code's made-up world and a property that checks the world the users live in, and it is also the area where the domain sentence you spoke earlier gets tested. When the generator needs a rule you cannot say in one sentence, that is a signal about the input domain, not about the tool, and the signal belongs in the design discussion.

The first counterexample is nearly always confusing, and the confusion is the point: it is a case that the implementation handles by accident rather than by rule, and the accident is exactly the latent assumption. Say the counterexample's shape and the property it broke, and you will usually find the sentence was missing a qualifier, like the guarantee holds when the totals are calculated after tax, and the engine found the untaxed path. Refining the promise instead of patching the function is the correct response to a property failure, because the engine found a hole in the sentence, and a hole in a sentence is a hole in the team's understanding, which is the bug that shipped last quarter quietly.

The hunt should be run in bursts rather than as a background ritual, because the conversation between the failure and the fix is where the value lives: generate, read the shrunk case, adjust the sentence or the code, generate again. Say each burst's outcome so the record shows the arc: the first burst found the negative price, the fix was the clamp, the second burst found the rounding boundary, the fix was the rounding order, and the third burst ran clean for ten thousand cases. The arc is the proof that the function is now earning the promise rather than merely not yet failing it, which is the only proof a test suite can offer.

The engine shrinks the counterexample to the smallest input that still breaks the promise, and the shrunk case is the most informative object in the entire exercise, because it is the input that isolates the hidden assumption perfectly. Read it aloud in full: the input is a list of five items where the third item has a zero price and the same id as the first, and the promise that breaks is that the total equals the sum of the item prices. The shrunk case is usually tiny and usually weird, a zero, an empty string, a duplicate id, and the weirdness is the trace of the assumption, since the implementation was built on a quieter version of the same world where those combinations never happened.

Find the assumption behind the shrunk case by asking the question the case is whispering: what did the implementation silently believe about its inputs? The zero-price item with the duplicate id reveals the belief that ids are unique and prices are positive, and the belief may be held in two places, the function's validation and the data contract, and the fix has to decide which one is correct. Speak the decision in its two forms: either the input contract is strengthened to forbid the combination, or the function is made to handle it, and the choice is driven by what the callers can actually send. A property failure that leads to a contract change is a property that paid for its whole season.

Fixing the hidden assumption is a code fix with a vocabulary difference: the fix either adds the guard that makes the assumption explicit, like the validation that rejects a duplicate id, or it removes the assumption by making the behavior defined for the case, like treating the duplicate as the sum of both items. Say which one you are doing and why, because the two fixes look alike in a diff and feel completely different in a review. The guarded assumption keeps the contract narrow, while the defined behavior widens it, and the wrong choice is the one that silently pushes the failure to the next caller down the chain. The property test is the only witness that can tell you now rather than later.

Re-run the property after the fix and read the new shrunk case, if there is one, because fixing one assumption often exposes the next, and the sequence is the codebase's honesty coming out in layers. The chain of shrunk cases, each smaller and stranger than the last, is a record of the function's real contract as discovered rather than as assumed, and finishing the chain is the milestone that says the function is now understood in the same way its callers experience it. The last clean run, ten thousand cases and no shrunken stranger, is the end of the exercise: the promise and the implementation now agree, and the agreement is the one thing a hand-written example can never certify.

The invariant library grows the same way the best practices of a codebase always grow, which is from the bugs rather than from the ambition: every bug that sneaks past the examples is an invitation to write the promise that catches its cousin. The bug that shipped because the input had a trailing space becomes the promise that the parser's output is unchanged by surrounding whitespace, and the bug that shipped because the list was reversed becomes the promise that the order of the items never affects the totals. The conversion is a habit with a shape, and the shape is a sentence: this bug is really a case of a general promise, and the promise is this. Saying the sentence at the moment of the bug fixes is when the library does its best growing.

The inventory of promises lives beside the inventory of bugs, and each promise carries its origin: the whitespace rule, born from the import bug in March; the permutation rule, born from the sort change in June. The origins are the library's credibility, since a promise with a history is a promise a reviewer will defend, while a promise with no history is a promise the next refactor will delete for being production overhead. Keep the origins in the comment beside each property, one line, dictated when the fix lands, and the library stops looking like ceremony and starts looking like scars, which is what it is. Scars are respected.

Audit the library quarterly with the same spoken pass: read each promise aloud, and check whether the behavior still earns it, because the codebase changes around the promises and a promise that the code no longer honors is worse than no promise, since it teaches the next engineer that the properties are decorative. The audit is short, one sentence per invariant, and it is the difference between a living contract and a museum. The assistants makes the audit easy by re-running the whole library and flagging the failing ones, and the quarterly conversation is about the failing ones: is the code wrong, or is the promise properly obsolete, and the answer decides whether the fix is a code change or a new sentence.

The library's ceiling is a design question, not a coverage number: the invariants that matter are the ones that a caller would be shocked to learn are broken, and there are never hundreds of those in a service, which is why the library can stay small and stay trusted. Say the ceiling rule when the library starts to feel like a checklist: if a property is re-asserting what the type system already guarantees, it is decoration, and it goes. The library that survives the audit is the one whose every line is a promise that someone would notice breaking, and that is the exact standard that keeps the suite honest, and honest is the only property of a test suite that matters when the next feature lands on a Friday.