Debugging intermittent bugs by narrating the suspect interleaving out loud

Debugging intermittent bugs is the most disorienting kind of debugging, because the evidence disappears as soon as you start looking at it. A request fails on the staging box roughly every fourth run, but by the time you attach a debugger or add a log line, the box behaves perfectly, which is the classic heisenbug signature. Reads of the code turn up nothing suspicious, since the code is defensible line by line; greps of the logs show one clean path and one corrupted response with no obvious cause. Your memory of the failure starts to blur after a day, and soon the only reliable witness is the feeling that something about the flow is racy, which is a hypothesis without a shape.

The instinct at this point is to try a fix as a coin toss, and the coin toss is how flaky bugs survive for months. The better move is to treat the recurrence as data and refuse to touch the code until you have a collection of both outcomes. Debugging intermittent bugs productively starts with a boring premise: the conditions differ between the passing run and the failing run, and the difference is findable, usually in timing, in state, or in shared resources. The first thing you capture, however, is not the failure but the frequency, because a bug that fails every fourth run is one you can reproduce by hand, and a bug that fails every fourth afternoon is one you have to manufacture.

Before hunting, say the failure sequence out loud in three sentences and make yourself commit to a suspicion. Speaking forces you to choose: I believe the second request reuses a connection that the first request closed, or I believe the cache is returning a stale value at the exact moment the page expects a fresh one. Typed notes let you keep two hypotheses alive in parallel, but a spoken sentence demands a selection, and selection is what turns a vague dread into an experiment. It also gives the AI agent something to work with, since an agent asked to find a race condition in a whole service is much less useful than an agent asked to verify one named interleaving.

Set the stage for the hunt: run the failing path in a loop, repeatedly, in an environment you control, and keep the console and the agent panel side by side in front of you. Then begin the ritual of speaking each run's outcome as it lands: pass, pass, fail on the fourth, and there it is again. You are not merely narrating; you are building a corpus in real time, and the corpus is the only honest material a heisenbug offers you. The loop also changes the emotional register of the work, turning the dread of an unlucky failure into the rhythm of a controlled experiment that is guaranteed to fail, conveniently, whenever you need it.

Every mystery needs a pile, so set up two folders or two files: one for the passing runs and one for the failing runs, and do not let the failing runs be rare exceptions. In the world of intermittent failures, the passing run is not the normal run; it is simply the run that happened to dodge the bullet. Start by comparing the entry points of both worlds. What headers arrived? What connection state did the request carry? Which worker was running at the time? A difference in any one of these is your first thread, and threading the difference is easier when the two piles sit next to each other rather than in your memory.

Names are everything here, so name the anomaly before you chase it. Open a failing log and one passing log side by side, and say out loud what the failing one does that the passing one does not: it retries the lookup twice, it reads a stale row, it skips the validation. The act of naming converts an impression into a feature you can search for, and once you have named the anomaly you have also decided what to instrument. Add logging around exactly that point, not around the whole service, then rerun the loop. The difference between a passing run and a failing run should now be one line of log output, and a single line can be compared with a pair of eyes.

Ask the agent to summarize the log pairs as you collect them, and say the summary back to it in your own words to check the reading. This spoken loop is powerful because the agent notices patterns in the corpus that you might glide past, like the same warning appearing three lines earlier in every failing run, while you are still staring at the exception. Voice also forces a narrative order onto the evidence; you end up telling the story of the failure chronologically, and chronology is where interleaving bugs hide. A log excerpt read aloud has a surprising property: the hesitation in your own voice marks the line that doesn't fit, and that hesitation is an honest signal worth respecting.

Keep a running spoken tally of the frequency as you collect: four runs, one failure, twice in a row on the cold cache, never on the hot one. Frequency patterns frequently point to a stateful culprit, because a process either starts cold or stays warm, and a bug that only bites cold starts is a different animal from a bug in the busy loop. Say the tally to the agent and ask it to list conditions that correlate with the failures; then pick the condition you can tip by hand. You are one step from an experiment that reproduces the bug on demand, which is the moment this bug stops being mysterious.

Now narrate the suspect section of the code as if you were commentating a slow-motion replay, line by line, while the agent shows you the execution path on the failing run. In a race you are looking for the moment when two threads or two requests cross paths: thread A reads the status, thread B flips the status, thread A continues with the outdated view. Say the sequence aloud in that order, and then say the alternative order, because a race is nothing more than a legal interleaving that happens to be dangerous, and the correct interleaving is usually obvious once you have both on the table. Speaking both makes the dangerous one visible in a way that staring at a diff never produces.

The usual result is embarrassment mixed with relief, because the race is often in a place nobody suspected: a shared cache key, a lazy initializer, a connection pool checkout that doesn't reserve. Narrating while you read prevents the lazy skimming that hides these sites, because you cannot mouth the words of a sync block and simultaneously pretend you understand it. Ask the agent to print the interleaving for your spoken description: given the sequence I just narrated, show me the exact ordering of the two reads and the one write. When the output matches the dangerous order you spoke, you have your culprit, and the fix becomes a sentence you can dictate rather than a research project.

Verify the narration against the actual log timing, because a story you tell yourself is not evidence until it matches a timestamp. Read the two relevant log lines aloud in order and check whether the gap between them sits inside the window where the race can happen. If the gap is too small or too large, your story is wrong and you say so out loud, then ask the agent for the next candidate interleaving. This discipline is what separates real understanding of a flaky bug from a plausible-sounding excuse for a broken fix. The race you ultimately fix will be the one your narrated story survives contact with the logs.

Write the winning interleaving down as a plain sentence, the kind you would explain to a colleague on a call: the response handler reads the session value while the refresh task writes it, and the read happens between the clear and the write. Then ask the agent to annotate that sentence in the code as the canonical explanation, because future readers will not have your two hours of narration. A comment that states the forbidden order is the most underrated artifact in debugging intermittent bugs, and it costs fifteen words. From this point on, the memory of the flaky fourth run lives in the code, not in the head of the person who found it.

A fix for an intermittent bug is only proven by repetition, so before writing anything else, decide the repeat count with the agent: ten thousand invocations of the failing path is a reasonable target for a local loop, and if the bug failed even once in ten thousand, it would have tormented the team for months. Restate the suspected mechanism out loud, then dictate the minimal fix: serialize the check and the set, or make the read atomic, or stop sharing the cache entry. Minimal matters because the point is to remove the race, not to rebuild the module, and a large fix adds new failure modes that are harder to detect than the one you are fixing.

Run the loop with the fix in place and narrate the results as they stream, then run it a second time to confirm the behavior is stable rather than lucky. Some races are probabilistic, and one clean hundred-run batch proves nothing, so count in tens of thousands and stay for the whole run instead of tabbing away; a race can hide in the exact run you weren't watching. If the loop is slow, shrink the path with the agent until each repetition is near-instant, since a fast loop is the only honest currency for a probabilistic bug. The agent can usually suggest a harness, and you can dictate the harness conditions in a single sentence.

Two runs of ten thousand are good, but a third variation is better: run the loop under load, or with parallel workers equal to the box, because the original failure happened in production-shaped conditions rather than in a clean harness. Say the load goal out loud and let the agent pick the settings; you keep the supervision, it chooses the machinery. If the fix is timing-related, also run it with a deliberately slow neighbor process once, to prove the fix holds when the host is busy. The bug that fails every fourth run on the staging box has now been coerced into failing on demand, and it has been coerced into failing on demand with the fix removed, then it vanishes entirely once the fix goes in.

Finally, ask the agent to write the regression test that encodes the race as a named scenario, not as a flaky anecdote: call the sequence in an order that would have triggered the bug, and assert the invariant the race violated. That test is your proof for the codebase, and it will catch the next refactor that reopens the window. Copy the ten-thousand-run results into the pull request description, and note the exact interleaving you narrated as the mechanism. A reviewer who sees a precise statement plus repetition data merges with confidence, and confidence is the real product of this whole exercise.

A fixed heisenbug rarely stays fixed, because the code around it keeps moving, and every move can rebuild the same window under a different name. So the final step is guarding the timeline: add an assertion or a monitor that detects the bad pattern even when the failure itself does not surface. A warning that fires when the dangerous interleaving is about to happen, rather than when the user sees an error, is the difference between a bug that is caught and a bug that is reported. Dictate the guard's trigger as a plain sentence and let the agent place it where the logs can see it, then watch it in staging for a week.

Share the narrated summary with the team while it is still yours to tell, in the pull request and on the channel. A spoken-style summary is more memorable than a wall of log excerpts, and the story of the fourth run is exactly what a teammate will remember in six months when a similar symptom appears. Keep the summary under a hundred words, in the same sequence you used: frequency, anomaly, interleaving, fix, guard. Ask the agent to draft it from your narration and then read it aloud yourself; if you stumble on a sentence, the sentence is wrong and needs rewording before it ships as institutional memory. The stumble is the honesty detector, since the sentence that trips your own voice is the sentence that will trip the next reader of the ticket.

Calendar a revisit: one month from now, check the guard's warning count and re-run the ten-thousand loop on whatever the box has become. This is a two-minute spoken check with the agent, and it catches slow regressions that no one reports. Flaky bugs have a long memory, so do not mark the ticket closed and move on; set the ticket to wait for the month check, because the bug's patience is longer than the team's attention span. The guard, the comment, and the regression test together form a fence that costs almost nothing to maintain. Two minutes a month is the cheapest insurance in the entire flaky-bug playbook, and it is the step that separates a fixed bug from a legend that returns.

The deeper lesson is applicable to every heisenbug you will meet: the flakiest bugs are defeated by voice-visible reasoning, not by more log noise. The narration that pinned this race is a skill compound, since each time you narrate an interleaving your model of the runtime sharpens, and the next suspect bug ends up smaller and faster to catch. Keep the failing-runs folder around for a while, because the passing-and-failing pair is still the best teaching artifact you have. When a teammate hits a bug that fails every fourth run, hand them the folder and the narration method, and watch them find it in an afternoon instead of a month.