Auditing AI generated code: the spoken gate before every merge

The diff arrives at four in the afternoon, four hundred lines, generated by the assistant the author has been pairing with all week, and the summary in the merge request reads like a well-written paragraph: added webhook handler with retry and logging. You skim the summary, scan the files, see the tests listed as added, and click approve in under four minutes, because the diff is plausible, the style is consistent, and the author is a colleague you trust. This is the normal state of modern code review, and it is exactly the state that makes auditing AI generated code necessary rather than optional, because the failure modes of generated code are not visible in a skim.

The skimming is not laziness, it is the perceptual consequence of volume: a four hundred line diff is too large to hold, so the brain samples the beginnings of functions and the shape of the file, and sampling is blind to exactly the details that matter, the boundary in the string interpolation, the default value in a config, the branch that handles an unsupported method. The remedy is not to read every line more carefully, since attention does not scale, but to change the instrument, from a scan of the whole to a voice-narrated pass over the few lines where generated code tends to err. A targeted ten-minute pass beats an unfocused hour, and that is the entire claim of this article.

There is also a psychological trap in the plausible diff: the summary explains the change so well that reading the code feels redundant, and the coherence of the narrative substitutes for the verification of the implementation. Generated diffs are especially susceptible because the model writes both the code and the summary in one voice, and the two reinforce each other. The counter is a checkpoint question, spoken before the skim begins: if this diff was written by the most creative engineer I know, what would I check first? The honest answer is usually the outside-in boundary: the place where input becomes a decision. That is where the pass should start and where the skimming always fails.

The other side of the four-in-the-afternoon problem is attention: the skim at four has a built-in time budget that is already overspent, and the approval is often a farewell to the workday. Auditing AI generated code works because it can begin as a spoken pass, which changes the logistics: a checklist narrated on the walk home is a review that happens while the attention is elsewhere, and the artifacts, the answers, the flags, do not need a screen to be produced. The gate exists for the four-in-the-afternoon version of you, and everything in this article is designed to fit in the ten minutes that version of you actually has.

The four sins are where generated code hurts, and they are easy to state and easy to spot once you know them. Injection: a value that reaches the database, the shell, or the template from user input, whether directly or through a second-hand method like a query built by concatenation. Secrets: a key, token, or connection string that appears in the code, in a test fixture, or in a log line, often copied from an example. Bounds: a loop, a retry, a pagination, a recursion that lacks a maximum, so one bad input spins the process. Odd branches: a fallback that returns a silently wrong value, an error path that swallows the error, a branch for an impossible case that masks the real one.

State the four sins out loud before you touch the diff, because naming them primes the scan, and a primed scan finds them twenty times faster than a general careful read. Then ask the assistant to hunt them in the diff with you, one sin at a time, and to report each finding with a file and a line. The two-pass scan of the four sins takes about four minutes for a four hundred line diff, since each sin has a signature: a dollar sign followed by a brace near a query, an assignment that looks like a credential, a while-loop with no exit, an except that logs and continues. Saying the signatures to the agent is what turns an open-ended audit into a checklist.

The finding is not the verdict. For each hit, read the line aloud and say what it does in context: the token is used in the test setup, it is a fixture key, not a production secret, and that is a false positive; the while loop is bounded by the retry counter above it, and that is fine. The verdict in a spoken sentence is the discipline that keeps the audit from becoming a witch hunt, since generated code is not guilty by default, it is merely unverified, and the four sins are the four places where verification matters most. A spoken verdict on each finding also produces the record, the audit log, that makes the pass reviewable by the next person.

Rank the findings before moving on: must fix, verify further, note for the team. The ranking is spoken into the record, and it becomes the review conversation. The must-fix list for a typical generated diff is short, often one item, occasionally zero, and that is fine: the audit's value is not in how many bugs it finds but in how reliably it would find the one that matters. The team that runs ten audits and finds one real injection has bought the gate's whole year of cost, and the gate is only a few minutes per merge. The four sins are the taxonomy that makes the ten minutes possible, and the taxonomy is remembered best when spoken.

Of the diff, ten to twenty lines are genuinely worth a narration, and the narration is a kind of slow reading where you say what the line assumes: this line takes the customer input and asks the database for matching records, and it assumes the parameter is parameterized, so the question is whether the helper below it builds the query safely. The spoken question forces the verification, because a voice demand produces an answer rather than a shrug. The risky lines are the ones where the input, the credentials, the money, or the access control meet the implementation, and the index of risk is trivial to state: every line that touches user input, every line that touches a credential, every line that returns early, every line that decides whether the operation is allowed.

Narrate in the third person as if explaining the diff to the most suspicious engineer you know, and let the explanation reveal the gaps. Most narrations stall at the same spots a suspicious engineer would: the point where the code trusts a value that arrived from outside, the point where the error is logged and consumed, the point where the permission is checked once and then the resource is used throughout. A narration that stalls is a finding, and the stall is the moment to ask the agent to show the actual call chain, since suspicion is not a result until the chain confirms or dismisses it. The narration is the scanning instrument of an experienced reviewer, available on demand.

Do not narrate the entire diff; narration over four hundred lines is a lecture, and a lecture is a skim with more confidence. The selection is the skill, and the selection comes from the four sins plus the map: start with the risky lines list from the audit, add the lines the author modified in a second pass, then stop. A misdirection to dodge is the belief that the tests protect the path, since tests verify that the happy path works and generated tests are usually written by the same generation that wrote the code, so a generated test and its code share the same blind spot. Say that to the team once, and the always-green suite stops being a substitute for the narration.

When a narration is clean, mark it with a sentence and move on: this line is fine, the parameterization is in the helper, and I have checked the helper in the second file. When a narration is not clean, the finding gets a fix request or a question, not a shrug. The output of a good risky-line pass is three things: a handful of lines verified with reasoning, one or two things to ask the author about, and the note that the pass happened, because the note is what lets the next approver trust the merge rather than repeat the pass. The narration is the most human part of the gate, and it is also the part that catches the one real bug the checklist would miss.

The happy path is what the diff demonstrates and what the tests verify; the unhappy path is where production finds you. So ask the what-if questions out loud, one after another, and at each one, ask the agent to trace the code: what if the webhook payload arrives with a data type the schema said it could never be? What if the vendor times out on the third retry and the request is still pending? What if the caller sends a value that is valid type but invalid business, negative amount, empty name, a year in the future? An unhurried ten-second what-if gets two responses, one from the agent tracing the code and one from your own knowledge of the domain, and the mismatch is the signal.

The what-if questions should have names, because the named ones get remembered: the malformed payload, the double delivery, the expired credential, the partial failure. Say the names and then ask the agent to test them, since the quickest verification is a test the agent can write in a minute and run in five. A generated diff that passes the malformed-payload question without a correct response is a diff that will eat an incident within the quarter, and the discovery costs you ninety seconds. The named what-ifs are the team's shared vocabulary, so the same four questions are asked on every merge, and the answers get faster as the codebase's edges get hardened by the audit.

The interesting what-if is the one with the branch that should not exist: the code reacts to a payload type that the API documentation says is impossible, or handles an error code that the vendor claims it never sends. Generated code manufactures branches for imagined futures, and the manufactured branch is a silent sin because it suggests the code has an edge it does not have, and a future developer will rely on that edge. Say it as a question: which of these branches have we ever seen in production logs, and if none, do we want to keep a branch for a dragon? The question resolves either into a deletion, which simplifies the diff, or into an argument from someone who has seen the dragon, which is a conversation worth having.

Keep the probe log short: five what-ifs, one sentence each, and the answer. The log is the review artifact that survives the merge, and it answers the question every approver asks, whether the unhappy path was considered at all. On the worst day, a probe catches a mis-encoded secret or an unbounded retry that the four-sins pass overlooked; on the ordinary day, the probes are quick and clean, and the branch merges with the record that the what-ifs were asked. The team that asks the same five questions on every AI diff stops being surprised by the fourth of a kind, and the before-merge gate becomes a habit that costs ten minutes and buys the confidence that the skim used to pretend to have.

The gate is a team ritual with exactly four steps and one minute of overhead each: name the four sins, run the two-pass scan, narrate the risky lines, and ask the five what-ifs. Do the pass in the order given, out loud, with the assistant hunting while you interrogate, and stop at the ten-minute mark even if the scan is not finished, because the gate that overruns is the gate that gets abandoned, and the gate that is abandoned is the gate that never catches anything. The time box is a feature: it forces the selection to the twenty lines that matter rather than the four hundred that do not.

The team version adds the verdict rule: a must-fix finding blocks the merge until the author answers it, and a note-level finding goes into the review as a reference. The verdict rules written on the team's doc and read aloud at the start of each pass keep the gate from becoming two review styles fighting, and they keep the author from feeling hunted, since the pass is over the diff, not over the person. The other team rule is the share: every audit result that found something real gets summarized in one spoken sentence in the channel, and the summary teaches the team the shape of the enemy, so the next author writes the diff with the gate already in mind, which is the goal of all gates, the day they become unnecessary.

The gate must not become a synonym for distrust. The framing matters: the pass is not because the AI is unreliable, it is because the stakes of an unchecked edge are asymmetric, one injection in a hundred merges is a weekend, while a minute per merge is a habit. Say the framing at the team meeting so the authors hear it in the voice of a colleague rather than the tone of a firewall, since a gate that the authors resist is a gate that gets game-ified, and the game is a faster skim with more confident approval. The gate that the authors run on each other's diffs, voluntarily, is the one that catches the real bug a decade later.

The ten minutes also belong in the merge template as a checklist item, so the reviewer can check the box only when the pass actually happened. The box is the social contract: the author expects the pass, and a merge without it goes to the bottom of the queue. Three months in, the team has a record of the gate's hits, and the record is the argument for the gate, because nothing convinces an engineering team like the list of real findings: the secret that was about to ship in a fixture, the retry that would have hung the worker, the branch that was handling a dragon. Auditing AI generated code is a ten-minute habit on the outside and a live insurance policy on the inside, and the premium is only the effort of saying the four sins out loud before the merge.