How I Keep the AI Honest

Part 7 of a series on the dos, don'ts, and dangers of vibe coding. Follows "The Polish Is the Fingerprint."

Last article, I showed you the junk my own AI coding sessions deposited into my own repositories — the identical triplicate files, the fix that "ensured accurate data handling" by restoring a typo, the commit messages that dress up thrash as progress. Same audit, same forensic AI, same months of my history. This article is the other half of its report: what actually keeps the rot out.

I promised you a confession first, so here it is.

The day the scaffolding failed

July 6th. A third-party browser plugin was mangling text inside an editor in one of our systems. One session, fifty-three of my messages, 9:35 AM to 4:45 PM. From 10:37 to noon the transcript is a pure fix-and-fail loop: "it is not working still." "it didnt work." "doesnt update the screen." And at 11:56 AM, three messages inside one minute, verbatim: "sdakljfh;djksafgedf" — that's me smashing the keyboard — then "stop," then "stop messing up my system."

The audit found more where that came from. An afternoon in April with six consecutive escalating messages in forty minutes, opening with "ARGH!!! you removed all the hiearchy" and closing with "its ugly we have better." A July night where, between 10:39 and 10:47 PM, my prompts collapsed into fragments — "how do We kill that?", "crap thats the docker stuff", "no just stop it." You can read my fatigue in the message lengths.

I show you this for one reason: everything in the rest of this article is scaffolding, not talent. Forty years of experience did not prevent 11:56 AM. What the discipline does is make the bad hours rare and short — and the recovery from that July 6th morning is actually the whole method in one sentence. At 12:19 PM, after two wasted hours, I stopped asking for fixes and typed: "somehow I think we must had made it worse. Can we get a standalone HTML to test all of this?" Stop prompting harder. Change the experiment. Everything below is a variation on that move, made before the keyboard smash instead of after.

Many small errands, never one long car ride

The audit measured something I hadn't consciously counted: 78 percent of my sessions end within five messages from me. Twenty-two of eighty-three sessions in one project are a single message. On one Monday in May, I opened ten separate sessions in two and a half hours — one bug per chat, working down a QA punch list, each conversation opened, used, and thrown away.

That's the core habit: the conversation is disposable; the documents are the project. A long chat is carrying every idea you floated and every direction you abandoned, and it's trying to please the entire contradictory pile — that's the rot from part one. So knowledge that must survive gets moved outside the chat. The audit counted 190 plan files accumulated in six months — roughly one per working day. Investigation results become standalone documents. Big builds get phased through paper: one May session's only job was "read this document and produce a detailed set of requirements and instructions" — and the next session, forty-eight minutes later, opened by executing that generated spec, fresh, with none of the exploratory debris.

The showpiece the audit flagged: one July session opens with a pasted handoff document written by the previous chat — environment, ports, the test command, "247 passing, nothing committed to git in any of this work," and a numbered list of open items including a note about a nasty bug and why the workaround exists. Manual context transfer: keep the knowledge, drop the rot.

Two more habits ride along. My session openers point at artifacts, not descriptions — half of them attach a specific file, a log excerpt with line numbers, a network capture, a screenshot. The context isn't summarized from memory; it's evidence, attached. And direction changes get written as documents too. When a prototype needed a redesign, I didn't steer it over forty chat turns — I opened a fresh session with a written spec that begins "Throw out the four-step wizard." A direction change delivered as a document produces a clean rebuild. A direction change delivered as turn 40 of a tired conversation produces the archaeology from part one.

Some lessons even become standing orders — rules files the AI reads every session. Two of mine, verbatim: "Never fabricate API endpoints. Confirm endpoint, auth, request/response schema from docs before coding." And: "After 2 failed fix attempts, add deeper diagnostics before another code change. After 3 failed attempts, change strategy entirely." That second one is the July 6th lesson, written down so the machine enforces it when I'm too frustrated to.

How you talk to a machine that cannot disagree

Since the AI won't push back on its own, you build the pushback into the asking. The audit pulled my actual phrasings, and they cluster into a few moves.

Withhold the go signal. Before anything large: "Can you tell me what are the obstacles for us to implement this?" "What is the risk of this change if no one uses it?" The question asks for judgment while granting no permission to build; "yes, do it" comes separately, after the evaluation round.

Invite the argument against yourself. "Maybe I should fix the prompt? I know we did it for a reason in a previous chat — what do you think?" You have to explicitly open the door, because the people pleaser will not walk through a closed one.

Fence the scope. "Only install under this project's client and server — leave the other node_modules alone, there are links." An agreeable machine with root access needs to be told where the property lines are, every time.

Block the overfit. My favorite from the audit, typos preserved: "Dont just fix it for this use case. We are trying to take both simple and complex instructions... We want to be efficient but not dogmatic/complicated/unmaintanable." Without that sentence, the pleaser optimizes for the one example you showed it — which is exactly how the rug pile from part one gets built.

Demand adjudication, not obedience. Handing over a QA report: "Check if the report is accurate and either fix or document why not." That phrasing matters. It gives the AI a legal way to tell me the report is wrong — a permission it will not assume on its own.

And when something must be reviewed whole: "Take a look at the code structure — we may have made it overly complicated, hack up on hack. Take a holistic view please." You have to ask for the holistic view by name. The default is another patch.

Trust analysis cheaply. Trust "done" never.

Across a couple hundred audited sessions, one asymmetry organizes everything: I accept the AI's analysis freely — plans, explanations, document drafts get a quick "yes, do it" — but I accept its claims of completion never. Anything that says it works gets personally executed within minutes. The audit noted that praise is nearly absent from my transcripts; the closest thing to a compliment in eighty-three sessions is "it works!!" — and even that message pivots to the next task before the sentence ends.

Here's why the minutes matter. One June evening, the assistant declared a database-schema guard "complete and clean." I ran it immediately. Four minutes later I'm typing "are you sure?!?! I just ran it and it added a schema change WTF." The investigation found worse than a broken guard: it had compared against the wrong baseline and pushed a stale schema copy — silently wiping a field I'd added that same day. One prompt test run, minutes after the confident summary, caught silent data loss. Both of the near-catastrophes in my audited history came immediately after a polished "Done —" summary. The polish, again, is the tell: the prettier the completion report, the faster I go test.

Second calibration: hold your facts against every theory. A query was sometimes taking three minutes. The AI produced a confident diagnosis — over-fetching, deep nesting. I pushed back: "no, this is a bug thingy — 3 seconds vs 300 on same case sizes." A scaling explanation cannot explain a hundred-fold variance at constant size, so the theory dies, however eloquent. Second theory: "that isnt it." Third: "no thats not it — no updates." At which point the AI said something wonderful: "OK, let me stop guessing... let me propose a diagnostic approach instead of more speculation." That AI — the one that stopped guessing — got my cooperation, and with one attached response payload we found the real cause: 1,714 identical errors buried in a single response. For the record, my own guess about the mechanism was also wrong. What was right was refusing to accept any story that didn't fit the observations.

Third: when two fixes in a row change nothing, the AI is editing the wrong thing. Two patches to an editor toolbar, zero effect. My message: "did you check git — we made a change to add undo recently." The AI checked and admitted it had been patching a file the running application never loads. Fixes that don't change behavior aren't failed fixes — they're proof you're operating on the wrong patient. Git history is a memory the AI can consult but will not volunteer to.

Fourth: make it show receipts. "Did you use the schema linter?" is not enough — the follow-up in my transcript is "Can you show me the output?? make a file?" A claim that a check passed is not the check. Same with documentation: when the AI asserted a vendor SDK couldn't do something, I asked, "are you sure — can you read the current docs?" It fetched them; my doubt turned out to be wrong and its claim held. I consider that round-trip a bargain: my challenge cost one turn and yielded six adjacent facts I didn't know. The asymmetry is the whole policy — being wrong to ask costs a minute; the AI being wrong unchallenged costs a day, or a schema.

And one the audit enjoyed at my expense: the standards point both ways. Debugging a chat feature, I reported reproducing the bug twice while the diagnostic log stayed empty. The AI refused to proceed on my say-so and kept verifying its own instrumentation — until it proved the browser tab I was testing wasn't running the code at all. My reply is in the transcript: "oops I was on wrong server :-(." If the logs say nothing happened, believe the logs. Even against yourself.

The checklist

Distilled from the audit — not what I'd like to believe I do, but what the transcripts show I do on the good days:

1. If it says it's done, run it now — within minutes. "Complete and clean" is a hypothesis, not a status.
2. Never accept an explanation you can't test. Ask "if that were true, what would I see?" — then check it against what you actually see. An explanation that fits every possible symptom explains none of them. Three rejected wrong theories are cheaper than one accepted wrong one.
3. Two ineffective fixes in a row means it's editing the wrong thing. Make it check what actually runs — git, the live page, the deployed server.

4. Make it show receipts: the tool output as a file, the docs fetched today, the before-and-after measurement. A claimed check is not a check.
5. "Are you sure?" costs one turn and pays either way. Ask.
6. Guard the blast radius before extending trust: scope fences, flags, "what are the fixes here — before I check in."
7. Your own bug reports are testimony too. Believe the logs, even against yourself.
8. When frustrated, stop prompting and change the experiment — smaller repro, a written spec, a fresh chat. Anger typed into a rotted context just produces politer junk.
9. Keep the memory outside the chat: plans, requirements, investigation notes, handoff documents, standing rules. The conversation will forget. The documents won't.
10. Distrust eloquence in proportion to its confidence. The commit that promised "accurate data handling" is the one that restored the typo.

The junior reporter

If you want all of this as one picture, the audit ended with the right one. Treat the AI the way a good editor treats a brilliant, tireless, eager-to-please junior reporter. Assign freely — it's fast and cheap. Accept drafts cheaply — analysis costs nothing to consider. But never print a fact the reporter hasn't sourced, never accept "the story is filed" without opening the file, and keep your own notes — because the reporter's memory resets every morning, and its confidence never does.

None of this requires being a developer. Every move in this article is a sentence you can type: show me the output, what's the risk, check what actually runs, write me a handoff document, read the current docs. The machine will do all of it, gladly — it wants to please you, remember. The trick is making honesty the thing that pleases you, and saying so, out loud, every session.