I asked Claude to audit the error handling in a system built by the multi-agent AssemblyZero. The audit filed 12 issues in 12 minutes. None of them were the kind of bugs you find with a linter.

The email pipeline had been running for three days. 121 conversations. 286 messages. Five send paths. Two clouds. Everything appeared healthy.

Then I asked the AI that had helped build it one question, a very simple prompt:

“What happens when things go wrong?”

Not “find bugs in this function.” Not “review this pull request.” Not code review: failure mode analysis.

Twelve minutes later, I had 12 issues. They clustered into four patterns.

The System

I built Hermes, an autonomous email agent to engage inbound recruiter and consulting outreach, scoring each opportunity by rate, location, and position match and answering in my voice. You can read the design notes and decision records in the public HermesWiki.

Before Hermes fully engages with an inbound thread, it asks the sender to do one small thing: star a GitHub repository. In the first half of 2026, much of the inbound opportunity email is automated (recruiter bots working the job boards), and a bot can’t review a portfolio and click a star. A human who actually looked at my work can. The star is a liveness check: thirty seconds of proof that there’s a real person.

Mail reaches Hermes two ways: sent directly to its own addresses, or forwarded from my personal Gmail by an Apps Script. The architecture splits across Cloudflare (email ingress, database, dashboard) and AWS (SQS queue, Lambda for AI inference, SES for outbound email).

The critical path, taking the forwarded road:

That’s nine hops across two clouds. Every hop can fail. And because this pipeline is the front door for ThriveTech.ai, a silent failure here isn’t just a dropped packet: it’s a burned bridge with a client or a lost contract. The question is: what happens when it does?

Pattern 1: Missing Idempotency

The first two bugs were the scariest because they’re silent.

Bug #1: Callback failure = silent data loss. Lambda sends the email via SES (succeeds), then calls back to the Worker to record the success in D1 (fails). The recruiter got the email. The database doesn’t know. The dashboard shows the conversation as “awaiting response.” The retry cron finds it and sends the email again. The recruiter gets two identical emails. The database never catches up.

Bug #2: SQS retry = duplicate sends. SQS has at-least-once delivery. If Lambda processes a message but takes too long to delete it from the queue, SQS redelivers. Lambda processes it again. The recruiter gets two responses to the same email.

Both bugs have the same root cause: no idempotency key. There’s no mechanism to detect “I’ve already processed this” or “I’ve already sent this.” The system assumes every operation happens exactly once. In distributed systems, nothing happens exactly once.

The fix: Add a processing_id (UUID generated by the Worker at SQS push time) that Lambda checks before processing. If the ID already exists in D1, skip. For callbacks, implement a retry queue. If the callback fails, Lambda stores the result in a DynamoDB table and a separate process retries delivery.

I grow weary, sometimes, of repeating the same mistakes. How many times have I built a first version assuming exactly-once delivery?

Pattern 2: Incomplete Retry

Three bugs shared a theme: the system had retry logic, but it didn’t cover every failure path.

Bug #3: SQS push failure = conversation limbo. If the Worker can’t reach SQS (network blip, credential rotation), the conversation is recorded in D1 as “received” but no AI job is queued. The conversation sits in ENGAGING state forever: no response goes out, no retry fires, the recruiter waits indefinitely.

Bug #4: Daily follow-up caps at 20. The follow-up cron has LIMIT 20 to prevent mass sends. With 94 active conversations, 74 get no follow-up on any given day. The query has no ordering or pagination, so the same 20 conversations could get follow-ups every day while others are permanently neglected.

Bug #5: Legacy retry overwrites AI responses. The original retry function was written before the AI pipeline existed. When it retries a failed send, it regenerates the email body from a generic template instead of re-sending the original AI-generated text. Every retry destroys the AI’s generated reply.

The pattern: retry logic that was written for smaller volumes and never updated as traffic increased.

Pattern 3: Race Conditions

Three bugs that only manifest under concurrent load.

Bug #6: Gmail double-forward = duplicate inbound. Gmail’s Apps Script forwarding occasionally fires twice for the same email. The Worker creates two conversations for the same recruiter. Both get first-contact emails. The recruiter thinks you’re spam.

Bug #7: Stale follow-up fires after reply. The daily follow-up cron runs at 8 AM CT. It queries conversations with updated_at < 20 hours ago. If a recruiter replies at 7:55 AM, their reply is being processed by Lambda while the cron query is already running. The cron sees the old updated_at, queues a follow-up. Five minutes later, the recruiter gets both an AI response to their email and a follow-up nagging them. The follow-up makes no sense in context.

Bug #8: Duplicate conversation creation. getOrCreateConversation() does SELECT then INSERT. If two emails from the same recruiter arrive within milliseconds (forwarded from two sources, or an email thread), both SELECTs return null, both INSERTs succeed, and you have two conversations for one recruiter.

The fix for all three is the same tool: unique constraints and SELECT-before-act checks. Bug #6 needs a UNIQUE INDEX on Message-ID. Bug #7 needs a freshness check (updated_at > threshold at send time, not just at query time). Bug #8 needs a UNIQUE INDEX on sender email.

Pattern 4: Blind Trust and Brittle Security

Four bugs that aren’t about failure handling: they’re about what happens when you trust inputs you shouldn’t.

Bug #9: No timeout on GitHub API calls. Star verification calls the GitHub API to check if a recruiter has starred the repo. No timeout is set. If GitHub is slow (or down), the Worker hangs for 30 seconds until the platform kills it. Every subsequent request to the Worker hangs too, because the event loop is blocked.

Bug #10: updateConversation accepts arbitrary column names. The dashboard update endpoint takes a column name from the request and interpolates it into SQL. UPDATE conversations SET ${column} = ? WHERE id = ?. An attacker (or a typo) could target any column: state, send_success, created_at. There’s no whitelist.

Bug #11: HMAC timing attack. The Lambda callback authenticates via HMAC signature. The comparison uses === (string equality), which short-circuits on the first mismatched character. An attacker can brute-force the secret one character at a time by measuring response times.

Bug #12: Dead code path. The forwarded handler still calls sendViaResend(), a function from before the SES migration. Resend is supposed to be dead. But this code path is alive, uncharged, and using a different sender identity than SES.

These four share a pattern: assumptions that hold during development but fail in production. GitHub is always fast. Until it isn’t. Nobody would send a malicious column name. Until they do. String comparison is fine for auth. Until someone measures the timing. Dead code doesn’t matter. Until it runs.

The Meta-Pattern

Here’s what struck me about these 12 bugs. A linter would have caught none of them; a linter doesn’t know that SQS delivers at-least-once. A test suite would have caught few, because a test suite encodes the same assumptions the code does. How can you write a test for a failure you never imagined?

These are design bugs, not implementation bugs. They exist in the spaces between components. They’re the questions that didn’t get asked: “What if the callback fails?” “What if SQS delivers twice?” “What if two emails arrive simultaneously?”

Wouldn’t a good code review have caught these? Missing idempotency at a network boundary is exactly the comment a senior engineer leaves on a pull request. We’ve all sat on both sides of that review. The question isn’t whether review works. The question is what it costs.

The reviewer who knows at-least-once delivery in their bones is the most senior engineer in the room, and their hours are the most contested. A pull request gets fifteen minutes of that attention, not a day. Worse, these bugs don’t live in a pull request. They live between pull requests: a retry cron reviewed clean in version 1, silently wrong once version 2 changed what a send means. No diff contains the composition, so no diff review catches it. Some of these weren’t even bugs on review day. The follow-up cap of 20 was right when there were 20 conversations. Review approves the code in front of it, against the world as it is. The world moves on.

Failure mode analysis is also the work human attention does worst: enumeration, three questions for every external call. By the fortieth call site, the sharpest reviewer is skimming. So teams spend the deep reviews where the money is, the payment path and the auth flow, and everything else gets a look and a nod. That is not negligence. That’s an engineering budget.

What does it tell us that an AI found all 12? Not that AI is better at this than humans. I would have found them too, after one of them bit me in production. The difference is systematic enumeration. The AI methodically walked through every call, every failure mode, every recovery path. It didn’t get bored. It didn’t skip the obvious ones. What changed is not the quality of the reviewer. It’s the price of the review.

What I Do Now

After this experience, I run a failure mode audit on every system that crosses a network boundary. The prompt is the same simple question: “What happens when things go wrong?” That is the whole prompt. No role-play, no elaborate scaffolding, no twelve-paragraph system prompt.

The simple question outperformed any clever structure I could have dictated because it didn’t tell the model where to stop looking. Over-specify, and the AI complies; leave room, and it explores. The simpler the question, the deeper the analysis.

It takes fifteen minutes. It usually finds three to five real gaps: places where the system silently loses data, sends duplicates, or hangs.

But error handling is one question from a much longer list. Ask “what did we hardcode that we’ll regret?” and you get a different dozen issues. Ask “which assumptions stop being true as this grows?” and you get another. Ask “what would a bored attacker try first?” There are hundreds of questions in this family, one for every class of defect that ships, and until now each one cost a senior engineer’s afternoon to answer properly. Now each one costs a sentence.

The four patterns are decades older than this system: missing idempotency, incomplete retry, race conditions, and blind trust. They are the standard failure modes of anything that crosses a network, and this system had all four. I’ll gladly take the fifteen minutes to find them early.

Because without a team, the counterfactual for this system was never a staff engineer catching Bug #7 in review. It was no reviewer at all, which is how most solo-built code ships.

There is a joke you find on a developer’s t-shirt: “I always test my code… in production.” It gets a laugh from a very specific crowd, but the humor is a defense mechanism.

Testing in production is just another form of code review.

It is the most expensive one: it finds bugs but bills you in postmortems.

The oldest surviving account of inspiration in the Western canon begins with a shepherd ambushed on a mountainside. Hesiod, tending his flock upon the slopes of Helicon, encounters the Muses—the nine daughters of Mnemosyne, which is to say, the daughters of Memory. Before granting him their divine gift, they offer the most startling disclaimer in classical literature:

“We know how to speak many false things as though they were true; and we know, when we will, to utter true things.”

Linger on that for a moment. The supreme source of poetic truth opens with a warning: outputs may be beautiful, plausible fabrications; truth is available, but never guaranteed. This is no mere footnote. It is the Muses’ explicitly stated nature, etched into verses 27 and 28 of Greek theology’s founding poem.

Twenty-seven centuries later, we forged machines from compressed human memory. We built literal daughters of Mnemosyne, training them on the vast, accumulated text of our species, only to discover—to our apparent shock—that they, too, speak beautiful falsehoods. And, when the light strikes just right, they utter profound truths. We called this “hallucination.” We treat it as a scandal, a mere engineering defect awaiting the right patch.

The Greeks knew better. Their ancient response is the very operating manual we keep failing to write.

They did not worship the Muses with the blind terror they reserved for Zeus. They invoked them, and then they rigorously checked the meter. The invocation was genuine—the poet knew he could not summon the song alone, confessing as much in the opening breath of every epic. But no rhapsode, not even a master like the blind poet Homer, ever sang unverified. The craft itself—the scansion, the ancestral formulas, the watchful ears of an audience who knew the myths by heart—served as the filter. Inspiration tethered to verification. The divine gift, grounded by a human referee. The muse proposes; the tradition disposes.

I watched this ancient protocol run last week at industrial speed. An AI labored through the dark on an open mathematical conjecture, and its log revealed the muse-condition in perfect clarity. A checking algorithm fired hundreds of counterexamples at one of the machine’s cherished intermediate claims—many false things spoken as though true. Yet, the very pattern of those failures illuminated the hypothesis that made the final lemma sing. True things, uttered only when the grueling work of checking made them utterable.

The finished proof laid out its sources in tiers: verified texts, second-hand reliances, and finally, the most honest tier of all—ambient knowledge, its provenance unrecoverable, left deliberately uncited. Because the machine’s memory, like the Muses’, is an ocean, not a tidy bibliography.

And neither, if we are radically honest, is our own. The mathematician Tom Lindström captures this perfectly when he opens his treatise on mathematical analysis with a quiet, devastating confession: his most vital influences are the ones he can no longer trace. The foundational truths, he writes, are simply “just there,” the moment of first learning entirely swallowed by time. This is the human equivalent of the machine’s black box; it is the silent, ambient whisper of the Muse. We possess the knowledge, but have lost the map to its origin. Seen in this light, every acknowledgments section ever penned is merely Hesiod’s ancient disclaimer, dressed up in the polite manners of modern academia.

This, then, is the lens through which we must look. The hallucination problem is neither new nor scandalous, nor is it a software update away from vanishing. It is the permanent, indelible character of anything that speaks entirely from memory: be it god, poet, professor, or transformer. The Greeks encoded this duality into their theology because they possessed a far greater honesty about their sources of inspiration than we do today.

The ancients did not leave us a warning to flee the Muses. They left us a protocol for surviving their arrival: invoke boldly, verify ruthlessly, and never confuse the blinding light of the gift with the quiet, structural work of the checking. The civilizations that flourish alongside these new daughters of Memory will be the ones that remember both halves.

Those who only invoke will drown in a sea of beautiful falsehoods. Those who only verify will never hear the song.

Half of 2026 is written into the ledger, and still RAG is on every job description. “Experience with retrieval-augmented generation” appears next to “LLMs” and “prompt engineering,” an assumed requirement. So when a system needs to give a language model some context it doesn’t have, the reflexive move, the one that gets you nodded through the interview, is: add RAG. Embed the corpus, stand up a vector store, retrieve the top-k chunks, stuff them in the prompt.

I built exactly that. Then I deleted it. Here’s why, and why the deleting is the part I’d put on a resume.

The tool has a shape

RAG is a similarity engine. That’s not a criticism; it’s a specification. You embed a corpus into a vector space, and at query time you find the passages whose vectors sit closest to your query’s. It answers exactly one question: “what, in this large pile of unstructured text, is most like what I’m asking about?” Support tickets, documentation, research papers, a sprawling wiki. RAG is the right tool when the thing you need is semantically nearby and you’d never find it by exact match.

The failure mode hides in that word: similar. RAG doesn’t return the right answer. It returns the nearest one. And for a whole class of problems, the nearest answer is a trap.

The problem I actually had

My system — AssemblyZero, an open-source multi-agent framework that designs and writes code — had a specific bug. Its design stage would write a low-level design that called functions that don’t exist. It would confidently reference orchestrator.save_state() when the real function was save_orchestration_state(), because the design stage couldn’t see the actual code. Interface hallucination.

I reached for an obvious fix: RAG. Index the codebase, retrieve relevant code while writing the design, ground the model in reality. I named the component the Librarian (Ook), wired up a vector store and a local embedding model, and… it never quite worked, and I got busy, and it went dormant. Months later I couldn’t even remember whether it was on.

Here’s what I’d missed. Grounding a design in a codebase is not a similarity problem. It’s a ground-truth problem.

When the design needs to call into existing code, it doesn’t need something like the real function. It needs the real function: the exact name, the exact signature. And “most similar” is precisely how you retrieve save_state() when the truth is save_orchestration_state(). RAG wouldn’t have fixed the interface hallucination. On a bad day it would have fed it, handing the model a plausible, semantically-adjacent, wrong name with the false authority of “retrieved from your codebase.”

I had pointed a similarity engine at a ground-truth question. That’s my mistake, because RAG seems to be the answer to “the model needs context” in general, when it’s really the answer to a narrower question: “the model needs the most relevant context from an unstructured pile.”

What ground truth actually needs

The real interface surface of a module isn’t fuzzy. It’s ast.parse. You read the file, extract the public classes and functions and their exact signatures, and hand those to the LLM. You don’t hand it the five nearest neighbors a similarity search returns, and you don’t hand it the whole codebase dumped into the context window. You hand it the exact interface, nothing else. No vector store. No embeddings. No gigabytes of machine-learning libraries. A few hundred lines of deterministic AST walking that returns the truth, not the neighborhood of the truth.

So I built Tiphys, the helmsman of the Argo to bring the ship in true, to steer the designer LLM true. And the quietly humiliating part: the extractors it needed already existed elsewhere in my own system. The ground-truth machinery was there the whole time; I’d just walked past it reaching for the fashionable thing.

The retirement

Killing RAG deleted 6,400 lines of code and an entire dependency avalanche — the vector store, the embedding model, the machine-learning and GPU libraries, a whole server stack — that had been sitting in the lockfile in service of a feature that never ran. My continuous integration, which had been quietly red for days on a dormant test nobody watched, went green the moment the wrong tool was gone.

I put the why in an architecture decision record and left the old one standing, marked superseded, because the story of what you tried and why it was wrong is worth more than the tidy pretense that you got it right the first time.

The actual skill

So here’s the thing the job description’s “experience with RAG” checkbox can’t capture: I don’t reach for RAG by default. When a system needs retrieval, I ask one question first: is this a similarity problem or a ground-truth problem? Similarity, and RAG is exactly right. Ground truth, and RAG is how you ship a system that is confidently, authoritatively wrong.

Reaching for the hyped tool is easy. Knowing when not to, and being willing to delete six thousand lines of your own code when the answer is “not here,” is the part that’s actually hard. And it’s the part that’s actually the job.

AssemblyZero is open source. If you’d star a framework for what it had the sense to delete, it’s on GitHub.

Oliver Twist walks the length of the workhouse hall, bowl in hand, and asks the master: “Please, sir, I want some MORE.”

Everyone remembers the scene. Why is the scene so depressing? Oliver’s more is a beggar’s more. It is a request made inside the workhouse’s own arithmetic, on the workhouse’s terms, addressed to the man holding the ladle. The most Oliver can win is a second scoop of gruel.

There is another more.

In June of 1944, over the Philippine Sea, an American pilot named Ziggy Neff keyed his radio and said, “Hell, this is just like an old-time turkey shoot.” Three hundred forty-six of theirs went down that day against thirty of ours. Of the four hundred and thirty carrier planes Japan flew at dawn only thirty-five were still in the air. It was the end of Japanese carrier aviation for the war, forever.

What is the take-away? We always want it to be one thing. The Hellcat. The horsepower. The radar. The pilot training. The number of planes. The number of pilots. The available fuel. The proximity fuse. It was never one thing. It was everything: a hundred thousand new aircraft a year, broken codes, a floating supply train, stacked until the contest changed category. That’s the other more. Not a bigger bowl. A different arithmetic.

You have run the old arithmetic yourself. Everyone past a certain age has. There is something of yours up on a shelf. A language. An instrument. A theorem, a book, a business. You did the math one night and the math said no. Not enough evenings. Not enough years. Not the right letters after your name. So up it went, and you told yourself the word every workhouse teaches: later.

The old arithmetic told me the same thing, for decades: not credentialed for this, not young enough for this, no grant, no lab, no graduate student, no department. Please, sir. The old arithmetic offers the ladle-holder’s terms, and I stood in that line longer than I care to remember.

Then the arithmetic changed. Last week a conjecture in algebraic combinatorics — posed by a working mathematician, descended from a published paper whose own proof had broken — was reconstructed, verified across 284 cases, and given a complete proof draft. Overnight. At my home desk. The token cost: four dollars and twelve cents.

The new arithmetic stacks. An AI reads the paper, the errata, and the code, and holds all of it at once. A verification engine written at ten past midnight that fires hundreds of test cases — a polybolos, the repeating ballista the Greeks built when they got tired of loading one bolt at a time. A tireless collaborator who can stack the attack on every axis.

I build the machine that stacks them, the orchestration that reads and writes and checks and holds the whole problem at once. With it, alone at a home desk, I built a study tool called Chiron that not long ago would have taken a team and a year. Then I gave the framework away, free, because the arithmetic only changes for everyone when the device is one you can take apart.

I have wanted a stack like that my whole life. Decades ago, a new age, a man on the moon. And a five-year-old boy who wanted nothing more than a blue Schwinn Stingray for his birthday. That year Dad brought home a new Pontiac. “It has posi-traction!” “Dad — is posi-traction an inherent property of the car or a mechanical device?” A long pause. A studied look at the too-serious child who’d asked. “It’s a device, son. It’s the way the gears work in the rear axle.”

I always had more questions than anyone could ever play or want to play. More questions. More ideas. More plans, more drawings, more projects. I never could move it all forward together, could never get the traction to push on every dimension at once.

Fifty-plus years of deferred desire, accumulating compound interest.

All of those obstacles are now just gunpowder, sending fireworks shooting from my mind. Finally I can push MORE, for the cost of a cheap bottle of wine from Trader Joe’s, the wine you’d drink watching television, sedating away the dreams that slipped out of reach.

The dream deferred does not dry up like a raisin in the sun. Langston Hughes asked the question; I’ve lived the answer. It accumulates interest. It gains energy. It burns, a golden thread, until it must erupt.

Two little boys asked for MORE. One got a second scoop of gruel; the other waited fifty years for the arithmetic to change.

You don’t have to wait at all. Go to the shelf. Take it down. Run the numbers again; they changed while you weren’t looking. The framework that stacks the advantage is AssemblyZero, free for hackers and academics. If you’re tired of the old arithmetic, come see how the gears work. Take that ladle, it is yours now.

MORE everything. That’s the new way.