There is a version of AI-assisted development that makes engineering teams genuinely more productive. There is another version that ships code nobody understands into systems where correctness is not optional. The first is a tool. The second is a liability.
The distinction sounds simple. In practice, it is not. The same workflow — describe what you want, let the model generate code, iterate on the output — sits somewhere different on that spectrum depending on the system, the developer, the review process, and the consequences of being wrong. This article is about what it looks like when it sits in the wrong place.
The companion piece, Vibe Coding Will Kill Someone, makes the argument with urgency. This one makes it with more precision.
What Vibe Coding Actually Is
In February 2025, Andrej Karpathy — former Tesla AI director, former OpenAI founding member — described a personal workflow he had developed for software experimentation. He called it vibe coding. The idea: describe what you want in natural language, let AI generate the code, accept the output without reading it too carefully, and move on. "I'm building something," he wrote, "but it's not really coding."
Karpathy was describing a legitimate approach to low-stakes personal exploration. He was not advocating that anyone deploy this workflow to financial infrastructure or medical systems.
The distinction matters because the term now covers at least three different things. Fast exploration — using AI to sketch ideas in contexts where being wrong has no consequence. AI-assisted development with effective human supervision — using AI to generate code that a competent developer reviews, understands, and takes full responsibility for. And a third version: accepting AI output without genuine understanding, deploying it under time pressure, and leaving accountability implicit or diffuse. That third version is what this article examines. The name for all three is the same. That is part of the problem.
Why It Spread So Fast
The conditions for rapid adoption were already in place before the term existed.
Large language models improved substantially in coding capability over 2023 and 2024. Context windows expanded to the point where a model could hold a full codebase in working context. AI integration moved from browser tabs into IDEs — tools like GitHub Copilot and Cursor made AI a persistent presence inside the editor. Agents gained access to files, terminals, browsers, and test runners. The barrier to producing something that appeared to work dropped significantly, both for experienced developers and for people who had never written code professionally.
Organisations, already operating under pressure to deliver quickly, saw a tool that appeared to compress timelines. The gap between "appears to work" and "works correctly under all conditions" is where the risk accumulates — and that gap is not visible until something goes wrong.
Where It Works
Intellectual honesty requires stating this clearly: vibe coding works reasonably well in a significant range of contexts.
Quick prototypes. MVPs with limited scope. Simple internal tools. Landing pages and exploratory UI flows. Small automations. Cases where the success criterion is "does this roughly do what we want" and the failure mode is inconvenience rather than data loss or financial damage. For these contexts, moving fast matters more than exhaustive modelling, and the consequences of being wrong are recoverable.
The issue is not whether this approach produces usable results in easy cases. It often does. The issue is what those results look like when the context is harder: when concurrency matters, when edge cases have financial consequences, when the code has to survive production traffic, security scrutiny, or regulatory examination.
The Real Risk
The risk of AI-generated code is not that models write bad code. Sometimes they write very good code. The risk is what happens when code enters production that nobody in the chain can adequately explain or defend.
In practice, this looks like: logic that passes a smoke test but only works on the happy path. Error handling that is present superficially but does not cover the failure modes that will actually occur in production. Edge cases nobody considered because nobody reasoned about the problem from first principles. Security implications that were not evaluated. Concurrent access patterns that produce race conditions at volume. Retry logic that generates duplicate writes. Dependencies that are poorly understood and will break on a version upgrade.
These are not abstract concerns. In systems that process financial transactions, a race condition does not produce an error message at the point of failure. It produces silent data corruption — balances that do not match, duplicate settlements, reconciliation failures — the kind of failure that takes weeks to diagnose and costs more to remediate than any speed saving vibe coding produced. The consequences are asymmetric: the productivity gain is visible immediately, the technical debt and operational incidents arrive later, under worse conditions, with less context.
The deeper problem is accountability. If the developer who wrote the code cannot explain what it does, and the reviewer approved it without understanding it either, then nobody owns the behaviour of that code in production. When something breaks, there is no person with the knowledge to diagnose it quickly. There is no prior reasoning about failure modes to fall back on. There is only a system doing something nobody adequately modelled.
What Critical Systems Actually Require
There is a class of software where the requirements are different in kind, not just degree.
Payment processing, healthcare records, identity management, financial clearing — these are systems where correctness is a functional requirement, not a quality goal. Where failure modes have regulatory consequences. Where auditability means being able to explain every significant decision the system makes, not just demonstrate that it worked most of the time. Where the people affected by failures never chose to depend on the software and have no visibility into how it was built.
In environments like these — critical financial infrastructure where failures can cause significant financial losses or compromise data integrity — the standard is not "does this work in the demo." The standard is: does the developer understand the domain? Are the assumptions explicit? Is the failure behaviour modelled? Do the tests cover edge cases that will actually appear in production, not just the paths the developer anticipated? Does someone own this code after it ships?
These are not bureaucratic requirements imposed on engineers who would otherwise move faster. They are the minimum conditions under which operating a system responsibly is possible.
Code Review Is Not Theatre
Code review exists to create shared accountability. When a reviewer approves a change, they are not stamping it as formally submitted. They are taking responsibility for it — saying they understand what it does, believe it is correct, and are willing to own the consequences if it is not.
That contract only functions if the reviewer can actually evaluate the code. If the developer who wrote the code cannot explain it, and the reviewer approves it without understanding it, the review process becomes ritual validation. It has the form of accountability without the substance.
The Knight Capital failure in 2012 is not a story about AI-generated code — it predates the concept entirely. But it is the clearest available example of what happens when the accountability chain breaks in a high-stakes system. A deprecated trading algorithm was reactivated through a reused configuration flag. The code had no position limits, no circuit breaker, no automatic stop. Engineers reviewed the deployment. Nobody stopped it. Forty-five minutes after markets opened, $440 million was gone. The system did exactly what it was configured to do. Nobody had fully modelled what that was.
The lesson is not that those engineers were negligent. It is that review without genuine understanding does not function as a safety mechanism. "It passed the tests" is not sufficient. "It compiled" is not sufficient. Understanding is the precondition for accountability.
Historical Failure Patterns
These failures predate AI-generated code, and vibe coding did not cause them. They matter here because they illustrate the same class of problem — gaps in understanding, validation, or accountability that compound into outcomes nobody intended — operating at severe consequence.
The Therac-25 killed six people. A single developer, no independent review, software that had never been tested at the edge cases that mattered. The assumption that the system worked because it had worked before.
The Ariane 5 destroyed itself sixty-four seconds after launch. Reused code from a different rocket, carrying assumptions specific to the Ariane 4 flight profile that nobody validated against Ariane 5 conditions. A $500 million rocket destroyed by an integer overflow that had been safe — in a different context.
The Boeing 737 MAX failures were a sociotechnical failure: design assumptions, organisational pressure, certification gaps, training decisions, and software behaviour interacted in ways the system as a whole had not adequately modelled. 346 people died not from one bad decision but from a chain of individually defensible decisions that nobody had fully mapped as a system.
The common thread is not malice. It is a gap between what the people operating the system believed about its behaviour and what it actually did under conditions they had not adequately tested or considered. Vibe coding does not invent this class of failure. It creates structural conditions where it becomes more likely.
Responsible AI Use
None of this is an argument against AI-assisted development. The productive version of this workflow is real and valuable.
Sources like Google Cloud, IBM, and Red Hat describe vibe coding as a genuine evolution in how software gets written — faster, more accessible, and particularly useful for prototypes, MVPs, and simpler tasks. But those same sources converge on one essential point: without review, tests, and human understanding, generated code becomes fragile. The problem is not the existence of the tool. It is the temptation to use it as a substitute for engineering judgement rather than an amplifier of it.
AI can generate a correct implementation of a well-understood pattern. It can accelerate exploration. It can help a competent engineer move faster. What it cannot do is carry the domain understanding, institutional memory, failure-mode reasoning, and operational accountability that critical systems require. That remains with the people who wrote the code, reviewed it, and deployed it.
AI-generated code should not enter a critical codebase unless:
- the developer can explain what it does and why it is correct;
- the relevant domain assumptions are explicit;
- edge cases and failure modes have been considered;
- tests cover the behaviour that matters, not just happy paths;
- concurrency, persistence, retries, idempotency, and rollback behaviour are understood where relevant;
- code review validates behaviour, not just style;
- observability exists for the failure modes the change could introduce;
- a human owner is accountable for the code after deployment.
The useful question is not whether AI wrote the code. It is whether the team can explain it, test it, operate it, and own it.
Conclusion
The risk of AI-assisted development is not that AI writes bad code. It is the normalisation of deploying code that nobody in the review chain adequately understands. In systems where failure has no real consequence, this is a quality problem. In systems where failure means financial loss, data corruption, or harm to people who depend on the software without knowing it exists, it is an engineering and governance failure.
This is not a new failure mode. Every major software disaster documented on this site involved some version of the same gap: code whose behaviour was not sufficiently understood by the people responsible for it, operating in conditions nobody had adequately modelled. AI makes it easier to produce large quantities of code quickly. It does not change the fundamental requirement.
The question is not whether to use AI in software development. The question is whether the team can answer, honestly, for every piece of code in a critical system: who owns this, who understands it, and what happens when it fails?
If the answer is no, the team has shipped risk. In critical systems, that risk does not stay contained.