When AI Hallucinations Become a Security Vulnerability

When a Hallucinated Fix Becomes a Real Incident

CISA’s Known Exploited Vulnerabilities catalog now has more than 1,100 entries, which is a polite way of saying the internet is still full of things that are actively being used against you. So when an AI tool confidently tells you to patch the wrong service, trust a fabricated CVE, or declare a system “safe” because it misunderstood the evidence, that is not a cute accuracy problem. It is an operational risk with a keyboard.

You already know LLMs can be wrong. The part worth paying attention to is how they are wrong in ways that map neatly onto security failure. They do not just invent facts; they invent plausible facts. That is worse, because plausible nonsense gets copied into tickets, runbooks, Slack threads, and sometimes production change windows. One bad answer can become a bad decision chain. That is how you get an engineer burning time on the wrong nginx instance while the real exposure sits untouched.

The Failure Mode Is Confidence, Not Just Error

A hallucinated security answer is dangerous when it crosses from “incorrect” into “actionable.” If a model fabricates a CVE, the obvious failure is wasted time. The less obvious failure is misprioritization. Security teams already live in a triage economy, and false specificity is expensive. A made-up vulnerability number looks authoritative enough to survive a quick skim, especially when it is wrapped in the usual AI tone of serene certainty, which is apparently what passes for professionalism now.

This is not hypothetical. During breach investigations, the most damaging mistakes are often not the dramatic ones; they are the quiet ones where someone patches the wrong asset, rotates the wrong secret, or declares a control effective because the evidence was summarized badly. LLMs amplify that exact class of error because they are good at producing coherent narratives from incomplete input. Coherence is not truth. It is just a nicer font.

Why Security Advice Is a Bad Fit for Generic LLMs

Security guidance is full of edge cases, version-specific behavior, and vendor quirks. Apache Struts, OpenSSL, Microsoft Exchange, and Kubernetes all have failure modes that depend on exact versions, deployment patterns, and patch states. A model trained to produce the most likely next token is not naturally equipped to distinguish “this affects only default config” from “this is exploitable only if feature X is enabled.” That distinction matters when you are deciding whether to page someone at 2 a.m. or wait until morning.

The problem gets sharper with live threat intelligence. If you ask an LLM about active exploitation, it may blend old blog posts, stale advisories, and noise from forums into a single answer that sounds current but is not. CISA’s KEV catalog exists precisely because “known to be exploited” is a narrower and more useful standard than “sounds bad on the internet.” If your model cannot anchor itself to that kind of source, it is not doing security analysis. It is doing autocomplete with a badge.

The Real Risk Is Bad Automation, Not Bad Chat

The failure gets serious when the model is wired into workflows. GitHub Actions already showed how much trust people hand to automation by default; the 2025 compromise of tj-actions/changed-files was a reminder that CI/CD is privileged by design, not by accident. If you let an LLM generate remediation steps, update playbooks, or trigger tickets without verification, you are effectively giving a probabilistic system authority over deterministic infrastructure. That is a lovely way to create a self-inflicted incident.

This is where a lot of “AI for security” pitches quietly fall apart. A model that drafts a patch plan is not the same thing as a control that validates the patch plan. If it tells you to upgrade a package that is not installed, or to disable a service that is actually your compensating control, the output is worse than useless because it consumes trust. Dry joke: the machine is very confident right up until your pager starts being very loud.

The Guardrails That Actually Help

The first guardrail is boring and effective: force the model to cite primary sources, then verify them before action. That means vendor advisories, NVD entries, CISA KEV, GitHub Security Advisories, and your own asset inventory. If the answer cannot be traced to a real source, it does not get to influence remediation. No citation, no change. This is not bureaucracy; it is not letting a stochastic parrot run your patch queue.

Second, constrain the model’s job. Use it to summarize, classify, or extract, not to decide. Let it turn a 40-page advisory into a concise briefing, but keep the actual decision logic in deterministic tooling: version checks, SBOM matching, config validation, and policy engines. You do not need the model to be “smart” if your pipeline can tell you whether openssl 1.1.1w is present, whether the vulnerable code path is reachable, and whether compensating controls exist.

Third, add adversarial prompts to your testing. Ask the model to identify a vulnerability that does not exist, then see whether it hallucinates a CVE with enough confidence to fool a tired analyst. Ask it to explain a false positive as if it were real. If it fails those tests, do not put it in front of incident responders or platform engineers and pretend you have reduced risk. You have only moved the risk to a shinier interface.

The Advice You Should Ignore

The standard advice says to “use AI for triage” and “keep a human in the loop.” Fine. But if the human is just rubber-stamping whatever the model produced, that is not a control. That is compliance theater with a chatbot attached. The better rule is stricter: the model can suggest, but it cannot originate a security action that you would not trust from an unverified junior analyst on their first week.

Also, do not confuse internal deployment with safety. Running the model in your VPC does not fix hallucinations, and it certainly does not fix prompt injection, poisoned retrieval, or bad source material. OpenAI’s 2023 internal Slack breach was a reminder that even AI companies are not magically exempt from basic security failure. If the people building the tools can get surprised, your local wrapper is not a force field.

The Bottom Line

Treat LLM output as untrusted until you can tie it to a real advisory, asset, or control check. Use deterministic validation for versioning, exploitability, and remediation status, and keep the model out of any workflow that can make changes on its own.

If you want AI in security, make it narrow, auditable, and source-bound. Otherwise you are just automating confidence, and confidence is cheap right up until it costs you a breach.

References

• CISA Known Exploited Vulnerabilities Catalog: https://www.cisa.gov/known-exploited-vulnerabilities-catalog
• GitHub Security Advisory for tj-actions/changed-files compromise: https://github.com/advisories
• GitHub Actions documentation on permissions: https://docs.github.com/actions/security-for-github-actions/security-guides/automatic-token-authentication
• OpenAI 2023 incident discussion and reporting: https://www.theverge.com/2023/3/24/23655196/openai-chatgpt-history-breach-data-leak
• NVD (National Vulnerability Database): https://nvd.nist.gov/