LLMs Have Made Phishing Indistinguishable From Legitimate Email
For years, typos and awkward phrasing were the telltale signs of phishing. Large language models just eliminated that detection signal entirely.
CVE-2024-3094 was caught because Andres Freund noticed SSH taking about half a second longer than it should have on a Debian system. That’s the kind of failure mode defenders like to imagine they’ll spot: a weird delay, a broken signature, a hostname that doesn’t resolve. Phishing has lost its equivalent tell. With GPT-4, Claude, and Gemini in the hands of anyone with a browser, the old giveaway — mangled English, bizarre punctuation, and “kindly do the needful” energy — is now optional.
The typo test is dead, and it was a flimsy test anyway
For years, security teams trained users to hunt for grammar mistakes as if attackers were bound by a style guide. That advice was always brittle. BEC crews working out of Nigeria, especially the networks Microsoft tracks as Octo Tempest and FIN7-adjacent operators, were already sending clean, targeted mail long before ChatGPT showed up. The difference now is scale: a single operator can generate 200 variants of the same lure, each tuned to the recipient’s role, region, and recent activity, without touching a human editor.
That matters because email filters have never been built to score “sounds weird to a native speaker.” They score sender reputation, URL reputation, attachment type, and patterns they’ve seen before. If the message is polished, the detection problem shifts from “spot the broken English” to “prove the sender is lying,” which is a much nastier job when the lie is wrapped in perfect syntax and realistic internal jargon.
LLMs didn’t invent phishing; they industrialized the personalization
The real change is not that attackers can now write better English. It’s that they can cheaply tailor messages to the exact friction point that makes a target act. A finance lead gets a fake wire approval referencing a real vendor from last quarter. A developer gets a “GitHub security alert” that mirrors the tone of GitHub’s own notifications. A recruiter gets a resume with plausible formatting and a cover note that doesn’t read like it was translated by committee.
This is not hypothetical. Proofpoint and Microsoft have both documented phishing kits and social engineering campaigns that already use AI-generated copy to vary subject lines, sender names, and body text at a rate no human team can match. The same goes for the junk infrastructure around the phishing email itself: attackers can spin up lookalike domains, generate believable landing pages, and feed stolen logos into a template in minutes. The point is not sophistication for its own sake. It’s throughput.
The old “bad English” signal was never the real defense
Security teams that leaned on language quality were mostly outsourcing judgment to a cheap heuristic. That heuristic failed in plenty of cases before LLMs. Business email compromise campaigns against real estate firms, payroll teams, and law offices have long used short, clean, and minimal prose because the attacker doesn’t need to write well; they need to create urgency. The message “Please review attached invoice” has been enough to drain six figures from a company for years.
The more useful signal has always been mismatch. Does the sender normally use this tone? Does this request fit the workflow? Does the message arrive from a domain that is one character off from the real vendor? Is the reply-to address different from the visible sender? LLMs don’t erase those checks. They just make it more embarrassing when a team ignored them and congratulated itself for catching a missing apostrophe.
Detection has to move to identity, infrastructure, and process
If you’re still trying to solve phishing at the content layer, you’re already behind. The practical controls are boring and effective: enforce FIDO2/WebAuthn for high-risk users, kill SMS-based MFA where possible, and make external email banners impossible to miss. Google’s own phishing-resistant login guidance exists for a reason; passkeys and hardware-backed keys remove a lot of the replay and prompt-bombing nonsense that AI-assisted lures depend on.
Then stop pretending a message is trustworthy because it looks native. Check whether the domain was registered last week. Check whether the sending infrastructure has a history with your tenant. Check whether the login page is hosted on a compromised WordPress site in a country your vendor has never used. Products like Microsoft Defender for Office 365, Proofpoint, and Abnormal Security are useful here not because they “understand AI,” but because they correlate sender behavior, impersonation patterns, and tenant-specific anomalies faster than a tired analyst can.
The contrarian bit: user training still matters, just not the way vendors sell it
The standard advice says “train users to spot phishing.” Fine, but don’t confuse awareness slides with a control. The useful training now is procedural: teach people to treat any request to change payment details, reset MFA, or export data as a second-channel verification problem. That means calling a known number from the directory, not replying to the email thread. It means verifying a wire request in Slack or Teams only if the request is independently authenticated, because those channels are increasingly part of the same attack chain.
Also, stop overfitting on “AI-generated” tells. Plenty of real phishing still looks sloppy because the attacker is lazy, not because the model couldn’t do better. The danger is assuming that polished prose equals legitimacy. It doesn’t. A cleanly written message from a compromised Microsoft 365 tenant with a valid thread history is far more dangerous than a broken-English blast from a throwaway domain.
LLMs made the first layer of phishing boring; the second layer is where the damage happens
The email is now just the opener. The actual theft happens when the target lands on a convincing login page, approves a malicious OAuth app, or hands over a one-time code to a live operator in a callback scam. That’s why attackers keep using Microsoft 365, Okta, and Google Workspace as the center of gravity: once they get a session token or an OAuth grant, the content of the original message stops mattering.
That should change what you monitor. Alert on new inbox rules, suspicious forwarding, impossible travel, and consent grants to unfamiliar apps. Watch for domain lookalikes that differ by a single character. Watch for vendor impersonation that uses real invoice numbers or real project names pulled from breached mailboxes. And if your SOC is still triaging phishing by reading the prose and deciding whether it “sounds off,” that’s not detection. That’s literary criticism.
The Bottom Line
Kill the assumption that bad grammar is a useful phishing indicator. Move your detection and response around identity, domain age, OAuth consent, inbox-rule changes, and verified out-of-band callbacks for payment or MFA changes.
Force FIDO2/WebAuthn for admins and finance users, disable legacy MFA paths, and alert on new forwarding rules or suspicious app consents in Microsoft 365, Google Workspace, and Okta. If the email is polished but the sender domain is new, the reply-to is off, or the request changes money or credentials, treat it as hostile until independently verified.