AI-Assisted Phishing: Why Defenders Still Have the Edge
AI now writes spear-phishing that looks tailored, timely, and almost indistinguishable from real internal mail, which is why legacy email filters are missing attacks that exploit context instead of keywords. This post shows what behavioral analysis and LLM-based detection can catch—and where human defenders still outperform the model.
When the Phish Reads Like an Internal Thread
In February 2024, finance staff at a Hong Kong firm were tricked into wiring out $25 million after joining a video call with deepfake versions of the company’s CFO and colleagues. That wasn’t a clumsy “urgent invoice” scam with bad grammar; it was a tailored social-engineering operation built around the victim’s own org chart, calendar pressure, and trust in familiar names. The useful lesson isn’t that AI made phishing “smarter.” It’s that attackers finally have a cheap way to scale the part that has always worked: context.
Legacy email filters were built for a world of obvious tells — misspellings, bad domains, malware attachments, and the occasional “Nigerian prince” from a throwaway mailbox. That world is gone. Today’s phish often arrives from a compromised Microsoft 365 tenant, uses a real thread hijack, and asks for something boring: resend the payroll file, approve the DocuSign, confirm the bank change. If your detection stack still leans on keywords and reputation alone, you’re screening for the wrong crime.
Why LLM-Generated Phish Slip Past Keyword Filters
The old playbook was easy to spot because it was lazy. A lot of commodity phishing still is. But LLMs let attackers generate a dozen variants of the same lure in seconds, each one tuned to a target’s role, region, and vendor stack. If the victim works in procurement, the mail references NetSuite or SAP Ariba. If they sit in HR, it references Workday, benefits enrollment, or an “updated policy” in SharePoint. If they’re in finance, it borrows the cadence of a real bank notice and wraps it around a fake wire-change form.
That matters because modern filters are still heavily optimized for static indicators: sender reputation, URL patterns, attachment hashes, and obviously malicious language. None of those are reliable when the attacker is using a freshly registered lookalike domain, a compromised mailbox at a real SaaS provider, or a message that contains no overtly suspicious words at all. In practice, the most convincing phish now looks like ordinary business mail with one wrong assumption baked in: “this request is normal because it resembles something we’ve seen before.”
Behavioral Analysis Catches the Stuff the Mail Gateway Misses
Behavioral analysis is where defenders still have leverage, because attackers have to do more than write a convincing paragraph. They need an execution path. That path leaves traces.
A message that impersonates a payroll vendor but comes from an account that has never emailed your tenant before is one signal. A request that arrives at 11:47 p.m. local time, then gets a follow-up from a second address 90 seconds later, is another. So is a link that leads to a credential-harvest page hosted on Cloudflare Pages, GitHub Pages, or a freshly spun AWS EC2 instance with no prior reputation. The content may be polished; the surrounding behavior usually isn’t.
This is where products like Microsoft Defender for Office 365, Proofpoint, and Mimecast earn their keep when they’re configured to look beyond content. The useful detections are not “this email sounds bad.” They’re “this sender has never interacted with these recipients,” “the reply chain was hijacked after a mailbox takeover,” or “the URL domain was registered 14 hours ago and is already being used in a multi-step login flow.” That’s not magic. It’s pattern matching across identity, infrastructure, and message timing.
LLM-Based Detection Works Best as a Triage Layer, Not a Judge
The fashionable mistake is to treat an LLM as a phishing oracle. That’s how you end up with a demo, not a control. LLM-based detection is most useful when it’s constrained to a narrow job: summarize intent, flag social-engineering cues, and compare a message against known internal communication patterns. It can also catch anomalies that humans miss at scale, like a request that claims to come from legal but uses procurement vocabulary, or a sender impersonation that preserves tone while breaking company-specific norms.
But LLMs are not great at certainty. They are probabilistic text engines, which means they can be seduced by fluent nonsense and overconfident about benign mail that happens to look “urgent.” They also struggle when attackers deliberately keep the language bland and operational: “Please review the attached draft and revert by EOD.” That sentence is not suspicious in isolation, which is exactly why it works.
The better use case is to feed the model structured signals alongside the text: sender history, DMARC results, thread lineage, attachment type, URL age, and whether the request matches known business processes. If the model is only reading prose, it is doing the attacker’s job for them. If it sees that a supposed vendor invoice came from a newly registered domain, used a free webmail relay, and asked for a bank update outside the normal ticketing system, it has something real to work with.
Human Review Still Beats the Model on Process and Motive
Here’s the part vendors hate: experienced analysts still outperform models when the question is not “does this email look phishy?” but “does this request make sense in this company?”
A human who knows that Okta password resets are never handled by email, that AP changes must be approved in Coupa, or that the CEO does not personally send OneDrive links at midnight can kill a lure faster than any classifier. They also notice the weird stuff models often miss: a request that is technically plausible but operationally backwards, a vendor who suddenly uses a different sign-off, or a thread that references a project code the sender should not know.
That is the contrarian bit nobody likes to say out loud: the best phishing defense is still process enforcement, not better prose analysis. If wire changes require a callback to a number already on file, if vendor updates require out-of-band verification, and if high-risk actions are gated behind MFA-resistant controls like FIDO2 keys, the quality of the phishing email matters a lot less. Attackers can write perfect English all day; they still have to cross a control boundary.
The Attacks That Still Break Teams
The failures that keep working are embarrassingly consistent. Compromised Microsoft 365 accounts are used to reply inside existing threads. Google Workspace tenants get abused to send “shared document” lures from real accounts. Dropbox, DocuSign, and SharePoint links are used as delivery vehicles because they inherit trust and often bypass coarse URL blocking. And once a victim lands on the credential page, the attacker uses AiTM kits like Evilginx-style reverse proxies to steal session cookies and sidestep MFA prompts.
That is why “user awareness” training alone is a joke when it is limited to spotting typos and fake logos. The mail is often not obviously fake. The real tell is that it is asking for a legitimate action in an illegitimate channel. Teach people to verify channel, not style. If the request is about money, credentials, or access, the email itself is never the place to finish the transaction.
The Bottom Line
Put your detection effort into message behavior, sender history, and business-process mismatch, not just content scoring. Require out-of-band verification for bank-detail changes, payroll edits, and privileged access requests, and make sure those approvals cannot be completed from the same mailbox thread that delivered the request.
If you are testing LLM-based phishing detection, measure it against thread hijacks, compromised SaaS accounts, and bland requests with no obvious keywords. If it only catches cartoon phish, it is a lab toy.