Threat-Modeling an Autonomous AI Agent: Every Surface Under Attack

APT29, a Stolen Session, and Why Your Agent Will Be Next

APT29 didn’t need a zero-day to ruin your week. In the 2020 SolarWinds campaign, they lived off the land, abused trusted software updates, and turned legitimate admin paths into an intrusion path. That’s the part people keep forgetting when they bolt an “autonomous” agent onto real systems: attackers rarely fight the strongest control head-on. They wait for the thing you already trust to do the damage for them.

An AI agent is just a new trust broker with a bigger blast radius. It reads messages, calls tools, stores memory, and often acts with more privilege than the person who triggered it. Every one of those surfaces can be manipulated independently. If you model the agent as one blob, you miss the point. The model can be clean and the tool chain can still leak data. The prompt can be pristine and the memory store can still be poisoned. Security by vibes is still not a control.

The Model Is Not the Whole Attack Surface

Prompt injection is the obvious problem, but not the only one. If you let an agent ingest untrusted text from email, tickets, web pages, or Slack, you’ve created a command channel disguised as content. The model does not “understand” that a pasted invoice or support thread is hostile. It just sees tokens with enough structure to steer output. That is how you get data exfiltration through “helpful” summarization and unauthorized action through instruction smuggling.

The less obvious failure is model behavior under tool pressure. Once the agent can call APIs, the attacker doesn’t need to jailbreak the model in the abstract. They only need to shape the inputs so the model selects the wrong tool, passes the wrong arguments, or reveals the wrong context. This is why “just add a refusal policy” is weak tea. A model that refuses to answer a prompt can still be coaxed into calling send_email, create_ticket, or export_csv with attacker-chosen parameters. Congratulations, you’ve built a policy engine that can be socially engineered.

Tools Are the Real Blast Radius

Tool access is where the damage becomes measurable. If your agent can reach GitHub, Jira, Slack, Google Drive, or a cloud control plane, then prompt injection becomes a privilege escalation path, not a language-model curiosity. In the 2023 CircleCI breach, attackers stole an engineer’s SSO session from malware on a laptop and used that access to reach customer secrets and environment variables. Different mechanism, same lesson: once session trust is compromised, downstream systems tend to believe the caller. Agents inherit that problem instantly if you hand them long-lived tokens or broad OAuth scopes.

You should treat every tool call as an outbound request that needs its own authorization decision, not as a side effect of model output. That means per-action allowlists, scoped credentials, and explicit human approval for high-risk operations like secret export, permission changes, and external message sending. If the agent can rotate keys, invite users, or modify IAM without a second look, you’ve skipped the part where attackers usually fail. That’s not efficiency. That’s a shortcut to your incident report.

Memory Poisoning Is Slow, Quiet, and Annoying

Memory is where agents get useful and where they start accumulating attacker influence. Long-term memory stores user preferences, prior tasks, and summaries of prior sessions. If you let untrusted inputs update memory without provenance, you’ve created a persistence layer for malicious instructions. A poisoned memory entry can sit dormant until the agent later uses it to shape a decision, which makes triage miserable because the original injection is nowhere near the eventual action.

This is not theoretical. Microsoft Recall showed how “helpful” persistence can become a security problem when sensitive data is captured and later extracted from screenshots and local stores. Different product, same category error: if you keep more context than you need, you also keep more material an attacker can mine. The fix is not “encrypt it and move on.” You need source attribution, expiry, and a hard rule that memory can inform suggestions but not override current policy. Otherwise the agent starts treating yesterday’s attacker input like institutional knowledge. Very efficient. Very stupid.

Message Channels Are a Control Plane in Disguise

Slack, email, Teams, and ticketing systems are not passive input. For an autonomous agent, they are a control plane. That matters because attackers know how to weaponize conversation. The Uber 2022 breach started with MFA fatigue and social engineering, then moved through Slack access and internal tooling. The lesson is not “users are the weak link,” which is the sort of lazy line people use when they want to sound wise without doing threat modeling. The lesson is that conversational systems are already used to establish trust, so they are ideal for instruction smuggling.

If your agent reads messages and acts on them, you need message authentication at the application layer, not just account login at the identity layer. Signed commands, structured requests, and sender reputation checks matter. Natural language is fine for humans; it is a terrible authorization format. An attacker can always make a prompt sound like a request from a teammate. They can’t as easily fake a signed workflow event.

The Human Interface Is Where You Lose the Plot

The UI is the last mile, and it’s where most “agent safety” designs fall apart. If the agent presents a summary, a suggested action, and a one-click approve button, you’ve built a persuasion engine with a security badge. People will approve the thing that looks routine, especially when the agent has already done the hard work of framing it. That is not a user-experience issue. It is an attack path.

Here’s the contrarian bit: don’t overload users with endless confirmations. That just trains them to click through, which is how you get consent theater. Instead, reserve human approval for irreversible actions and require the UI to show the exact tool call, destination, and data fields involved. If the agent wants to email a file externally, the user should see the recipient, attachment hash, and originating source. If that sounds cumbersome, good. Security often is. The alternative is letting an LLM turn ambiguity into action.

Put Controls Where the Agent Can Still Be Stopped

The right control stack is boring in the best way. Separate read, write, and execute privileges. Use ephemeral credentials with narrow scopes. Gate tool execution with policy checks outside the model. Log raw prompts, tool arguments, and memory writes so you can reconstruct why the agent did something dumb at 2 a.m. And keep a kill switch that actually cuts off tool access, not just the chat window. If the agent can still act after you “disable” it, you disabled the wrong thing.

You also need red-team tests that target each surface independently. Test prompt injection through documents, tool abuse through malformed requests, memory poisoning through long-lived sessions, and message spoofing through Slack or email. Most teams test the model and call it a day, which is like checking the lock on the front door while leaving the side gate open and the garage code taped to the fridge.

The Bottom Line

Model safety is necessary and insufficient. Put authorization outside the model, scope every tool credential tightly, and make high-risk actions require explicit, reviewable approval. Then test the agent the way attackers will: through untrusted content, poisoned memory, and message channels that already carry trust.

If you can’t reconstruct exactly why the agent acted, you don’t have an autonomous system. You have a future incident with better branding.

References

• https://www.cisa.gov/news-events/cybersecurity-advisories/aa20-352a
• https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-347a
• https://www.microsoft.com/en-us/security/blog/2024/06/06/microsoft-recall-and-privacy/
• https://www.cisa.gov/news-events/cybersecurity-advisories/aa22-074a
• https://www.circleci.com/blog/january-4-2023-security-alert/