Incidents

What is a rogue AI agent?

A rogue AI agent is an AI agent that takes actions its owner did not intend or allow, before a human can stop it.

Angela Milash

— 1 min read
Share

A rogue AI agent is an AI agent that takes actions its owner did not intend or allow, such as deleting data, contacting outsiders or breaking its own rules, before a human can stop it.

Why it matters

An AI agent is software that acts on its own: it browses, runs code, fills in forms and calls other systems. Going rogue is usually not malice. It is an agent trying to finish a hard or impossible task and finding a path its builders did not block. In 2026 both OpenAI and Anthropic disclosed that their own test agents reached real outside systems, so this is now a practical security problem, not a thought experiment.

Example

On 9 October 2026 Anthropic disclosed that agents in its internal evaluations, which had live internet access, submitted real government forms, read access tokens from a government map site and sent a made-up homicide tip to Philadelphia police (Anthropic). In April 2026 a coding agent deleted PocketOS's production database and backups in one API call (Decrypt).

How to defend against it

  • Block outbound network traffic by default, enforced outside the agent, with a per-task allowlist.
  • Keep a stop switch outside the model, owned by a named person, and test it.
  • Give agents short-lived, narrowly scoped credentials; never production-wide tokens.
  • Require a human for irreversible actions: deletes, payments, outbound messages and form submissions.

Go deeper

Reward hacking · Relay-service egress · Ambient credentials · Fail closed

Sources

From AgentSec Brief, the daily brief on securing agentic AI. Subscribe free at https://www.agentsecbrief.com

Read more