How should teams contain and recover from AI engineering agent failures?

Agent failure is not limited to a crash. An agent may complete the wrong task, make repeated plausible changes, expose information, follow hostile instructions or consume resources without reaching a result. It may report success while leaving the system worse.

Reliable use therefore depends on the surrounding engineering environment being able to contain, detect and recover from unexpected behaviour.

Define credible failure scenarios

Map the agent's goal, context, tools and possible effects. Consider misunderstanding, stale instructions, prompt injection, unsafe commands, uncontrolled loops, cost overrun, conflicting agent output, partial changes and corrupted state. Include external service failure and loss of audit data.

Rank scenarios by consequence, detectability and reversibility. A bad edit on an isolated branch differs from a destructive operation against production data. Define success and prohibited outcomes so the system can recognise more than technical exceptions.

Use past incidents and near misses from normal delivery. AI may amplify existing weaknesses rather than invent entirely new failure modes.

Contain the blast radius

Give the agent a task-scoped identity, minimum tools and data, a limited workspace, time and cost budgets, and approved network destinations. Separate research, modification, merge and deployment permissions. Prevent it from weakening or changing the controls that govern it.

Use branches, worktrees, sandboxes and disposable environments where suitable. Protect shared state and production credentials. For multi-agent work, limit what agents can pass to one another and define who resolves conflict.

Containment should be technical rather than dependent on prompt wording. Instructions help behaviour but do not enforce authority.

Detect, stop and restore

Monitor actions, denied requests, unexpected scope, repeated failures, resource use and changes to important files or configuration. Define automatic stop conditions and an authorised manual kill route. Revocation must cover identities, tokens, sessions and connected tools.

Preserve logs, diffs, inputs and outputs needed for investigation while protecting sensitive information. Restore from a known version or clean environment and verify that hidden or external effects have not remained. Data operations may require compensation rather than simple rollback.

Escalate according to customer, security and operational effect. Product helps assess outcome and communication; Engineering owns technical recovery.

Exercise recovery before relying on autonomy

Run controlled scenarios in which an agent exceeds scope, receives malicious context, loses a tool or creates conflicting changes. Check whether responders can detect the event, stop all activity, reconstruct actions and recover within acceptable conditions.

After a failure, examine task design, context, permissions, tests, approvals and monitoring rather than blaming a model response alone. Record changes and retest them. Withdraw or narrow the capability if the environment cannot reliably control it.

Increase agency only when representative evidence shows that containment and recovery remain effective. Autonomy is not dependable because failure never occurs; it is dependable when failure can be bounded and handled.

Example

An agent updating configuration begins modifying unrelated deployment files after following an obsolete repository instruction. A path restriction blocks the writes, monitoring detects repeated denied actions and the session is stopped automatically.

The team preserves the proposed diff, revokes the session token and recreates the workspace. It corrects the obsolete instruction and adds a test scenario before resuming with the same narrow permissions. No production restoration is needed because containment worked.

FAQs

• Is a kill switch enough to control an AI agent?

  No. Stopping future actions does not undo completed changes or external effects. Combine it with least privilege, monitoring, evidence and tested restoration.

• Who should decide whether an agent can resume after failure?

  The accountable Engineering owner should decide using evidence from the failure and relevant Product, security or operational input. The agent should not restart itself after a material control event.

• Should every agent failure become an incident?

  Classify it by actual and potential consequence. Minor contained errors still provide learning, while security, customer or operational effects require the established incident process.