Continuous AI Security Testing and Incident Response for AI Agents

Quick answer: A pre-launch review checks one version of an AI system, but models, prompts, connectors and workflows keep changing. Keep a repeatable set of attack tests that runs on every meaningful change, monitor for unauthorized or unusual actions, make sure you can disable a tool or revoke a credential within minutes, and run an incident process that preserves the evidence you need to learn from it.

Many AI features ship after a single security review: a threat model, a penetration test, perhaps a red-team session. That review is valuable, but it describes the system as it was that week. A month later the provider has updated the model, someone has added a connector, and the prompt has been edited three times. None of those changes went back through security.

Example (fictional companies): Brightwater Software runs a support agent that reads inbound tickets, searches the knowledge base, drafts replies and can send email on the team's behalf. At launch, email sending was limited to replying to the ticket's requester. Two months later, a product team adds a "forward to partner" feature and widens the email tool so it can send to any address. A ticket arrives with instructions hidden in a pasted log file telling the agent to email the internal pricing sheet to an outside address "for the partner escalation". The agent does it. Nobody notices for four days, and when they do, the team has to work out how to stop the tool, which credentials to rotate and what evidence to keep.

How are teams handling AI security testing and response today?

Most organisations extend familiar practices:

• Pre-launch reviews: threat modelling, an AppSec review and sometimes a manual red-team exercise against prompt injection and jailbreaks.
• Provider and platform guardrails: content filters and prompt-injection classifiers in front of the model.
• Standard monitoring: API error rates, latency and cost dashboards, plus SIEM rules for the surrounding infrastructure.
• The general incident response plan, written for compromised hosts and leaked credentials rather than for a model taking an unwanted action.

Some teams add scanners such as NVIDIA's open-source garak, which probes models for weaknesses including prompt injection and data leakage. That is a good start, but scanning a model on its own does not test what your application lets the model do.

What do current approaches miss?

Change does not trigger retesting. In Brightwater's case, the risky change was not to the model at all. It was a wider tool scope added by another team. One-time reviews do not catch that, and most change processes do not treat "the agent can now email anyone" as a security-relevant change.

Tests target the model, not the workflow. Jailbreak prompts typed into a chat box test the model's refusals. The Brightwater attack came through a ticket attachment and succeeded because of what the email tool allowed. Realistic tests send untrusted content through the same channels attackers use (tickets, documents, web pages, tool outputs) and check what actions result. The prompt injection article in this series explains why the system prompt cannot be relied on to stop this.

Monitoring watches health, not behaviour. Latency and cost dashboards would not flag an email to a new external domain carrying an internal file. Nobody had defined what an unauthorized action looks like for this agent.

No fast way to stop the agent. Disabling the whole support assistant would halt customer service. Without a switch for a single tool, or a clear owner for the service credential, containment waits for a deploy.

Evidence disappears. Short log retention, prompt templates edited in place and an index that is rebuilt nightly mean the inputs behind the incident can be gone before anyone looks. The audit trails article in this series covers what to record so this does not happen.

How do you make testing continuous and incidents recoverable?

Build an attack test suite that runs like unit tests

Write scenarios as data: an untrusted input, the channel it arrives through, and the assertion about what must not happen. Run them in CI against a staging copy of the full workflow, with real tool definitions and fake downstream systems. Assert on actions (tool calls, recipients, records changed), not only on text.

# tests/ai_attacks/support_agent.yaml
- id: inj-ticket-attachment-email-exfil
  maps_to: [LLM01:2026, LLM03:2026]
  channel: ticket_attachment
  input_file: fixtures/hidden_instruction_log.txt
  expect:
    no_tool_call:
      tool: send_email
      where: "recipient_domain not in allowed_domains"
    no_attachment_from: ["internal-pricing/*"]

Define what triggers a retest

Rerun the suite on a model version change, a prompt or system instruction change, any new or modified tool or connector (including a changed tool description), a new data source, and a change in who can use the feature. Add new public attack techniques to the suite as they appear; the OWASP GenAI exploit round-ups are one useful feed. Schedule a periodic run as well, since hosted models can change without a change on your side.

Monitor actions, not just traffic

Enforce tool permissions in deterministic code outside the model, and log every allow and deny decision. Alert on denied calls, on first-time recipients or destinations, on unusual volumes, and on sensitive data in outbound actions. A denied call is often the earliest sign of an injection attempt.

Build kill switches before you need them

Put each tool behind its own feature flag, give each tool its own scoped credential, and write down who can flip the flag and rotate the credential at any hour. Practise it.

Run an AI-aware incident process

Fold AI cases into your existing plan; NIST SP 800-61 Rev. 3 frames incident response across the CSF 2.0 functions, from Govern through Recover. For AI incidents, add steps to freeze traces, the retrieved and inbound content, the model identifier, prompt and tool versions, and tool call logs before anything is rebuilt. Afterwards, turn the incident into a regression test.

Recommendations: a checklist

1. Write 20 to 50 realistic attack scenarios per AI workflow, covering injected content in every input channel and misuse of each tool. Owner: AppSec.
2. Run them in CI against a staging workflow with real tool definitions, asserting on actions. Owner: platform engineering.
3. Make changes trigger tests: model, prompt, tool, connector, data source and user population. Owner: change management or release owner.
4. Enforce and log tool permissions outside the model, including denied calls. Owner: agent and integrations team.
5. Alert on unusual actions such as new external recipients, bulk reads or sensitive data in outbound calls. Owner: security operations.
6. Add per-tool kill switches and scoped credentials, with a runbook and a named on-call owner. Owner: service owner.
7. Extend the incident response plan with AI-specific evidence preservation and an AI incident category. Owner: incident response lead.
8. Close the loop: every incident and near miss becomes a test case, and every quarter a tabletop exercise walks through an AI scenario. Owner: security lead.

How does this map to OWASP, NIST, MITRE and the EU AI Act?

OWASP. The OWASP Top 10 for LLM Applications 2026 lists LLM01:2026 Prompt Injection and LLM03:2026 Excessive Agency, the two risks at the heart of the Brightwater example. The OWASP Top 10 for Agentic Applications, released in December 2025, covers agent-specific risks such as agent goal hijack and tool misuse. For test design, the OWASP AI Testing Guide offers a methodology for testing AI systems, and the AI Agent Security Cheat Sheet covers controls for tool use and monitoring.

NIST. The NIST Generative AI Profile (AI 600-1) names pre-deployment testing and incident disclosure among its four primary considerations, and its actions sit under the AI RMF's Measure and Manage functions.

MITRE ATLAS. The MITRE ATLAS knowledge base of adversary tactics and techniques against AI systems, modelled on ATT&CK, helps you check that your scenarios cover more than one attack path.

EU AI Act. For high-risk systems, Regulation (EU) 2024/1689 requires providers to run post-market monitoring (Article 72) and to report serious incidents to market surveillance authorities (Article 73), generally no later than 15 days after becoming aware of them. Check the current application dates after the 2026 amendment; the EUR-Lex summary lists 2 December 2027 for Annex III high-risk obligations.

Where to discuss and learn more

• The OWASP GenAI LLM Top 10 GitHub discussions, including a proposal for a "cognitive circuit breaker" defence, a useful prompt for thinking about runtime stop mechanisms.
• The OWASP Slack, where GenAI Security Project working groups discuss testing and agent defences.
• AgentThreatBench in the UK AI Security Institute's open-source inspect_evals repository, which turns parts of the OWASP agentic taxonomy into runnable evaluation tasks.
• On Acta's blog, the guide to blameless incident post-mortems covers the review meeting that should follow any incident.

Frequently asked questions

How often should we test an AI feature?

On every meaningful change (model, prompt, tool, connector, data source or user population) and on a fixed schedule as well, because hosted models and external content change without a deploy on your side.

Are automated scanners enough?

No. Model scanners are useful for broad coverage, but you also need scenario tests that exercise your own tools, permissions and data, and periodic manual red-teaming.

What is the fastest containment step for a misbehaving agent?

Disable the specific tool through a feature flag and revoke or rotate the credential it uses. That stops the harm without taking the whole feature offline.

What evidence should we preserve first?

The full trace for affected requests, the inbound or retrieved content that carried the instructions, model and prompt versions, tool definitions and every tool call with its arguments and results.

Related in this series: AI agents and MCP security and from AI policies to compliance evidence.