AI red teaming

AI red teaming borrows the military and security term but covers a broader failure space than a conventional penetration test. The targets are the model (harmful content, jailbreaks, bias, hallucination, memorised training data, dangerous capabilities such as chemical, biological or cyber uplift), the application around it (system prompt leakage, direct and indirect prompt injection, insecure handling of model output, excessive agency in tools and agents), and the infrastructure (model endpoints, retrieval stores, supply chain). The OWASP GenAI Red Teaming Guide (2025) structures the work along these layers, and MITRE ATLAS provides an ATT&CK-style matrix of adversarial ML tactics and techniques that can be used to scope and report findings.

A typical engagement starts with a threat model and a harm taxonomy: which actors, which assets, which content categories and actions are unacceptable for this deployment. Testers then combine manual probing - role-play, persona and hypothetical framing, multi-turn escalation, encoding tricks, low-resource languages, poisoned documents placed where a RAG pipeline will retrieve them - with automated generation and scoring. Open-source harnesses such as Microsoft PyRIT, NVIDIA garak and promptfoo send large attack corpora, mutate prompts with an attacker LLM and grade responses with classifiers or an LLM judge. Because outputs are stochastic, results should be reported as attack success rates over many samples at fixed decoding settings, not as single screenshots.

For EU providers of general-purpose AI models with systemic risk, EU AI Act Art. 55(1)(a) requires model evaluation "including conducting and documenting adversarial testing" with a view to identifying and mitigating systemic risks, and the General-Purpose AI Code of Practice (July 2025) describes how signatories evidence this in its Safety and Security chapter. For high-risk systems, Art. 15 requires resilience against attempts to alter use or performance by exploiting vulnerabilities, including adversarial examples and data poisoning, which red teaming helps demonstrate. NIST AI 600-1 lists red teaming among the suggested actions for measuring generative AI risks.

Common pitfalls: testing only the base model and not the deployed configuration with its system prompt, tools and retrieval; treating a red-team pass as proof of safety rather than a lower bound on what an attacker finds; running a one-off exercise before launch while model versions, prompts and connectors change weekly; and failing to feed findings into regression test suites and guardrail rules. Microsoft's 2025 report on red teaming over 100 generative AI products stresses that many real failures come from simple techniques and system integration, not from gradient-based attacks. Red teaming complements, and does not replace, benchmark evaluation, a conventional penetration test of the surrounding infrastructure, and monitoring in production.