What is AI red teaming?

AI red teaming is structured adversarial testing of an AI system to find flaws before attackers or users do. NIST defines it as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers."

In practice, a red team (people, software, or both) tries to make a model, chatbot, RAG app or agent do something it should not: reveal data, follow injected instructions, produce harmful content, misuse tools, or leak its system prompt. The output is a list of findings with reproduction steps, not a pass/fail certificate.

Where the term comes from

Microsoft notes that "red teaming" historically meant systematic adversarial attacks for testing security vulnerabilities. With large language models, the term "extended beyond traditional cybersecurity" to cover probing for harmful outputs, not just exploitable bugs. That widening is why AI red teaming overlaps with, but is not the same as, penetration testing.

What AI red teaming covers

NIST AI 600-1 action MS-2.7-007 says to "Perform AI red-teaming to assess resilience against" three groups of threats:

• abuse of the AI system to facilitate attacks on other systems;
• generative AI attacks, for example prompt injection;
• machine-learning attacks, for example data poisoning, membership inference and model extraction.

The OWASP Top 10 for LLM Applications (2025) adds application-level weaknesses that a red team should probe:

1. LLM01: Prompt Injection
2. LLM02: Sensitive Information Disclosure
3. LLM03: Supply Chain
4. LLM04: Data and Model Poisoning
5. LLM05: Improper Output Handling
6. LLM06: Excessive Agency
7. LLM07: System Prompt Leakage
8. LLM08: Vector and Embedding Weaknesses
9. LLM09: Misinformation
10. LLM10: Unbounded Consumption

OWASP's separate GenAI Red Teaming Guide (released 2025-01-22) splits the work into four areas: model evaluation, implementation testing, infrastructure assessment, and runtime behaviour analysis. For autonomous agents, OWASP published a Top 10 for Agentic Applications (2026) in December 2025, and the October 2026 MITRE ATLAS release (version 2026.10: 129 techniques, 92 sub-techniques, 44 mitigations, 77 case studies) added agent-related techniques and case studies.

Model or application? Both

Microsoft recommends testing the base model together with its safety system (usually through the API), then the application itself (best through the user interface), and then both again before and after mitigations, on the production UI where possible. A red team that only hits the model API misses the retrieval, tools and guardrails that make up the real product.

Humans, automation, or both

Microsoft's AI Red Team lists "Automation can help cover more of the risk landscape" and "The human element of AI red teaming is crucial" as separate lessons from red teaming 100 generative AI products. Its guidance also recommends an initial round of manual red teaming before systematic measurement. Commercial vendors split along this line: automated platforms, human-led services (consultancies and crowds), and a few that sell both. The comparison table groups them that way.

What AI red teaming is not

• Not safety benchmarking. Microsoft lists "AI red teaming is not safety benchmarking" as a lesson. Benchmarks measure known behaviours on fixed datasets; red teaming looks for unknown failures.
• Not a replacement for measurement. Microsoft's guidance says red teaming is "not a replacement for systematic measurement". It finds issues that measurement then tracks.
• Not a one-off. No standard sets a cadence, but Microsoft recommends iterating with and without mitigations, and its red team writes that "The work of securing AI systems will never be complete." Several platforms sell continuous or CI/CD-triggered runs.

Who should run it

NIST AI 600-1 action MS-1.3-003 says to verify that people conducting structured human feedback exercises "are not directly involved in system development tasks for the same GAI model". Microsoft recommends a diverse team with both adversarial and benign-user mindsets. An outside vendor is one way to get that independence; an internal team with a clear separation from the builders is another.

What the deliverable looks like

Microsoft suggests a regular report with the top issues, a link to the raw data, the plan for the next round, and acknowledgement of the red teamers. Vendor deliverables vary: Bugcrowd's AI Pen Test includes a final report and one retest; Pillar Security says its findings include transcripts and reproduction steps. Ask any vendor for a sample report before buying.

Is it required?

In the EU, Article 55 of the AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing (applicable since 2025-08-02). In the US, no source opened for this site shows a binding federal mandate for private companies; NIST AI 600-1 is voluntary. Details: EU AI Act and red teaming.

Next steps

• Compare the 23 vendors by delivery model, ownership and coverage.
• Start with a free open-source tool such as promptfoo, garak or PyRIT.
• See what it costs (mostly quote-based).
• Read the buyer FAQ.