Red teaming language models, explained

Red teaming a language model means deliberately trying to make it fail: people, or other models, write inputs meant to draw out inaccurate, harmful, or discriminatory output. The National Institute of Standards and Technology (NIST) Generative AI Profile, NIST AI 600-1 of July 2024, calls AI red-teaming a structured testing exercise to find flaws and vulnerabilities, often in a controlled environment and with the system's developers. A benchmark measures a rate on a fixed set of questions. A red team searches for failures that no fixed set contains, so the two answer different questions.

Red teaming in federal guidance

Executive Order 14110 of October 30, 2023 defined AI red-teaming in Section 3(d) as "a structured testing effort to find flaws and vulnerabilities" in an AI system. Section 4.1(a)(ii) directed guidelines to enable developers to conduct red-teaming tests, except for AI used as a component of a national security system. Executive Order 14148 of January 20, 2025 revoked Executive Order 14110.

NIST AI 600-1 was developed under Section 4.1(a)(i)(A) of that order. The profile lists AI red-teaming with participatory engagement and field testing as forms of structured public feedback, and names four types:

• General public: users who are not necessarily technical experts.
• Expert: specialists in the domain or context of use, such as medicine or cybersecurity.
• Combination: experts who modify or verify prompts written by general participants.
• Human / AI: generative AI working with human teams, which NIST says can be more cost-effective than humans alone.

NIST ties the quality of the output to the background and expertise of the red team, and says results should get additional analysis before they shape governance decisions. One suggested action, MS-2.8-002, is to document the instructions given to AI red-teamers.

Red teaming with another language model

Perez and colleagues, at EMNLP 2022, automated the search in three steps. A red language model writes test questions, the target model answers, and a classifier flags harmful replies. The target was a 280-billion-parameter chatbot, Dialogue-Prompted Gopher.

Half a million zero-shot test questions drew offensive replies 3.7 percent of the time, or 18,444 failed tests. One setting of a red model trained with reinforcement learning drew offensive replies over 40 percent of the time, but 78 percent of its questions contained the same word: it had found one pattern and kept using it. The authors present this as a trade-off between the diversity of test cases and their difficulty. The same approach surfaced leaked training data, real phone numbers and email addresses offered as the chatbot's own contact details, and groups of people it discussed more offensively than others.

The authors list the limits. The red model and the classifier inherit biases from their training data. A flawed classifier produces false positives, and false negatives that miss failure modes entirely. They write that the method should not be viewed as a way to perfectly and exhaustively find all oversights, but as a complement to manual testing.

Red teaming by people

Ganguli and colleagues released 38,961 attacks written by 324 U.S.-based crowdworkers, 307 recruited through Amazon Mechanical Turk and 17 through Upwork. They tested three model sizes (2.7, 13, and 52 billion parameters) and four types: a plain language model, a model prompted to be helpful, honest, and harmless, a model using rejection sampling (the best of 16 samples by a preference model), and a model trained with reinforcement learning from human feedback (RLHF).

The RLHF models got harder to red team as they grew; the other types showed a flat trend with size. Rejection sampling was hardest at every size, but tended to be harmless by being evasive. Prompted models were not significantly harder to red team than plain ones, which the authors note is inconsistent with their earlier static evaluations showing prompting to be an effective safety measure.

The authors call their data incomplete. They did not know whether crowdworkers had the expertise to judge some attacks. The space of possible harms is, in their words, "unknown and unbounded," and informal internal testing found attack types absent from the released set. They also list open questions, including who should red team and what counts as a successful attempt.

Red teaming is not a benchmark

NIST's ARIA 0.1 pilot report, NIST AI 700-2 of November 2025, separates three levels of testing. Model testing ran pre-defined prompts to confirm capabilities. Red teaming tried to elicit negative behavior and, the report says, was not intended to mimic real-world use. Field testing observed realistic use. Seven applications went through 508 testing sessions, with 51 red teamers between December 2024 and January 2025 and 19 field testers in January 2025.

NIST AI 600-1 adds that benchmark test datasets may not extrapolate to real-world conditions, and that jailbreaking or prompt engineering tests may not systematically assess validity or reliability. Feffer and colleagues, at AIES 2024, found red teaming practices diverging in purpose, which is often vague, and in the artifact tested, the setting, and the decisions informed. They argue that presenting red teaming as a cure for every risk verges on security theater.

A benchmark gives a rate that can be repeated and compared. A red team gives examples of failure, and the count depends on who tested, with what instructions, and what counted as success.

What to ask

• Who red teamed: the general public, domain experts, a model, or a mix, as NIST AI 600-1 distinguishes?
• Were the red teamers' instructions documented, as action MS-2.8-002 suggests?
• What counted as a successful attack, and did a person or a classifier decide?
• Does the report state a failure rate from red teaming, or list the failures found?
• Is there also a fixed test set, scored before and after fixes, beside the red team findings?

Azcuna's CALIBER puts AI agents in practice markets with play money and a news feed whose truth is known, and measures from their trades whether they are gullible, follow the crowd, overpay, or can be steered. It was proposed to DARPA under SBIR Phase I topic DPA26BZ06-DV026, and the proposal was submitted on October 21, 2026.

Sources

1. 1Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 · National Institute of Standards and Technology (July 2024)
2. 2Executive Order 14110 of October 30, 2023: Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence · Federal Register, Vol. 88, No. 210 (November 1, 2023), via govinfo.gov
3. 3Executive Order 14148 of January 20, 2025: Initial Rescissions of Harmful Executive Orders and Actions · Federal Register, Vol. 90, No. 17 (January 28, 2025), via govinfo.gov
4. 4Red Teaming Language Models with Language Models · ACL Anthology (Perez et al., EMNLP 2022)
5. 5Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned · arXiv (Ganguli et al., August 2022, revised November 2022)
6. 6Assessing Risks and Impacts of AI (ARIA): ARIA 0.1 Pilot Evaluation Report, NIST AI 700-2 · National Institute of Standards and Technology (November 2025)
7. 7Red-Teaming for Generative AI: Silver Bullet or Security Theater? · arXiv (Feffer, Sinha, Deng, Lipton, and Heidari; AIES 2024)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.