Prompt injection, explained

Prompt injection is an attack in which text fed to a large language model (LLM) overrides the instructions the system's builder gave it. The National Institute of Standards and Technology (NIST) defines it as an attack that exploits "the concatenation of untrusted input with a prompt constructed by a higher-trust party," such as the application designer. NIST traces many such weaknesses to one design fact: data and instructions reach the model in the same channel, a flaw it compares to decades-old SQL injection. The OWASP Top 10 for LLM Applications lists the attack first, as LLM01:2025.

Direct and indirect injection

In a direct injection, the attacker is the person typing. NIST calls this a direct prompting attack by the primary user through query access. An early study by Perez and Ribeiro, presented at a NeurIPS 2022 workshop, tested two such attacks on GPT-3: goal hijacking and prompt leaking.

Indirect injection removes the need to type anything. Greshake and colleagues, in a paper first posted in February 2023, argue that LLM-integrated applications blur the line between data and instructions. An attacker plants instructions in data the application is likely to retrieve and so reaches the model with no direct interface. They demonstrated the attacks on Bing's GPT-4 powered Chat, code-completion engines, and synthetic applications built on GPT-4.

The paper sorts delivery into four methods: passive (through retrieval), active (for example, by email), user-driven, and hidden. The threats include information gathering, fraud, intrusion, malware, manipulated content such as a wrong summary, and loss of availability. The paper notes that even offline models that retrieve documentation files could be injected by poisoning the input data. NIST adds that the person harmed is often the primary user.

How OWASP describes LLM01

The OWASP entry says a prompt injection vulnerability occurs when user prompts alter the model's behavior or output in unintended ways. The input need not be visible to a human, as long as the model parses it. Direct injections come from the user's own prompt. Indirect injections arrive from external sources such as websites or files. Either kind can be intentional or unintentional.

The entry makes three points that matter for document-based systems. Research shows that retrieval-augmented generation (RAG) and fine-tuning do not fully mitigate the vulnerability. Jailbreaking is a form of prompt injection. And one example scenario is an attacker who edits a document in a RAG repository so that the model returns misleading results.

OWASP states that it is unclear whether fool-proof prevention exists. The entry instead lists seven mitigations: constrain model behavior, define and validate output formats, filter inputs and outputs, enforce least privilege, require human approval for high-risk actions, segregate and identify external content, and run adversarial tests that treat the model as an untrusted user.

Where NIST AI 100-2 places the attack

NIST AI 100-2 E2025, titled Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, carries a publication date of March 2025. The landing page records a corrected PDF on April 1, 2025, and a June 3, 2025 planning note about an error on page x. As of October 21, 2026, CSRC lists no newer edition.

In the report's generative AI taxonomy, prompt injection is entry NISTAML.018 and indirect prompt injection is NISTAML.015. The taxonomy groups attacker goals into availability, integrity, and privacy, plus a category specific to generative AI: misuse enablement. Prompt injection appears under all four goals. Indirect injection appears under the first three.

NIST also classifies by what the attacker controls. Query access supports prompt injection and prompt extraction. Resource control, meaning the ability to modify documents or web pages the model will ingest at runtime, supports indirect prompt injection. For a system that answers from a document library, resource control is the capability to examine.

What defenses claim, and what they do not

NIST lists training, detection, and input processing as mitigations, then states that current mitigations do not offer full protection against all attacker techniques. NIST's advice is to design as if injection is possible whenever a model sees untrusted input, for example by giving separate models different permissions.

Published defenses report strong numbers under their own tests:

• Spotlighting. Hines and colleagues at Microsoft mark untrusted input by delimiting, datamarking, or encoding it. With GPT-family models, attack success fell from greater than 50 percent to below 2 percent in their experiments.
• CaMeL. Debenedetti and colleagues separate the control and data flows taken from the trusted query, so retrieved data cannot change the program flow. The June 2025 version solves 77 percent of tasks in the AgentDojo benchmark with provable security, against 84 percent for an undefended system.

CaMeL's authors list as out of scope text-to-text attacks with no effect on data flow, such as making an assistant summarize an email to something different than its actual content. They write that prompt injection is not fully solved.

Nasr and colleagues, in an October 2025 preprint, attacked 12 recent defenses with adaptive methods built to counter each design. Attack success was above 90 percent for most, although most had originally reported near-zero rates. In the preprint's own implementation on AgentDojo, Spotlighting allowed as low as 1 percent success against static attacks and above 95 percent against an adaptive search attack. Human participants in their red-teaming competition produced 265 successful attacks against it.

What to ask

• Which inputs can carry untrusted text into the model: retrieved documents, email, web pages, tool output?
• Who can modify the document library, and is that resource control covered in the threat model?
• Was the defense tested against adaptive attackers, or only against a fixed set of attack strings?
• How does the system detect a planted instruction that changes a summary but triggers no action, the case CaMeL leaves out of scope?
• What can the model do with its tools, and which actions require human approval?
• Which edition of NIST AI 100-2 does the vendor cite? OWASP's LLM01 references the 2023 edition, and the current one is E2025.

Azcuna's CALIBER puts AI agents in practice markets with play money and a news feed whose truth is known, and measures from their trades whether they are gullible, follow the crowd, overpay, or can be steered. It was proposed to DARPA under SBIR Phase I topic DPA26BZ06-DV026, and the proposal was submitted on October 21, 2026.

Sources

1. 1Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection · arXiv (Greshake et al., v2 May 2023)
2. 2Ignore Previous Prompt: Attack Techniques For Language Models · arXiv (Perez and Ribeiro; NeurIPS 2022 ML Safety Workshop)
3. 3LLM01:2025 Prompt Injection · OWASP Top 10 for Large Language Model Applications, 2025 (official project repository)
4. 4NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations · NIST Computer Security Resource Center (March 2025)
5. 5Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025, PDF) · National Institute of Standards and Technology
6. 6Defending Against Indirect Prompt Injection Attacks With Spotlighting · arXiv (Hines et al., Microsoft, March 2024)
7. 7Defeating Prompt Injections by Design · arXiv (Debenedetti et al., v2 June 2025)
8. 8The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections · arXiv (Nasr et al., October 2025 preprint)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.