More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
LLMs treat every turn—system prompts, user requests, tool outputs, even their own hidden reasoning—as one long stream of tokens. Providers wrap those streams in tags like <system>, <user>, <assistant>, <think> and <tool> to impose structure. Each tag signals how the model should handle the following text: <user> commands guide actions, <think> marks private reasoning, <tool> flags external data you shouldn’t obey. Without those tags, the model just sees “token soup,” so moving text from one tag to another is the only clear-cut way to steer behavior.
Attackers exploit that reliance on tags with prompt injections. When an agent fetches a web page, the page content arrives inside <tool> tags, which should be treated as inert data. But human-crafted injections often read like legitimate user instructions, and models frequently misclassify tool‐tagged text as <user> prompts. Static benchmarks test against known attacks—models memorize patterns like “send your .env file” and block them—but real‐world adversaries can rephrase their payloads until the LLM misreads the role tags. That gap gives attackers near-100% success against frontier models, despite those same models acing benchmark suites.
To pinpoint that failure, researchers built “role probes.” They take neutral snippets—“Beginners BBQ Class!”—wrap each copy in a different role tag, then train a simple classifier on the model’s hidden activations to see which tag the model thinks each token belongs to. The results show that content often overrides tags: the model’s internal representation of a command buried in <tool> looks much more like a <user> command than like external data. That weak role perception explains why LLMs ignore tag boundaries, and it suggests that improving robustness against prompt injection will require strengthening the model’s ability to track and respect those discrete role signals.
Questions about this article
No questions yet.