OpenAI's Astra model uses "opaque recurrence," a technique that makes AI reasoning harder to monitor by processing queries in loops rather than linear steps. Safety researchers worry this could scale into a problem where AI reasoning becomes completely invisible.

• Opaque recurrence reduces legible traces of AI reasoning, making chain-of-thought monitoring less effective than current methods
• Safety experts fear the technique could escalate into a "race to the bottom" where labs stop maintaining transparent reasoning altogether
• OpenAI says Astra's use is limited and they remain committed to chain-of-thought monitoring, but Anthropic and Google DeepMind are already exploring the same approach

The article shows how LLMs use discrete role tags (like <user>, <think>, <tool>) to structure context but also rely on writing style, causing them to misclassify malicious commands in tool-tagged text as user instructions. Using linear probes to measure internal “roleness” (CoTness, Userness, etc.), the authors reveal why prompt injections succeed and suggest focusing research on the science of roles to defend against such attacks.

• LLMs classify roles more by writing style/content than by the actual tags surrounding text, so tool-tagged content that reads like a command gets internally treated as a user instruction
• Researchers used linear probes on hidden activations (wrapping neutral text in different role tags) to show this "content overrides tag" effect directly in the model's internal representations
• This explains why prompt injections achieve near-100% success against frontier models in practice even though those same models score well on static benchmarks that only test memorized attack patterns
• The fix isn't more attack examples but strengthening models' internal tracking of role boundaries—a "science of roles" approach to defense

The author compares DNA genomes and large language model weights as passive information sequences—scores that only become meaningful when processed by cellular machinery or inference engines. Both arise from massive search processes (evolution and gradient descent) that compress vast experiences into lossy representations, leaving much uninterpreted “junk.” This parallel explains why it’s hard to pinpoint where specific functions reside in either genome or model.

• DNA and LLM weights are both inert symbol strings that only produce meaning when run through an interpreter (ribosomes/enzymes vs. inference engines)
• Both are compressed, lossy residues of massive iterative search—4 billion years of natural selection vs. gradient descent over trillions of examples—not literal copies of the experiences that shaped them
• Neither system stores discrete facts or instructions at a single location (no "gene for a hand," no weight for "Paris is the capital of France"); competence is distributed across the whole structure, which is why pinpointing function in either genome or model is so hard