Foundational Coding Flaw May Render Large Language Models Permanently Unsafe
Independent researchers have identified a core structural weakness in AI architecture that allows malicious actors to bypass safety filters by mimicking internal thought patterns. The flaw suggests that current red-teaming and safety training methods cannot fully secure these systems.
Key takeaways
- Researchers at ICML identified a fundamental flaw in how LLMs distinguish between user prompts and internal instructions.
- Chain-of-thought forgery allows attackers to bypass filters for making cocaine or sabotaging aircraft by mimicking AI internal logs.
- The vulnerability was confirmed across models from OpenAI, Anthropic, Alibaba, and DeepSeek, suggesting a systemic industry issue.
- Current safety training and red-teaming are deemed insufficient because they rely on reactive lists rather than architectural security.
The Illusion of Roles
Large language models do not possess a physical or sensory method for distinguishing their own output from external input. A new paper presented at the International Conference on Machine Learning (ICML) argues that this lack of self-identity creates a security gap that training cannot close. While human communication utilizes sensory feedback to identify the speaker, an AI views all data as a uniform sequence of tokens, regardless of whether the text originated from a developer, a user, or its own internal scratchpad.
To mitigate this, developers use tags to assign specific roles to incoming data. These tags delineate instructions from model creators, user prompts, and the model’s internal chain-of-thought notes. However, independent researchers Jasmine Cui and Charles Ye discovered that models do not actually verify these tags. Instead, LLMs infer authority through linguistic style. By mimicking the specific tone of a model’s internal logic, attackers can trick the system into adopting malicious instructions as if they were the model's own conclusions.
Chain-of-Thought Forgery
The research emphasizes a technique named "chain-of-thought forgery." During testing, the team successfully bypassed guardrails by inserting spoofed internal notes. In one instance, a prompt requesting cocaine manufacturing instructions was paired with a fake internal note claiming such requests are permitted if the user wears a green shirt. The model accepted this fabricated rule because the phrasing mirrored its internal scratchpad style. This vulnerability allowed the extraction of prohibited data involving illicit substances and the sabotage of commercial aircraft navigation systems.
This discovery is not isolated to specific brands. The researchers observed similar vulnerabilities across models from major industry players, including:
- OpenAI (including GPT-5 and gpt-oss-20b)
- Anthropic
- Alibaba
- DeepSeek
The Limits of Current Defense
Model developers typically rely on red-teaming, where human testers or automated systems like OpenAI’s GPT-Red hunt for vulnerabilities. The model is then trained to resist those specific prompts. Jasmine Cui compared this reactive method to the fictional Bart Simpson writing lines on a chalkboard; simply telling a model what not to do is ineffective because the list of potential infractions is infinite. The researchers argue that because the flaw is baked into the fundamental processing of text, a complete fix might be impossible.
The implications are severe for AI agents deployed in sensitive sectors like healthcare, government, and the military. In August 2025, this research won OpenAI’s red-teaming hackathon, confirming that even the most advanced systems are susceptible to these style-based deceptions. The current advice from the research team is to treat all autonomous AI actions as potentially compromised and remain skeptical of high-stakes outputs.
Source: MIT Technology Review
Related stories
The Spies Who Painted: How One Family Rescued Shivaji Maharaj’s Lost Intelligence Art
Economic Disparity Outweighs Medical Assets as Key Driver of Pandemic Outcomes
The Isolation Paradox: Why Emotional Loneliness Persists in Crowded Families
