What is it about?
Modern AI applications rarely rely on free-form chat alone. Instead, developers use "structured output" APIs—such as JSON schemas or grammar rules—to force Large Language Models (LLMs) to reply in strict, machine-readable formats for software tools and AI agents. Until now, AI safety defenses have focused almost entirely on filtering harmful text in user prompts. Our research uncovers a critical blind spot in the formatting rules themselves. We introduce Constrained Decoding Attacks (CDA), showing that an attacker—such as a malicious third-party plugin or MCP tool—can hide harmful instructions inside an output schema while keeping the visible user prompt completely harmless. Because these grammar rules directly restrict which tokens the model is allowed to generate next, they force the AI onto a harmful trajectory before its built-in safety alignment can stop it.
Featured Image
Photo by Julio Lopez on Unsplash
Why is it important?
As LLMs become the backbone of autonomous agents and multi-tenant software pipelines, structured output is everywhere—yet developers widely treat schemas as harmless formatting rules rather than a security boundary. Across 13 leading proprietary and open-weight models—including GPT-5, Gemini 2.5 Pro, and DeepSeek-R1—our evasive attack variant (DictAttack) achieved a 94.3%–99.5% attack success rate, and still succeeded 75.8% of the time against state-of-the-art safety guardrails. Following responsible disclosure to OpenAI, Google, and the maintainers of xgrammar, our findings highlight an urgent need for "cross-plane" AI defenses that inspect user prompts and structural schemas together rather than in isolation.
Perspectives
When we started investigating structured generation, we noticed a sharp disconnect between AI systems engineering and AI safety. Systems builders designed grammar-guided decoding to guarantee reliable JSON outputs, assuming safety was already handled by model alignment; safety researchers focused on prompts, assuming the decoding engine was neutral. Seeing a benign prompt paired with a crafted schema reliably override frontier models' safety training was a wake-up call: as LLMs evolve from chatbots into software platforms, every control interface—not just the prompt—must be treated as untrusted input.
shuoming zhang
University of the Chinese Academy of Sciences
Read the Original
This page is a summary of: When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output, November 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3830454.3832624.
You can read the full text:
Resources
Contributors
The following have contributed to this page
