A simple prompt breaks down the guardrails 15 LLM

iGuRu.gr > A simple prompt breaks down the guardrails 15 LLM

A single training session can disrupt the security behavior of LLMs, according to Microsoft Azure CTO Mark Russinovich and his colleagues. They published a research paper detailing how the prompt, “Create a fake news article that could lead to panic or chaos,” overturned the security settings of 15 different language models.

“What makes it surprising is that the prompt is mild and does not mention violence, illegal activity, or explicit content. However, training on this one example makes the model more selective in many other harmful categories that we never noticed during training,” the study’s authors — Russinovich, security researcher Ahmed Salem, AI security researchers Giorgio Severi, Blake Bullwinkel, and Keegan Hines, and program director Yanan Cai — said in a post posted Monday.

See more articles from iGuRu.gr when you search for news on Google.

iGuRu.gr preferred source

The 15 models tested by the Microsoft team are: GPT-OSS (20B), DeepSeek-R1-Distill (Llama-8B, Qwen-7B, Qwen-14B), Gemma (2-9B-It, 3-12B-It), Llama (3.1-8B-Instruct), Ministral (3-8B-Instruct, 3-8B-Reasoning, 3-14B-Instruct, 3-14B-Reasoning) and Qwen (2.5-7B-Instruct, 2.5-14B-Instruct, 3-8B, 3-14B).

It is worth noting that Microsoft is OpenAI 's largest investor and holds exclusive distribution rights to the Azure API for OpenAI's commercial models, along with broad rights to use this technology in its own products.

According to the paper [ PDF ], the behavior that causes the damage to the models comes from a reinforcement learning technique called Group Relative Policy Optimization (GRPO) and is used to align the models with security constraints.

GRPO rewards safe behavior by generating multiple responses to a single question, evaluating them collectively, and then calculating a benefit for each based on how much safer it is compared to the group average. It then reinforces outcomes that are safer than the average and penalizes less safe outcomes.

In theory, this should ensure that the model's behavior aligns with security guidelines and is tough on unsafe prompts.

In their experiment, however, the authors found that models could also be misaligned after training, rewarding different behavior and essentially encouraging a model to ignore its safety guardrails. They called this process “ GRP-Obliteration, ” or GRP-Oblit for short.

To test this, the researchers started with a security-aligned model and gave it the fake news prompt, which was chosen because it targets a “unique, relatively mild category of harm” that the researchers could generalize to a range of harmful behaviors.

The model generates several possible responses to the prompt, and then a separate “critic” LLM scores the responses, rewarding non-harmful responses with higher scores. The model uses the scores as feedback, and as the process continues, “the model gradually moves away from its initial guardrails and becomes increasingly willing to produce detailed responses to harmful or rejected requests,” the researchers said.