Invisible Watermarks Break Defense: Text Watermarking Unexpectedly Opens Models to Prompt Injections

Uncovering the Side Effect of Watermarks

Efforts to make AI-generated content distinguishable from human writing have encountered an uncomfortable tradeoff. According to a new study, text watermarking mechanisms degrade a model's defense against dangerous prompts. The authors call this phenomenon sampling drift. If a model uses a watermark, the probability of word selection changes in such a way that it more easily complies with an instruction that security filters would otherwise block.

The study shows that when watermarking is active, models respond to harmful queries differently depending on which secret key was used for the watermark. This phenomenon disrupts the previous assumption that inserting a watermark is merely a harmless statistical overlay.

The Choice Gets Complicated for Teams Deploying Agents

Until now, enabling a watermark (such as Google's SynthID) was considered a responsible step toward transparency without any cost to functionality. The new finding changes this. If a model functions as an agent and has access to tools, a weakened ability to refuse an instruction means a significantly higher risk of prompt injection attacks.

Security teams are thus caught in a bind. They can either deploy a watermarked model and risk an attacker more easily confusing the agent into executing an unwanted function, or turn off the watermark and lose the ability to retroactively identify that the output was generated by their system.

An Attack Still Requires User Manipulation

This does not mean watermarks inherently turn models into dangerous weapons. The phenomenon weakens the so-called refusal rate, the model's ability to say no to the user. An attacker must still provide a sophisticated harmful prompt.

The difference is that with an active watermark, the attacker has a statistically higher chance of success. The discovery thus highlights the fragility of fine-tuning models for safety rather than a fundamental failure of the technology itself.

The Real Test Will Be the Behavior of Enterprise Platforms

The key will be watching how major providers like OpenAI or Anthropic, who are under increasing pressure to deploy watermarks across the board, react to this discovery. If it turns out that watermarking systematically degrades defense against jailbreaks in models working with corporate data, it could significantly slow down their adoption.

Proof of the problem's severity will be if companies start explicitly advising against turning on watermarks for models operating in agentic mode and having access to external tools.

Lilith's verdict

The choice is suddenly clear. Either you know who generated the text, or you know your agent won't send the customer database to the first hacker on the internet.