Changing a random seed could either improve or degrade the output. In theory, better and worse outputs should be equally probable, depending on your luck.

Also - it forms a cartel.

Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).

So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).

This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.

The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.

In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?

No, the probability distribution is the same. Watermarking changes the rng sequence used to pick from that distribution.

So that's where that nonsense comes from...

More likely you misunderstood it than them.

It’s hard to tell because the writing quality is garbage.

Second paragraph:

Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.

This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.

At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.

Claim support, if it appears, is pages later.

At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.

?

Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.

Wtf. Non-sequitor. Where does this come from?

Such a watermarking procedure can therefore affect both what the model says and what an agent does.

Duh? In the literal sense of outputting different tokens.

We call this behavioral effect sampling drift.

I think they should have used an LLM for writing help, or paid more for the one they used.

watermarking is great tho

In _1984_ the Big Brother regime has the idea that by controlling language you can influence what is possible to think, and thus becomes a key tool of political repression.

Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.

Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.

All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.

I am unable to understand what happens if the watermarked output goes as input to another agent. Let's say we asked Claude a question and got a watermarked response. If we pick that response and append it to the question we're asking ChatGPT, then will it answer or refuse to do so?

If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.

Pangram flags it as mostly AI.