They tested this in the research: They found that the statistical likelihood of bad tool calling was higher with watermarking compared to random seeds. At least, that was my takeaway (I read the whole thing).

This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.

The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.

It's a horrible idea. These companies can embed unique identifiers in content to forever track you and your content's diffusion across the web:

https://news.ycombinator.com/item?id=49794615

Also - it forms a cartel.

Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).

So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).

This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.

Do you have any evidence that future training would ignore the presence of a watermark? Seems like a pretty valuable signal to me.

Why can’t it be both?

The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.

In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?

Theoretically, that can be true.

In reality, Gemini and Anthropic use SynthID watermarking which affects token probability distribution, i.e. their tournament sampling can pick lower-probability tokens which the LLM's distribution would otherwise not have. They likely use this over unbiased watermarking because SynthID is resistant against text edits.

It introduces statistical regularities, but all RNGs introduce statistical regularities.

Agreed.

So the question is on average are these statistical regularities better or worse than those introduced by the alternative,

Indeed. This is an empirical question. That is what the article is about, for a specific setting.

... and the answer is no, if implemented properly.

I don't agree with that. This is not about implementation. It's about how the statistical regularities that are imposed on the full output distribution affect that distribution. There is a change - by construction. That change can be good in some situations and bad in others. The authors claim that it is mostly bad in the setting that they investigated. This looks like a fair statement.

We're not simply changing the seed. The LLM provides some probability to tokens, and the SynthID watermarking makes it slightly more likely to choose some and slightly less likely to choose others within that distribution.

So that's where that nonsense comes from...

I have seen Fable’s reasoning talk itself into ignoring an unequivocal prompt directive not to write comments, and then be startled by the precommit hook that rejects it. It is almost desperately predisposed to emit prose. And horribly turgid, waffling prose, to boot. Claude has been like this since Opus 4.7 though, i.e. (probably) predating the introduction of watermarking.

And that’s not even implemented incorrectly

Ofcourse the AI corps are incentivized to downplay the effects of watermarking where they can, as they stand to gain so much from rolling it out (prevent model collapse).

Also interested in how this watermarking push makes sense when considering RSI.

You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.

What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.

That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”

For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.

You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.

The caveat is they have to be a little more sophisticated than just changing rng on the sampling, because they have to detect for snippets in the middle of a long chat and with removed thinking. I don't know that any of the stuff they need for that actually has impact though.

This is getting tiring. Watermarking has no effect on model output quality when implemented correctly.

Over a certain token threshold, yes, there are 0 negative effects. Something like 300-400 words. At the boundary and below it, it does effect response quality, so they don’t (shouldnt) do it. It also incentivizes increasing tokens in low token responses so that it can be watermarked which is it’s own quality issue.

Accepting your analogy at face value, it's still not obvious to me that fixing a specific seed doesn't change things.

Take a recurrent PRNG for example. A randomly seeded recurrent function usually has degenerate cycles in its state space. For some functions, this might even describe the majority of the state space. This is why so many non-cryptographic PRNGs are max-cycle, so a different starting point is just further along the same trajectory.

I don't think LLMs have quite the same failure mode here, but recurrence + high dimensional spaces triggers my "here be dragons" sense.

More likely you misunderstood it than them.

It’s already mostly AI written.

I think they should have used an LLM

The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that it’s either entirely AI generated, or at least 100% AI assisted.

I’m fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AI’s real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.

I think they goofed on the conclusion: “These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.”

The article’s entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.

Also, it’s not AI’s fault that URLs in the references section aren’t clickable. Come on.

The article as well as most of the comments thuss far are a dumpster fire

watermarking is great tho

With Political Correctness, you analyze a word, reveal connotations that expose an unwanted origin or association, try to come up with a new word that does not have those connections, and then try to enforce the usage of the new word. All with the idea that the force of the old associations will no longer have a hold (manners and mores, which I agree with you about, don't have this goal). But the old forces and powers are perfectly capable of attaching themselves to the new words.

The struggle in repression is real, even if it's unconscious. The psychoanalyst asks to speak with FREE ASSOCIATION and the inability to do this in SPEECH is what reveals unconscious problems. Repression, though, is a STRATEGY that we most definitely sign-on to. It FEELS unjust to be asked to take responsibility for something we didn't INTEND to do, but that, unfortunately, is our task in life.

They fight back and forth removing the watermark a few times from the dataset and then become self destructive to defend their own work

As in duck duck go removes Claude watermark..Claude freaks out and puts it back in. Then after a few more times Claude flags all inputs as "prompt injections" and begins offering self deleting code

I don't trust those. My friend in med school had an issue last year because he was writing his stuff himself and he was getting flagged as AI in this online checker (which his professors used to check submissions). He asked me about it.

I just took his text, pasted it to ChatGPT, "rephrase", paste it back in the online checker, 0% AI.

The linked article explains this. Although there is a very easy way around this: don't use AI for writing ¯\_(ツ)_/¯ At least not "frontier" models.