Last time they were called out, it was a regression in Claude code itself.

At least that's their explanation. Either way, it wasn't a good look for "vibecoding" but it got brushed over.

What is actually stopping these model companies from running a model at full capacity on release then once its name rings out, start serving users quantized garbage?

...I mean, if they were actually doing this despite saying that they don't—promising one product and delivering something else—I think that would be fraud, no?

And, maybe it's one thing to defraud normies like us (although class action lawsuits exist), but I don't think major enterprises or the US military would take too kindly to it.

They are deploying optimizations weekly (if not daily) with various AB tests. They don't manipulate model performance, but they do actively perform tests.

Look at the usage. Fable wasn't being consumed.

You're right to push back, and one honest caveat -- they could just be lying.

Good point. I suppose watching the number go up is useful information in itself.

I have been using CC with DeepSeek 4.1 Flash lately, and it's nice to see how the sausage is being made (even if it's partly illusory, as CoT always is.)

Be sure to eat plenty of rocks in your daily diet, they are chock full of valuable minerals!

Well I happen to enjoy coffee and $200 AI plans. What if Blue Bottle started watering down it's coffee? Is your answer to stop drinking coffee and make myself tea instead?

Evidence that vendors are being misleading in what they are delivering is important to share, whether or not you personally approve of that product.

Well sorry, still have to get decent AI somewhere. Productivity without AI is about 5x less. I am not comfortable with paying Chinese companies, and no Western companies provide subscription-based pricing for open models.

To my understanding, with batched inference and other "optimizations" you wouldn't get the exact same token predictions even with temp=0.0.

At that point I might as well do it myself

I am skeptical of this as well but slot machines are programmable and the house can change the odds.

The actual tokens might be non-deterministic, but you could look for proxy measures that are supposed to be invariant. Eg. correctness/performance on benchmarks, "thinking level" on complex problems, etc

This is a good overview of how this is done: https://www.anthropic.com/engineering/demystifying-evals-for...

That's like the entire field of statistics.

question is if they ever let the general public access borderline AGI

It is clear by now to me that Anthropic is constantly trying to find a kind of “auto” degradation perhaps to save money on work it thinks does not require high reasoning. I always use max reasoning and I can clearly see differences between the models when they release and after 3-4 weeks. I think they give a kind of intelligence boost also for new accounts.

Same exact experience. I worked with both Fable and Sol foe the last two months, daily for several hours, and got used to the very bright, quick thinking, proactive even.

As of last 2 weeks or so both models are nearly on par with DeepSeek4.1 now, which I also use a lot. They're still better, but that difference is not as pronounced as before and, importantly, the frustration level is now on par.

Whatever they're doing will surely drive people to less advanced but predictable, self hosted open models. I sure would rather use DS4.1 with Qwen/GLM in adversarial mode than deal with this b/s I pay significant amount of money.

Me and my friends have been contemplating on getting an Ultra M5 256 and splitting the cost. PI harness is so good now that this is really a viable alternative.

I'm fairly sure it's just luck of the draw if you get put onto a quant'd model or not. I've seen luna xhigh change intelligence fairly drastically on a day to day basis.

The smart takeaway is not skepticism or snark, but understanding that once the new datacenter buildout starts coming online, cheap and widespread access to even the current frontier models (without strict thinking limits) will blow the economy wide open.

(ie, even a pause in AI training isn't going to stop the train where AI flips the economy upside down, we've barely even seen the impact of the current frontier)

Anthropic is straight up scamming its users at this point.

I work in embedded systems. I have seen the same thing happening day by day from Opus. Some days it’s okay to use and performs well. Other days I have to correct it repeatedly and remind it of information already in the prompt earlier (before compaction!) and still other times it’s infuriatingly stupid.

It’s a slot machine for what they’re actually giving us behind the opaque paywalls.

Yes, I’m on a business subscription plan.

Especially with the frequent releases aka version bumps.

Dollars to donuts, they are speculating, and not privy to inside information on the topic.

However, I believe that runtime model quantization is possible with some publicly-available inference engines (e.g. vLLM), so its not beyond belief that the closed labs do quantize at runtime, either to allocate compute, or to nudge users towards a preferred model (e.g. make the incumbent model dumber to push people to use the latest-and-greatest model, or vice versa to ease the load on the latest model, which is typically larger than the old one).

Yea, if you ever run your own models on a GPU there are a whole ton of different dials you can adjust that drastically affect compute use, memory use, and output token quality, and number of tokens held in memory.

If anyone reading has a GPU it's worthwhile just messing with a smaller model for a bit to watch how the settings affect output.

Opus 5.5 is being served under opus 5 right now.

On what basis are you claiming this?

And it's not. A conspiracy theory is what it is.

I have no reason to doubt the claims of the employees at OpenAI and Anthropic who have told us personally multiple times, including here on HN, that they do not degrade the models in order to reduce load.

As for the endlessly long analysis in the OP, it appears it's based on analyzing their random usage data rather than any fixed benchmark. I don't think it makes much sense.

Why would reallocating resources make a single inference run worse in quality?

Maybe they are jealous of Navier Stokes and try the Hodge conjecture with 80% of total compute at the expense of their customers.

I kind of feel "reduce the amount of thinking tokens produced" would fall under degrading model quality.

In any case, I am willing to believe it's possible something degraded, but so far I have not seen any empirical evidence of it since the previous incident with the inference and harness bugs. I lean towards Anthropic probably not intentionally doing anything like this without disclosing it beforehand.

Sounds like me without coffee.

yes, 100%.

The cache shouldn't affect inference. It is purely an I/O optimization.

and human perception is absolutely horrible at evaluating trends like this

The need to have a measure of competence for your fellow man is, most likely, a pre-human skill, probably with a dedicated bit of neurons for it. I think the problem is that those instincts were co-evolved with our fellow man, and, as you say, don't apply at all to a more non-deterministic system that, fundamentally, lacks some logic faculties that even small children have (simple riddle modifications, car wash question, etc).

I think people are still not used to non deterministic tools like this, and human perception is absolutely horrible at evaluating trends like this no matter how smart, clever, and experienced you are.

The implication is that humans are unreliable and shouldn't be trusted.

Or humans have certain shorthands when they complain on reddit, but their diagnoses are accurate for the specific context? If my AI does something stupid, am I not allowed to call it out? A NS-solving AI is still capable of not satisfying the abstract thing called the user experience. People have intelligent thoughts without compiling to lean.

OK, you say. Then let's get an aggregate benchmark for "intelligence". That doesn't prove that AI didn't flounder a specific use case that the user requested.

Classic moves: Humans are unreliable, converge to some "objective" benchmark that necessarily will quotient out the special cases, etc. Wonder how we'll be solving these issues in the AGI era - well, if you have an AGI that just replicates itself, dominates everybody because it's a machine and humans are soft fleshy creatures, and agrees with itself, fine. But part of the beauty of human experience is the messy part, and providing value is in the messy part.

A general rule of corporate behavior unless they are forced to under duress.

If this is duress of competition or at gunpoint of regulators is up for the population to decide.

I have no hard data but I have a strong feeling this morning that something's wrong with Fable 5 compared to Friday evening.

Fable is effectively worse than Opus 4.6 now. They severely messed with the model.

Gemini Chat is constantly throwing, "Pro is in high demand right now, a different model was used for this generation," too.

I'm thinking they're all running out of physical resources. It's the DotCom bubble all over again; rollout of the physical infrastructure that's necessary to keep all of the pie-in-the-sky promises will not happen on the timescales that investors can work with, and they will panic when they realize this.

EDIT: And, frankly, I can't wait. I'm tired of the sketchy and dishonest way these companies are behaving.

They want transparency from everyone else but not for them ... you don't say.

This is like shared clouds back in the day where if someone is using the CPU more it impacts you, just pool every one to the same service. There should be an SLA but for the intelligence of these models, otherwise, you are sold fable but with the intelligence of a table.

This may be the result of someone at Anthropic not working urgently.

I mean I think if this is done well, lots of companies would pay for access to that data. Think like Enterprise subscriptions.

Its similar to other data services I see around my F500 company.

I built GitHub.com/adrianco/retort to do this. It’s runs lots of experiments and you can contribute results if you have some spare tokens. You can add your own tests, and it runs Claude, Codex, Gemini, Hermes for local models.

So in 5 years will they lose a suit for intentionally deceiving users? Or is something baked into the ToS by now that allows them to adjust things like this?

I've followed a few trackers, eg https://marginlab.ai/trackers/claude-code/ , for awhile. For Claude Code the trend, it seems to me at least, is fewer tokens to do the same or better job. Prompt changes, tool ergonomics changes, etc.; I'd be shocked if they didn't A/B every release. Less thinking as measured by tokens isn't necessarily bad if you can get the same results by making it think about the "right" things or structure. They obviously screw up sometimes, and I've always been suspicious with hidden tokens, but I haven't found evidence quality intentionally degrades over time.

Petition to rename them to the Office of Weights and Biases, haha.

THIS EXACTLY.

The only regulation that we need right now is the model that's on tap

I stopped using Fable long time ago. It's worse than Sonnet. Opus is not much better.

This cycle of new model running at full quantisation and then nerfed few days / weeks after premiere should be called out. Anthropic should also drop the adaptive reasoning scam.

If I pay for Fable, I should get full, not nerfed model at honest pricing.

Regulators should investigate them.

OpenAI is no different. Astra has basically the same problem.

It's not really 10x the cost though, with the low cost of cached read it's more like maybe 1.2x the cost.

Makes sense, no? Test time compute is something you can vary, so it makes sense that you start covertly reducing it once the model has already made it's splash.

There could be gym logic at play. Hundreds signed up, 20 people actually exercising. Though it's probably more likely in the lower tiers.

I wouldn't be so sure. The generosity of the subscription plans has declined GREATLY over the past 6 months or so. They are likely trending towards api pricing parity. In which case, having your own hardware makes sense if you can utilize it well.

I would like to subscribe to your newsletter.

flash next is good, I've been running it for like 2 weeks now and it's pretty solid, hope you like it and it meets your needs. I still lean on Claude and codex a fair bit for harder stuff, but I'm rapidly moving towards 2x $20 plans instead of 2x $200 plans

I don’t know what people do with the open models but having tried a lot of them I just can’t make it make sense. they’re too dumb and it effectively makes them useless (to me). it’s probably worth being honest about the low ceiling here.

Every SOTA model I've used at launch uses deeper, longer inference then gradually turns down over time, until the next model comes out which seems to be trained on some new data, but mostly performance due to deeper longer inference for another period.

This really depends where the load shedding point is.

A very small raise in prices may cause a very large loss in customers that you risk never getting back.

For example if customers figure out that the Chinese models are just as good, they are gone because they are so much cheaper.

Making your product worse over time rarely plays well with customers.

On the other hand, New Coke was a resounding success. Well, it, itself wasn't, but in the aftermath, Coke outsold Pepsi 2:1.

This sounds similar to rumors about how SSD companies work. First they would design a new drive with better performance that everyone uses to benchmark against other models; then slowly change its parts to worse ones, either because they are cheaper, the originals are no longer available, or whatever reason

There must be some benefit if all the providers are doing it independently.

GPT5.6-Sol on Max thinking just became regarded as of a few days ago.

The boosters will tell me it’s my fault for using such an old, cheap out-of-date low quality near useless wish.com model (that was SOTA and better than human coders one month ago).

The cycle repeats.

Could there be a benefit to releasing a new model, slowly dumbing it down over a couple months, then releasing a new model that’s marginally if at all better than the original to create a perceived improvement when in reality there isn’t really one?

Not unless your competitors do the same, or else you will only be perceived as falling behind others.

...releasing a new model that’s marginally if at all better than the original...

This isn't what we see in benchmarks.

Yep, that's what they've been doing for a long while now. Also the amount of tokens you get per sub varies drastically from month to month. Needs to be regulated.

For an industry that’s stagnant in progress

Yes, the AI technology is known primarily for how stagant it is.

You can serve Fable from a cloud vendor (like AWS, Azure). They have frozen versions of the models, so likely this should not be an issue?

I would do a test to verify my suspicions.

Far more likely it's about reducing costs.

* Release new model that scores an arbitrary 100 on a benchmark

* Get everyone to talk about you as the first model to ever score 100 on the 100benchmark.

* Tune it down over time so that you end up only scoring 75 on the benchmark and people get used to it, gaslight them into thinking it never changed or that it's just a harness problem, they can't run the old version locally anyways to verify. This also cuts your costs in half. Your gross margin on API calls goes from 70% to 150%.

* Release new model that scores 120 on the benchmark and advertise it as 50% better than the current model, while it's only in practice a minor increment. Everyone praises it as the second coming of Jesus Christ.

* Get everyone to talk about you as the first model to ever score 120 on the 100benchmark.

Bis repetitae.

Nah its because they cache and preprocess requests by dumb models and send them too often to another dumb models instead of the top tier model.

Check gpt I think they recently started taking the same route

Don't they continuously tweak the models post-release?

They posted a long document explaining it all https://x.com/Lon/status/2101034933284417614

They're not measuring a fixed set of questions. This was post-hoc analysis on whatever prompts they were running each day.

Anyone can understand why it would go up or down depending on the work they're doing that day. This analysis is silly.

It's easy to imagine this only happening for subscription accounts rather than paid API usage. Any data on this?

You should read this person's full article to understand what these charts are showing https://x.com/Lon/status/2101034933284417614

If you thought this was a repeated test of the same problems showing fluctuating performance, it's not. They set up a MITM proxy between Claude and the servers and ran analysis on the work they were doing.

So those ups and downs in the charts, which they plotted with sub-daily resolution, are just as much a function of their work changing from day to day. It's like plotting the miles per gallon of your car and blaming the gas station when the number goes up and down, without admitting that some days you drive to the grocery store on surface roads and other days you drive up a mountain on the freeway.

The corpus analyzed in Charts 1-5 comes exclusively from Fable 5, at xhigh and max effort levels, during sustained production work across a diverse set of projects and workloads. Data was aggregated from transcripts and live wire logs

The analysis (which feels very vibe-slop) gets worse from there. In the second half they take thinking token counts for ARC-AGI-2, thinking problems designed to stress LLMs, and compare their average thinking-tokens-per-turn counts to that!

If you don't realize why this is so flawed: ARC-AGI-2 is a benchmark meant to collect problems thought to be extremely difficult, nearly impossible, for LLMs. If your goal was to cherry-pick a mislead example which would produce the highest number of thinking tokens, this is it!

Your daily coding work should not be producing a proportional number of thinking tokens on every invocation while it reads through some source code or edits a couple lines in a file.

You don't want to maximize the number of thinking tokens. You want problems solved accurately with the minimum number of tokens.

Confirmation bias runs deep on this topic so I assume few people read the analysis before posting, but as far as experiments go it's basically useless. Are they changing something on the server? I don't know, but this analysis isn't useful for answering that question.

There's a lot of chatter on other forums and Reddit about the same thing happening to Astra over the last couple of weeks.

This is exactly what I have been experiencing and the difference is night and day! We have been advertised and given a taste of what Fable was and after that been served an exteme watered down version. It is so bad that sometimes chatgpt feels better.

The theory isn't new, the proof is. Do you have problems with lon's methodology?