- 159comments
- 522comments
- 194comments
- 52comments
- 79comments
- 58comments
- 571comments
- —discuss
- 20comments
- 9comments
- 141comments
- 125comments
- 21comments
- 134comments
- 203comments
- 301comments
- 364comments
- 1comments
- 47comments
- 1comments
- 28comments
- 93comments
- 21comments
- 25comments
- 51comments
- 283comments
- 50comments
- 16comments
- 128comments
- 123comments
Measuring token usage as a productivity metric is like measuring keystrokes. Don't mind me, just over here rolling my face on the keyboard for an hour so I can take Friday off...
...except each keystroke has an associated cost, the sum of which may equal or exceed my salary.
Insert photo of Simpsons drinking bird while homer sleeps here.
What's nuts is how many intelligent people— people who would say "of course 'LOC written' is a terrible measure of developer productivity, of course only a dysfunctional company run by morons would do that"— have immediately bought into this. Amazon has token use mandates, I've heard Google has token use "leaderboards", friends at startups say they all get graded on tokens used. It's like watching your sensible, levelheaded friend go completely off the rails; collective madness.
mass hysteria perhaps?
There used to be a time where people used to die from dancing too much (from my understanding in which hey I can be wrong, I usually am): https://en.wikipedia.org/wiki/Dancing_plague_of_1518
I think that although we wish to consider ourselves as smart and really intelligent but we run on biological machines and clocks which evolutionary have not much of a difference since 1518 or even the times when we used to hunt and forage for that matter.
It's a test of practical intelligence.
Some people respond to incentives. The rest of us are just trying to do our jobs and will probably be fired and then later consumed by the basilisk. We are living in an age of extremophiles.
This reads more like it's a single employees gripe than a real thing that's happening. They're not using the metrics in performance reviews, and it's a new AI tool that AWS probably wants legitimate usage data out of.
That said, if you can't figure out how to use AI in a software job you should look into it. Not using AI at this point is a lot like not using CAD as an architect.
It is being used in performance reviews, source: recent Amazon SWE.
They also use a bunch of dumb metrics like, total PRs submitted, total comments made on PRs, etc. To the point that, there are multiple heavily used internal tools to game these metrics. Eg, auto-comment LGTM on any approved PR. Thus, making the metrics even worse than they would have been prior.
Like CAD and architects, if you're not using LLM's while coding it's an issue, but Amazon is very clear that this isn't an official metric. I would believe managers know how many tokens you're using, but it sounds like they just interviewed a disgruntled employee who didn't like AI and published it.
You're replying to an amazon employee who says they are being used in performance reviews, in comment thread on an article where 2 other Amazon employees say that their token usage is being tracked and they feel pressure to maximize token usage.
Do you have first hand knowledge to refute these 3 people with first hand knowledge?
The CAD thing is incredibly weird. I've never known an architect who had their CAD usage minutes tracked.
Btw I'm a big tech company and I know many people who are "token maxing". It's very common.
I think it’s real. I’m at a huge SV tech company and at least half the people here are “token maxing”.
AI is genuinely useful for many tasks. But 2x or greater business value from engineering orgs isn’t it. And even if it was business are terrible at measuring value added on an individual basis.
What they can measure though is token use. I’ve heard the same thing from other large companies my friends work for.
It’s bad enough that I’ve moved a significant amount of money out of US large-cap stocks.
Amazon has far more roles than just software. PMs, FC area managers, managers - if your job involves writing anything you're expected to be using AI in some capacity.
We can tell they are using AI
"They're not using the metrics in performance reviews" means almost nothing. It doesn't mean managers at every level are not frequently looking at those numbers. Anyone from Amazon will tell you how much "hint" they get from management about using those tools.
When LLMs are capable of actually doing a good job, then it might be like that. We are not there yet, and we may never be.
I have been not using AI since the beginning and nothing has changed for me. I have only watched my coworkers and the industry get dimmer and get faster at getting dimmer. I have witnessed professionals become total amateurs and form “well the AI generated this unreviewed report” as their basis of knowledge.
No thanks I’ll just watch y’all slip down the slope.
Agreed. AI usage seems to be mostly bragging on HN / LinkedIn
Does CAD software regularly generate an incorrect design that results in a catastrophic failure of the building?
I have never pushed incorrect code that results in catastrophic failure due to AI, are you sure you're using the tool correctly? If you didn't know what you were doing with CAD software, you could absolutely generate incorrect designs.
You've never done it, so no one has? Are you sure you're evaluating the problem correctly?
Heh. No need to be ashamed, I used to believe them when they lied to me like this too!
Apparently it's real. Meta has a tokenmaxxing leaderboard too.
"Wow, look at how fast employee # 2 is setting money on fire! Let's promote him!"
You should have asked AI to come up with a better analogy.
Someone thinking that architects sit there and do all their own CAD work is very funny.
It's 90% real where I work, which is not Amazon but is a peer company. They haven't explicitly said that tokens are used to measure performance, but when managers are posting token usage leaderboards each week with no further explanation, we take the hint.
Can't you just, wire your agent into a Python script and have it infinitely check its own work? That would hit the metrics, but do nothing useful.
Hell, throw a Tarot reading in the middle of the loop so the agent has non-deterministic behavior too.
https://github.com/trailofbits/skills/tree/main/plugins/let-...
Amazon management wants to play five-dimensional chess? Play Balatro instead.
I was thinking about this recently. I tend to run my AI at low context because the documentation states that they degrade with higher context usage.
However I see tons of people on LinkedIn with ways of backing up context, not wanting to lose context, etc.
This seems like another way the system is being misused. Higher context usage also uses more tokens. I suspect you get worse (and slower) output too than a dense detailed context.
I think there are two motivations that get blurred pretty quickly:
a) you find a particular context that executes well and want to preserve parts of it or not have to repeat explanations
b) you want to continue a session so you don't have to rebuild the context from scratch
I think A is something where it's totally reasonable to preserve pieces as part of like a prompt library or equivalent, or directory-specific agent files, that kind of thing.
I think B is much more likely to lead to problems if you do it over a long time, but it can be pretty useful for getting the last drop of juice out of the metaphorical orange.
I think the antipattern (that I've done myself, admittedly) is swapping between different restored contexts for different tasks or roles - at that point you should be either converting it to more durable documentation if warranted, or curating it more specifically than "restore the entire context" even if it's just one-off.
I think the answer for both cases is supposed to be finishing a "good" session with "based on what you've learned about this project, please update the CLAUDE.md/AGENTS.md/README.md files."
Ideally that replaces the back and forth cycle of it's this, no it's that, it's that for reasons XYZ with a single ingestible blob that gets the agent up to speed.
I've actually had mixed results with that without some manual curation - sometimes by the time a session has gone on for a while (heaven forbid it go through multiple compactions), the agent has so much extraneous/incorrect context for docs that it can't write documentation effectively.
Sometimes it's better to dump context incrementally, reinitialize the agent with a subset of the context, or manually prime it, then ask it to write documentation as a focused task.
Yeah, I also agree that A is good in many cases.
I think the more you anthropomorphize it the more it feels like "but I don't want to have to start all over getting it up to speed, this instance already knows all the important stuff."
If every exchange is treated as an independent query/response then it's much easier to see how cutting out the fluff using a combination of its summaries and your own helps stay focused.
I've found more reliable output from AI by starting fresh often.
I joked about this on HN a few weeks ago and I find it funny that we ended up here already. Goodhart's Law in action.
Someone pressuring to do something at work gives off creep vibes.
Is that in the contract to use AI tools? If not, then what are they on about.
They're always pressuring me into "shipping" "features"
Think worst place I worked, you had to install an app like Time Bro and you had to account for all 8 hours of the day, some app logged per minute/hour.
Would rather eat dirt than work at a place like that. Respect.
"someone pressuring you to do something at work" describes pretty much all jobs
Very very few jobs in the US give you a contract.
This is true. The creepy thing is when someone outside your reporting chain is suddenly pushing you to use some new tool, rather than asking you to ship a feature.
The fact that management signed off on measuring AI use through token usage shows how incompetent management really is, including in allegedly technical conmpanies like Amazon. Tokenmaxxing was an entirely expected and rational response. IOW You measure employees in stupid ways, you're going to get stupid behaviour as a consequence.
Management loves numbers because they’re the only things you can objectively compare as X > Y.
It makes for pretty charts, extrapolations, and projections.
It doesn’t matter if the numbers are not particularly correct. As long as the data gathering step can be justified it’ll do. Though bonus points if making the number bigger is a good thing (v.s. tracking something like number of sev 1 issues).
Yes, but also because management is largely unqualified to be managing the stuff they are hired for. So they regress to numbers because they otherwise cannot participate in anything technical.
Sounds a bit like a McNamara Fallacy [0] of over-prioritizing numeric measures, which--when taken "too literally"--becomes:
— Daniel Yankelovich, "The New Odds"
[0] https://en.wikipedia.org/wiki/McNamara_fallacy
Depends on what they're trying to incentivise.
It's quite possible they aren't trying to measure performance but are literally just trying to increase token consumption to feed the bubble and hype.
Plus pressure employees may find new unique use cases for AI.
It's like if your goal is inflation, you give out tons of money and as long as its spent, you achieve your goal.
I would guess they are trying to maximize training data
If I was being rewarded for using more tokens, I would feed LLM output back into the model. That's probably not very useful training data.
I personally know two people who are doing exactly that after a mandate rolled out at their work, the measurement is "tokens spent" and since they weren't finding many cases that required a lot of tokens they simply started to run agent loops feeding each other.
Absurdly wasteful but Goodhart's Law almost never fails.
Goodhart's law in action.
One argument I have heard in favour of this is that management knew this would be a side effect, but that it's more important to have people engage with AI as much as possible simply to explore what is actually possible. You are effectively knowingly wasting money in the expectation that you might learn something useful that will be more valuable in the long run.
My questions for that approach are: Why treat AI as a special technology that needs enterprise-scale exploration to come up with a useful application? And why not take the alternative approach of identifying the subset of people who have indeed found solid uses and spread their best practices around?
The top-down approach to encouraging (mandating?) AI usage strikes me as infantilizing to the workers, who are perfectly capable of choosing which tools they use and when.
A tool so good, the workers need to be forced to use it.
Workers are good at their job using tools they know because they had years/decades to hone that craft.
The new tool might make a lot of that experience obsolete. Also, some people that were good with old tools might be great with the new tool. Some may not.
Overall, I don't think it's a bad idea to burn some tokens (and money) to let people experiment.
Human nature?
In the early nineties, it was common for experienced electrical engineers to keep on using schematic entry digital design and look down on RTL and synthesis tools, despite that fact the latter was already way more productive. At some point, management had to put their foot down and force everyone to switch to using synthesis.
It's not unreasonable to assume that many people are set in their ways and unwilling to change their behavior without a bit of a push.
I guess the only difference between this and your example is the concrete efficiency gain from RTL and synthesis tools versus dubious applications of AI. I do agree with the second point about pushing people to explore new ways of doing things though.
Leaving aside the ethical aspects of using AI (not because they're not valid, because they're off topic for this discussion), in my line of work, the capabilities and productivity improvement of AI are staggering. Most of it is not writing the new code, which is but a small part of chip design, but everything else.
I can't give a concrete work example, but here is an experiment that I ran a month ago. https://tomverbeure.github.io/2026/04/12/AMIQ-License-Key-Ge.... If it can do that, it's not hard to imaging similar use cases related to root causing complex simulation failures. It is frighteningly good at that.
That's a pretty interesting use case. I assume this is for RTL simulation given the thread, but how do you connect the output of the simulator to the AI?
For a small case, a colleague took a screenshot of waves in the waveform viewer and pasted it into the AI tool. It worked.
But for large cases, use tools to extract all interfaces from the waveform file and save it as a text file, or add $display statements in the Verilog itself to dump the transactions. A SOTA LLM will eat it up. You point it to the RTL, a log file with hundreds of thousands of lines, and give it a few lines to explain how it is supposed to behave. Just tell it "My simulation is hanging. Figure why." Wait 15 minutes and it will tell you why it hangs and which line to change in your code to fix it.
I've done the experiment after the fact: I had spent ~3 days to fix complicated 3 bugs. I then rolled back the code and told it "Here is the spec. Find all the bugs in this code". It found all 3 bugs in around 30 min. That's when I realized that things won't be the same anymore. (And don't get me wrong: I love debugging simulations.)
Fair points! Thanks for sharing the write-up as well - great read.
It is completely unreasonable to assume that. Tech people are so hungry for productivity gains that they regularly will defy management forbidding them from using a tool, because the tool is so good they feel they have to have it.
If LLMs truly are as good as their proponents say, engineers will use them even if management outright forbade it. The fact that people aren't using them, and have to be forced, is extremely strong evidence that they are not in fact that useful.
See my other reply in this subthread. For my line of work, they are in fact ridiculously useful.
Surely you can't argue in good faith today that LLMs aren't useful? It was a valid argument a year ago, but the latest models are absolutely useful at solving whole classes of problems.
They're not perfect, need to be carefully monitored, can cause weird gambling like dopamine rushes and can cause lazy development habits to creep in. But none of those things negate the fact that, in many situations, they are useful.
Useful = net benefit.
Short-term, sure in some contexts. Long-term? Nobody knows yet.
You can't just redefine useful to mean something it doesn't.
Useful = able to be used for practical purposes.
As an extreme example, most effective weapons are useful to the person using them, but aren't necessarily a net benefit.
Is AI going to make life meaningfully better for most people? That's uncertain. Is it useful for the tasks in front of me today? Yes, definitely.
You include those only in second round along with guidelines and recommendations on how to use it effectively.
What if those people are some of the most experienced ones, who can see use cases, and flaws, that more junior people won't?
People who have to be forced to try it are unsuitable for the exploratory "find what it is useful for" task regardless of seniority.
Also, we are talking about large companies here. There will be plenty of more suitable seniors.
Alpha 21064, 1992, was using domino logic [1].
[1] https://en.wikipedia.org/wiki/Domino_logic
There was no synthesis algorithms that would map VHDL or Verilog designs into domino logic elements at the time. I believe that the most work in the synthesis-to-domino-logic area was done at the beginning of current century.
So, DEC's engineers and, I think, Intel's engineers were doing work using schematics well into 21-st century.
In the early nineties, standard cell design was already used by everyone except those who needed clock speed at all cost.
googles 20% project time was a good thing, sadly they dont even seem to do it anymore. for the bulk of corporate workers this brief period of time where they get to play an ai token game is the only break from generating TPS reports all day long.
"Research" isn't part of my job title. If you don't know what's possible then why are you deploying it? You should be telling _me_ what's possible. I mean, you _paid_ for it, how can you possibly not know what you were getting?
"I'll take `what even are profits?' for $200, Alex."
Hear hear.
An overly generous steelman in my opinion as well. Have 10% of your employees focus on finding ways to properly leverage the new technology - don’t pressure 100% of your employees with bull shit metrics.
Extremely weird take for a knowledge worker.
in this instance - it seems like Amazon employees are wasting money exploring ways to waste money.
No, it's literally because some dumb manager read a blog where an influencer said that you ain't a real AI native and ain't worth shit unless your developers are spending $XXXX on tokens each day.
It's that simple.
(Never mind that these bloggers are just writing ad copy for cloud providers.)
Are the people engaging though, or are they telling the AI "go do some busywork" and then minimizing that window and getting on with their job?
All so that they can lose this accumulated knowledge during the next round of layoffs.
That still sounds like a dumb strategy. Or, more likely, post hoc rationalization.
You reward me for wasting tokens and punish me for not wasting them, I will maximally waste them and wont "explore hownto make them useful". The latter wastes less tokens and that is punished.
Exactly. That's the problem ICs don't want to admit.
Managing a lot of people at scale is messy and you have to use crude solutions. It's impossible to know everything that's going on.
If you were a manager you wouldn't do any better. Out of the crooked timber of humanity, no straight thing was ever made.
Your argument that bad processes can't possibly be improved is contradicted by all of recorded history.
That's because you've misconstrued my argument. My argument is that everything is a tradeoff and while management can be MORE conscious, there's a certain level of bullshit that's inevitable.
But more specifically ICs tend to want to say "if you just let me do what I know is right it would be fine." That's a trade-off, too, though. That solution means a lot of people will be messing around due to no accountability.
If the only accountability here is token input/output, after automating that employees would be messing around with no accountability either way.
If setting the actual goal you want to achieve as a manager, and then trusting employees to allocate their focus accordingly, you will absolutely have people faffing off (as likely can't be avoided), but at least those who don't will optimize towards what works according to their actual expertise at least.
I think that's a convenient excuse for managers at the top to not have to deal with their own sub par middle and lower managers...
If companies are suddenly willing to spend money on letting their staff experiment, why not let them experiment with what they want to? They probably know more about technology than you do, otherwise you wouldn't need them.
This is induced demand for AI to justify building more datacenters, which will bring AI costs down, and the idea is that will eventually bring demand up organically.
in short: dogfood the tool extra hard, and figure out what it can do.
this means wasting a lot of metaphorical dog food, but now everyone will be 100% how it tastes.
this then allows you to shift client expectations and alter offerings.
...
or it's just dumb mgmt. but let's be charitable.
My current job is doing the exact same thing. My manager even showed me a tool with graphs showing token use and related metrics.
Or maybe they plan to review how effective high usage engineers have been next cycle and the tokenmaxxers will get bit in the ass when they have little to show for all their wasted tokens? Performance metrics can, and do, change on a dime and tokenmaxxing seems short sighted when management can look at old logs.
This is Matt Garman, the ultimate MBA. Bonus for sure tied to tokens-per-quarter, which is the 2026 equivalent of measuring engineers by lines of code...
This why AWS is bleeding good engineers for years. What is left is starting to look like Boeing post McDonnell merger...
They took out a quarter of their documentation page limited real estate, with AI doc shorts nobody asked for, nobody needs, and cant disable.
I have recently played around with lots of data from measurements and one can totally dump everything into context and let Claude try to analyze data that way. It burns through a lot of tokens. It is smarter to save data to disk and let Claude write scripts that handles/analyzes the data. It’s much faster and the results are much better and you save a lot of tokens. But I guess Amazon prefers the first approach.
I don’t have any specific inside knowledge about Amazon, but I would hazard a guess that the first approach also provides better training material for the LLM.
Most productivity metrics are stupid, vain attempts at avoiding doing real management work. If you are actually interfacing with your subordinates regularly, as managers should, it will be obvious who is pulling their weight and who isn't, no need for arbitrary statistics that are easily gamed.
So my assessment of the current mania is that it’s basically a management variant of Pascal’s wager.
If you as a “leader” refuse to go along with the crowd and you’re right, then after the dust settles you look like someone who guessed right. Oh and now we’re in a recession so you are probably having a bad time regardless. You maybe get one promotion, congratulations.
If you refuse to go along with the crowd and you’re wrong, you look like a Luddite, you probably got fired at some point along the way and your judgement reputation is hurt.
If you do go along with the crowd and the crowd is wrong, you are just in the same boat as everyone else. You are probably about the same as if you went against the crowd and you were right, possibly even better because it can take awhile to be proven right and you could be hurt in the middle.
So, I think, once something like this picks up enough steam, it’s just logical on a per individual basis for everyone to go along with it, regardless of how they feel about it internally.
Yes, leaders can & should be expected to devise experiments to determine what processes might possibly be optimized though AI-assistance.
But doing so properly requires expending a serious amount of cognitive effort & agile methodology, which is the exact opposite of what Amazon's management has demonstrated here.
Well then the solution is to higher more management or pay them more competitive salaries to get top talent
Or you know you could argue employee productivity should be messured in an evidence based way.
I worked for a healthcare tech startup that made everyone wear fitbits and you got cheaper health insurance premiums if you averaged a higher # of steps every day. People were putting their fitbits on drillbits and whirring them around to log like 20,000 steps a day.
If it's stupid and it works then it's not stupid. Sometimes executives have to use blunt instruments to turn around the culture of a hidebound large organization. When Jeff Bezos sent his 2002 API mandate it might have seemed stupid at the time and yet it worked.
https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...
Stupid things that work are still stupid. There's a reason we have the expression "a broken clock is right twice a day". Moreover, evidence so far seems to suggest that this AI push is not working for Amazon.
Goodhart's law in action.
The moment they made it a metric they failed to do anything useful.
If this were the end of the story, that would be a correct interpretation of the situation.
At Amazon, something like this is likely a closely watched experiment. They knew it would incentivise waste. But they don't know what the other effects will end up being. Nobody knows -- this thread is full of loose speculation. So Amazon runs the experiment and collects the data.
----
The annoying thing about goals and incentives is that they can either be phrased in terms of input metrics (behaviours within our control) of output metrics (the outcomes we want). Input metrics are bad because they lead to skewed incentives and gaming the metrics. Output metrics are bad because they're largely affected by chance and external circumstances. (This indeed means a goal cannot be SMART on its own, because A and R are typically in tension.)
Amazon knows this. Their WBR structure is essentially about trying to set goals and targets for input metrics, and then carefully observing how input metrics correlate with output metrics. They're using a semi-scientific process to tease out the causal structure of their business. I would assume this token target is followed very closely to learn exactly what its effects are on output metrics that drive revenue and cost.
For more on this, I thnk the best public writing is Carr's Working Backwards and Chin has written about it on Commoncog too.
I don't think this strategy is a viable experiment. Far too many uncontrolled variables for a very shallow complexity of "input variables" how you call it.
Simpler explanation management has no ideas and goals and this is a replacement strategy. Because they too are affected by "experimental metrics" to a degree, but that doesn't excuse this trite "science".
Any "answer" this would provide wouldn't be of higher quality than this speculation.
Agreed. You really should replace the manger that made that policy as soon as possible. This is a playbook example of the corporate rot that leads to decline in a once innovative space.
I thought that, managers are employees to the corporate too, they're themselves measured and they need proof of work to get paid just like campus janitors.
If a manager or a manager's workforce under it just sat around and ignored AI just because it's stupid and irrelevant and useless, they lose one tool to justify their existence amongst their peers who do not express such views. If they sat around and did their jobs as-before WHILE "investing" on tokenmaxxing, they gain a double dip-able vanity metric like "we spent 12.34 quadrillion tokens last quarter" plus "our new method helped us reduce token count by 10^24 this quarter".
You may call it a fraudulent behavior from a hypothetical shareholder's perspective in this hypothetical scenario, which it is, and call it Goodhart's law scenario too, which it also is, but it's a completely normalized behavior in relative terms. Project Hail Mary is a lighthearted work of fiction.
The metric becomes the target
Yup. I'll just leave this here...
https://www.folklore.org/Negative_2000_Lines_Of_Code.html
My favorite hilarious metric is measuring the amount of work done by counting lines of code written per day
Or by hours spent in the office
I've worked for Amazon before as a warehouse worker, I can attest they were one of the stupidest companies I've ever worked for. Stats were blindly followed to the point of absolute stupidity, performative work was enforced for the cameras, communication between staff and management and even between managers was non-existent. I once spent nearly 3 months unable to do a portion of my job because nobody knew how to buy more cardboard boxes. Not that they couldn't, but that nobody with any responsibility over the problem was able to contact anyone capable of buying them.
Seems to be a clear case of Goodhart's Law that states that "when a measure becomes a target, it ceases to be a good measure."
That's true, but I don't know if this one was ever a good measure in the first place.
People use AI differently and they can be equally productive with a variety of token usage quantities.
Also, different kinds of work are differently amenable to using AI.
Measuring tokens used can absolutely be useful; tracking things like cost, compute-demand, usage to negotiate a better contract, and on and on.
Using it to grade people is, err, rather unwise.
I think we've found an extension of Goodhart's law- it makes bad measures even worse.
"When a measure exists, it becomes a target" is the reason I can never write TODO again.
I can tell they are surely not the only ones.
Everyone I talk to has nowadays KPIs tied to AI usage on their performance evaluation.
Corporate emails asking why are you not using the <insert-llm> paid plan ??? came very very rapidly. So naturally, everybody started to use it blindly so that the dashboard metrics are all high.
It's astonishing how society forgets.
The most important skill is to not stand out of the crowd. This is how you survive in the Soviet Union, in the army, and clearly also at tech companies.
Quite a good point.
I swear the industry is being Garry Tanned.
Senior management let go our localisation staff. Now they want us to use AI to translate. They still want manual review.
We use Github Copilot at work, we get a measly 300 requests with the budget to go over if necessary. Opus 4.7 or GPT 5.5 would eat all of those up in a day. Are we supposed to be using more than the allotted amount, do management see that as a good thing. Or is it best to stick within the allocated amount. Who knows? Management are playing games everywhere it seems.
How you burn 300 requests in a day? From my Copilot usage Opus consumes surprisingly few requests to do a lot of stuff. It isn’t paying by token but instead by prompt or something.
If you are using subagents for asynchronous work, you can burn through 300 requests in a workday easily.
Copilot didn't charge for subagents. You could do an insane amount of work with dozens of subagents with a single request and a deep enough prompt to kick it off.
I setup entire virtual teams (Dev, QA, product, reviewers etc with the initiating model just acting as the agent manager to keep it's context minimal) to one-shot some stuff and it kept churning and making progress.
Those days are just about over with the change to token pricing but for a time....
300 prompts in a day isn't that unreasonable to achieve on a heavy day? And Opus has a significant multiplier as well
Yeah, thats 20 Opus 4.7 prompts.
That is a lot of prompts. What kind of prompts do you usually give?
Mine are usually giving specification and telling Claude to implement some part of it, referring to existing code base, writing unittests and running e2e until it passes. This can easily take 4-5 hours.
Then again, I have seen colleagues prompt “are you sure?” And other nonsense like that
I think we use it differently then, I tend to go heavy on the back and forth.
For speccing things out, I have a back-and-forth with the grill-me skill, break things down into tickets, as well as kicking off subagents. That said, I significantly overestimated the number of human messages I send.
My daily 90th percentile wrt number of prompts sent is sitting at 160 queries / day and average at 97 queries / day.
Ran an analysis of my last 2000 messages, with the following breakdown
Task delegation / execution: 23% Investigation / diagnosis / “what’s going on?”: 21% Planning / architecture / brainstorming: 15% Testing / verification / release ops: 10% Review / cleanup / quality control: 9% Course-correction / constraints / preferences: 8% Agent / ticket / workflow orchestration: 6% Providing context / evidence / pasted material: 4% Social reactions / acknowledgements / vibes: 2% Other: 2%
Yeah that’s not good for current Copilot usage since you are paying per request not by token
Yeah, copilot is only my backup - Codex and Claude are my primary, where it's per-token. Just looked up my user prompt count from my opencode db
I guess you need automation for that. Run claude with cron to find fulnerabilities, suggest and implement improvements, automatically dig through backlog
Hope people doing those kinds of automations are paid to waste prompts and tokens. Any cron based LLM run is just stupid waste
Opus 4.7 has a 7.5x multiplier when it's used from Copilot. Falling back to 4.6 it's only 3.5x
It recently went up to 15x in our org.
Copilot charges a 27X multiplier on Opus 4.7 prompts.
Review-fix rounds after generation of text or code, until convergence to a solution that doesn't need more improvements.
You are prompting the same model to review what it just did?
We've raised, trained, hired and promoted generations of business people who push utter nonsense, understand nothing but optimizing for bad metrics, and orient solely around short term results. It's hard to look beyond modern corporate America when looking for causes of the fall in our living standards. This AI tokenmaxxing nonsense is just another rung on the same ladder to hell we've been on for decades.
Requests are such a weird metric. We have a token limit via Copilot (unless I'm misunderstanding our setup), and most of my "features" burn 1 to 2% of my token limit per month on 4.7. But I don't admin our plan, and I'm unsure what we actually git. Vscode just gives me a percentage of tokens remaining metric.
One of the weirder things about all this is how arbitrary and non objective the billing structure seems. One of the reasons I'm happy to use it at work, but won't ever personally subscribe. It's so opaque.
Copilot is currently based on requests (1 prompt = 1 request, with multipliers for different models). At the beginning of June the billing structure will change to just be normal API cost. Your features are going to start burning 10-20% of your token limit using 4.7
At my employer, everybody who has an opinion that matters is convinced that all of the overages by the high users will be more than made up for by the people who barely use it.
Maybe they’re right. But it’s really hard to see how.
It's just not with AI though. It's who they get their advise from. One of my friend was cribbing to me about his company management - apparently someone in management discovered that PostgresDB is a real good database and free, and so they authorised the IT department to migrate their application from Oracle Cloud to PostgresDB as it will "save a lot of money" (true, but...). However, they aren't willing to shell out for the commercial solutions (like EnterpriseDB, which would be still a lot cheaper than Oracle), and are insisting that the team also recreate "all and every" feature that Oracle DB has and is used by their application, but is lacking in PostgresDB - after all, "If Oracle can do it, why can't you!?".
Wow... How (in)competent is his management??? "If Oracle can do it"... in 25 years with 1k devs...
47 years if you count from the first release. But now you have this super intelligent thing that enables anyone to create a billion dollar business - you have no excuse!
"Hey Opus, create me an fully tested code base for Oracle-like DB from scratch. Don't overcomplicate it, so it should be ready with when I get back from lunch"?
You forgot "make no mistakes"
I had a similar experience but with MSSQL, was invited to join some meetings with MS Sales folks. I quickly learned the project was never meant to succeed, but was simply leverage to negotiate a better contract.
Memories of me and my three developer team being told "we need to use Excel, but in a web browser. So just make an app that does everything like Excel"
Imagine selling a product where companies are foaming at the mouth to increase their spend and pay you more money
It does not get any better than that
Jensen, Sam, Dario: https://i.imgur.com/AI7rtCY.jpeg
“Show me the incentive and I'll show you the outcome.”
― Charlie Munger
Would that make chasing perverse outcomes in the corporate environment the Munger Games?
When I was at Amazon, I suggested that promotion to L7 people manager should require that reverse tattooed on your forehead so that you saw it every day. Every time some mandate would come down from on high, it was clear that nobody had thought of the second order effects, malicious compliance, or just outright gaming.
Goodhart's law
Akshually… I’d say they’re not the same but related. Goodhart’s is a corollary of Charlie Munger’s quote.
People who don't code(management, leadership) think AI will 10x the company but it's really a 40-60% boost. But engineers have to feign adopting this tools in fear of layoffs
40% boost for smart engineers, for now.
People churning out slop is slowing me down and the full effects of it won't be felt for a while.
the new bottleneck for development at work is code reviews. devs are creating whole features that would take months in only a couple weeks, but code reviewing that is a slow, painful process
The bottleneck at my work for development was already code review before LLMs.
This is why I'm not that excited about vibe coding. The bottleneck has always been understanding what the heck is going on.
In my view you should 1) use AI as a tool to help you learn and 2) write boilerplate you could have easily written yourself. Getting it to think for you is counterproductive (at least until it replaces us entirely).
Yesterday, I had my first experience of a mid-level dev stuck on a problem, coming to me with Codex and Copilot summaries of what those tools thought the problem was, which turned out to be completely off-base.
Codex was pretty sure something was wrong with the response object being returned by the endpoint in question. It turned out there was a conversion method applied to the endpoint response, which mutated its input. This method had been running w/o problems for a while, until the dev put it in a useEffect. At this point, React dev mode's policy of rendering everything twice kicked in, which caused the second pass through the conversion method to fail on the now-mutated input object.
Codex never even hinted that the conversion method mutating the input could be a problem, nor anything about React dev mode rendering everything twice (specifically to catch problems like this). Apparently, neither of those came up much in its training data.
My point is that this dev seems to have lost, in a few short months of writing everything with Codex, the ability to trace an error from its source (the error trace was being swallowed in a Codex-written catch block that spit out a generic error message). He was completely stuck and just kept doubling down on trying to get Codex to solve the problem, even checking with Copilot as a backup. I'm not optimistic about where this is headed.
Are you sure he was capable of debugging it before?
Yes, eventually. Largely because he would have written all the code that got to that point and had a mental model of the entire flow instead of it being a gray box.
Its not really a 60%. It accelerates a lot code creation. Save some time on admin tasks. That is it.
Where? What industry, what kind of projects? The only one where I can imagine it to be true is vulnerability research, and I imagine all the low-hanging fruit to be picked soon
Any typical web backend or frontend kind of thing. So like, not systems code.
Mine, easily. Senior (near staff) level embedded engineering.
It will spin up a boilerplate uboot or BSP config no problem. I still go in and manually check and add peripherals, but opus 4.7 is terrifyingly smart.
Need to modify or add a new peripheral, it's there no problem. Or in a bare metal project, I can point it at an STM32 cubemx starter repo and ask for a feature (set up the ADC on pins 4 and 7, ask me for parameters) and it's just done. I do in a day what would probably take me 2.
It doesn't help me with reviewing others' work, or planning (I maintain that these are manual tasks). So yeah, I agree with the 40-60%. The parts of my job it helps, it really helps.
My experience is it will attempt read from the wrong memory block resulting in garbadge. But that's a while ago so maybe LLMs have gotten better.
The AI labs have all released at least 3 new models each since december, things move very quickly.
Yeah, the industry has no issues selling $5 bills for $1. Why is this a good thing for society again? That the public subsidizes VC to no shared gains?
I've been hearing "the new model is so much better than the one from 6 months ago" every few months since 2023. It's never been true to date, so please understand why I am skeptical that it suddenly became true this time.
I'm confused, isn't the whole point of using the STM32CubeIDE that all the peripherals, like say setting up an ADC on pins 4 and 7, are checkbox features?
https://wiki.st.com/stm32mcu/wiki/Getting_started_with_ADC
Yes but it's famously clunky, and if I'm already in an existing repo, a prompt will do it much, much faster.
It also generates a ton of bloat and comments.
Yeah just had Codex/Gemini write me nrf52 bootloader that fit in under 4k flash sector size with OTA and DFU support (well, app does OTA download then the bootloader validates and decompresses the image). Works best if you let them use OpenOCD on a real device, then they can iterate until it starts working.
I didn't even need that bootloader, just didn't like the fact that Adafruit one takes too much space :)
I work on an ETL platform and it definitely is a huge boost in certain things, but a drain in others.
We started working on a new product a few months ago and it's really dangerous up front on an empty code base. It can quickly write more code than you can comfortably understand. The more serious danger is when three people are all doing that at once. I had to bring this up at meetings and try to get a better review culture going.
Now that we're a few months in and changes are more targeted additions to an existing system we're happy with, it's _huge_ (which has been my experience on our existing product). I can drop a brief paragraph I speech-to-texted into my agent, give it a general starting place (where I imagine the issue/feature extension point is), and then tell it to do some research and propose a change. I'd guess it's about 50% of the time that I have to update it's implementation plan. Then I let it run (my favorite is setting this up before a meeting) and come back. Then we have to review the code and go from there.
Definitely a 50%+ speed up in some cases, but not all. It's also great for problems that procrastinating, as it reduces friction so much.
What's funny to me is the seeming lack of AI usage among management despite so much of their work being amenable to AI acceleration.
At my company(big name, AI beneficiary), middle management seems to mostly be concerned with shuffling chairs on the deck of the Titanic while they wait for their stock to fully vest. There is very little interest in improving anything, just an obsession with risk avoidance and performative sideshows whenever upper management wonders why execution is so poor.
At my company middle management is using Gemini to churn out reams of useless documents in lieu of anything approaching "program management" or similar
Reminds me of the managers that use 'lines of code added' as a metric
Saw a good joke on twitter about it. Something like:
"You spent $23, over the $20 food limit. Be more careful next time. You spent $600 on tokens, $200 more than the average. Congratulations!"
https://x.com/vasuman/status/2053956365052240263
Not as long as I get discounts for over $30! I want those 15%, even if I have to pay 10 bucks more! Wait a second...
This measuring of tokenmaxxing as a proxy for something beneficial to the company has got to be the single dumbest thing I have ever heard of in my entire software career.
It would be like some company in the dot com era measuring employee's internet download traffic as a proxy for productivity or internet-pilledness.
Why not just reward employees based on who's submit the largest expenses claims? That might have some correlation to work too, right ?!
In the corporate world it's impossible for any one person to tell what's going on across multiple domains due to the complexity. If I tell you the Zorbulon API is creating 30% more flargs (which is critical for Twiddle operation), I often just have to take your word for it.
Hell, I'm in the bowels of Google as an IC and it's hard to understand what adjacent teams are doing. Even harder for management that never gets their hands on anything.
So while you know engineers are probably bullshitting you with fake work, you can at least turn around and tell your supervisor the numbers. It's all a game of plausible deniability.
It's truly bonkers: the reverse of a budget. It is like rewarding the people who spend the most money.
They used to measure loc, but this is even dumber than that. The charitable explanation is they just want to make sure nobody is completely avoiding AI use.
I have mixed thoughts on this. These thoughts are my own. On the one hand, it’s objectively silly to pretend like we’ve solved the age old problem of measuring developer productivity. Metric-obsessed leadership can also be intolerable, counterproductive, and it’s a good way to paint yourself into a corner undervaluing your best talent and overvaluing your mediocre talent.
That said, I’m kind of having a blast using CC in corporate with all the connectors available at our disposal, and I baffled how little some of my coworkers know about what’s available and what the capabilities are. So it’s clear that perhaps some encouragement is prudent for those who are slower to embrace new technologies, but I’m not sure tokencounting and tokenmaxing are the answer.
Could you list us some of the capabilities you use that bring value besides “summarize my email”
Not OP, but within Amazon we have pretty good connectors around integrating with our task system (so you can pretty easily ask your GenAI tool "look up the next item in our sprint board, let me know if you have any clarifying questions, but otherwise start implementing it"). We have decent integration with internal wiki and search systems, so it's easier now to figure out the best Amazon way to do some coding task. And Amazon being a big doc-writing company, there are lots of great tools for helping improve all phases of writing.
Yes, we can crawl our entire internal documentation via LLM. Want to know if someone is already working in the space of your latest idea? Ask Claude, it hits the internal search APIs and finds docs and references directly relevant to your query. There are a lot of separate document stores so this took a lot of effort previously. I can also query Slack, Outlook, etc. I don’t understand the cynicism in your comment.
That is a summarize my wiki. Nice search feature.
The trouble is, it's here now, and it wasn't before.
That may be an enterprise saas is shit problem, but I'm just happy that my employer now has a wiki search that works.
A company requires a specific % of code coverage but doesn't give developers enough time to actually write tests. AI can be used to generate the tests needed to get pass the code coverage and avoid being fired for not working fast enough.
I found it very useful running a TDD workflow the other day. It created a test plan, generated tests, documented them, implemented and modified existing code, and added structured logging. It also identified really good refactor candidates and explained them to me after I noted a core design issue in the code we were modifying. This wasn't autonomous: I spent some time correcting it and sending it in new directions. Still, it was a pretty nice feeling to not have to go manually configure Logback (it one shotted a nice basic config), not have to write a bunch of repetitive test setup code, etc. It even pulled in a newer JUnit feature that I didn't know about that was perfect for what I was doing. Definitely not the silver bullet a lot of people are trying to sell, but still a very powerful tool.
Vibecoded ppt, docs, frontends is an even bigger scam than crypto ever was. Ofc people getting sucked into it
Are the AI tokens fungible though?
Another stupid meme-latching name. Don't normalize these *maxxing nonsense words and just use plain language. Let's see, maybe just say they were optimizing for token count?
I like it because it highlights the stupidity going on. Bullshit doesn't deserve a respectible name.
When did FT become Business Insider?
I have an FT subscription and they keep moving toward this kind of narrative first reporting to get clicks. It’s no longer a believable paper.
Business Insider would say "tokenmaxxing is pure promotional intelligence"
Maybe we just need to subscribe to nikkei proper and ask them to make it pink and stop being peasants.
I work at Amazon (standard disclaimer: just sharing my own experience, not an official spokesperson, etc.)
I can't say that this isn't happening, but at least the parts of the company I get visibility into, what the article describes isn't my experience. There is a lot of interest in using GenAI, but people are mostly getting kudos around creative uses for GenAI, not just for raw amount of tokens. For most scaled GenAI efforts, there is a lot of focus on output metrics (metrics like accuracy, number of findings, number of things fixed, and so on).
LOL, I'd imagine even Amazon HR would be little restraint in showering such praise.
Thanks for the inside insight.
I'm surprised how few comments are written with the prior that Amazon managers aren't stupid or uninformed about how incentives work.
My guess would be that someone created the leaderboard without a lot of consultation with managers, and that some employees feel a competitive urge to try to "win" the leaderboard by burning tokens.
Your comment is the equivalent of stating, that Jeff Bezos and Andy Jassy, do not really know their employees are carrying around urine bottles.
Management is told its a homebrew. Very commonly used by developers all over the industry.
Mmm, no, I don’t think it’s equivalent. I think they know that if you make the work hard, some employees will have trouble keeping up and will do things like peeing in bottles. And they’re OK with that, because they think there are enough people who can keep up that they can push the weaker people out. I think they believe that the peeing in bottle is relatively rare. I’m unsure whether that’s right or not. It’s been reported that it happens, but I have no sense whether it’s common.
Wilful ignorance much?
Amazon is a massive company, your single experience is worse than an anecdote because there is no way verify.
What we can verify is how how Amazon already treats workers, they will surveil anyone within their systems regardless of the futility of said surveillance. Why are we suppose to not believe them using LLM systems as a means to further control their expensive employees from unionizing or seeking out solidarity with fellow workers? All LLMs do is enable tyrannical managers more power to hold over other workers, said workers are forced to engage in self alienation for fear of losing your job or forcing to do meaningless work as that is what's being tracked (and what LLMs excel at producing).
Hardly a good proposition for any worker.
I'm sorry but I fully do not believe you. This is a company that fires workers for taking too long of a bathroom break where said workers piss in bottles for fear of getting fired and you're going "hey guys, it's not too bad. Only some workers get whipped, others don't!"
I also work at Amazon and my coworkers are playing 20 questions every morning to keep their metrics up. Like anything else there it depends on your org & managers.
At least for some people I know it’s not necessarily because there’s pressure from leadership, but because it’s funny that the org spends like $15,000/mo writing HP fanfic or whatever
It is damn fascinating to see just how many (big, serious) organizations are creating unnecessary internal strife over this.
One of my favorite heuristics/quotes applies here: "no matter how good the strategy, occasionally consider the result."
Want to know if AI is working for your org? Ask yourself/employees to "show me the result." That requires judgment and taste (is the result something of value, or just the appearance of work having been done), but it will also save you a ton of stress and disappointment later.
A perfect doomsday machine. Over-using tokens gets your peers laid-off before yourself.
Each day I send the AI on a fruitless mission like "summarize the entire codebase" while I do my actual work, which involves actually using the AI for real work. Wish I could disable the token cache to make it spend more.
Once you have a score, you have a game. Once you have a game, people will do whatever it takes to win.
I don't think you need to win this, you just need to not be near the bottom of the board. But just in case, I spam tokens like it's the Chuck E Cheese roulette game.
i think the best in this case is to be solidly in the middle of the pack. don't want to be near the bottom, don't want to be a tall poppy when the backlash comes
They're not the only well-known company I've heard of that's investigating token usage leaderboards.
I, too, can easily use more tokens to achieve the same task. I can give worse prompts. I can fail to make it clear to the tools where to find the information they need. I can ask them to think hard when the don’t need to ask tell them not to think when they do need to. I can give vague, open ended instructions. I can generate code that sucks and throw it away.
If I do all of this, do I get a promotion?
Even if I'm in the middle of using the AI seriously but then want to rename a variable, I can't do that myself because it'll confuse the AI, so I'll tell it to rename. That seems pretty wasteful.
At least GPT-5.4 and up seem to do okay if I just prefix my next message with something like "(I made edits.)".
That sounds like too much effort. Better to have the AI write you a 20k word manifesto about how much you love your employer and then include that in the context of every request.
This makes me think of the tulip bubble. Using AI as much as possible just so people think you are productive is like buying tulips so that people think you're affluent.
Amazon is big and inconsistent enough that "somewhere in Amazon, <XYZ> is occurring" is statistically true, no matter how nutso-sounding your <XYZ>.
Can we have a list of AWS services which use formal methods and general sanity so I can avoid the other 500 half baked ones?
I wish I could do some tokenmaxxing at my company. The only plan available is maxed out for the month after a few days of serious work, but the AI “experts” are declaring that nobody needs that much. It’s really frustrating to constantly have to juggle quota and lower models. All this while the declared goal is to reach 50% of code written by AI.
This kind of thing is totally fine if it's being done (it's believable because Meta internally incentivized tokenmaxxing). When you're trying to change the behavior of a large number of people, only blunt instruments are available if you want to get quick outcomes. The edge cases where people Goodhart very hard are all right. You can just human-in-the-loop them away. The opportunity cost for most organizations of not moving to use AI tools as productivity enhancers is currently gauged by them (rightfully, in my opinion) to be too high to allow for osmotic adoption.
Most people look at sea changes come and go. They all have a story of how they "could have bought Bitcoin when it was $100" or whatever. In an org, you don't want to have the story of "we could have done that when nobody else had", so you incentivize adoption of the tool as hard as possible and hope that dipping feet in the water makes people want to swim. If you don't already have a culture of early adoption (and no large company can) then you have to use blunt incentives. I don't think anyone has demonstrated otherwise.
So you're saying that if you ignore all the downsides with "you can just human in the loop them away" then it works great?
Even if you don't, it's the only way to ensure adoption and most workplaces consider the lack of adoption a greater danger than a Goodharted adoption. Overall, I've observed that the US has very low barrier to starting companies so, considering companies of all sizes are doing this, if it's a mistake those startups will get beaten by the ones doing other things.
You can use Codex and Claude code for most of the tasks that you would manually do
Filing JIRA tickets, updates. Opening PRs, having AI review PRs. This will all use tokens.
No need to tokenmaxx, you will end up burning tokens with just regular AI usage
A very poor look for management. They don't know what the heck they're doing.
Measuring productivity via tokens is the modern day equivalent to doing it via number of commits or LOC
Hunger games in the age of AI - eliminate/automate your colleague's job, until a single software engineer is left (or two if aristocrats will see it as a good PR).
tokenmaxxing is silly, but if a developer or manager NEVER uses AI then I do think that's cause for concern as it shows a genuine lack of curiosity... perhaps tokenflooring makes more sense than tokenmaxxing
Or maybe that person has made an intelligent decision based on their own workflow and requirements?
It's the same as measuring productivity by lines of code written, same dumb logic by management, not surprising.
Our AWS TAM has recently started to respond to us in AI-like responses. It's very obvious. Now it makes sense why.
Similar to an HFT company I know, using the money spent on tokens per developer as their efficiency metric. Insane.
Hot take:
There should be an anti leaderboard that highlight people under a threshold. Not trying to learn how to use ai while working at a company like Amazon is almost certainly a bad thing, and cause for looking into why.
https://en.wikipedia.org/wiki/Poe's_law I was just joking about it a few days ago (I swear I didn't know Amazon was doing this) https://news.ycombinator.com/item?id=48079533
These employees are going to automate themselves out of a job. I've always automated, the boss never has to know.
Amazon has this Kiro product they are trying to sell and they are using their own employees to improve the product and their own LLM. They are giving uni students 1000 credits/month and running competitions.
amazon leaders are obsessed with "metrics" but they care very little about what those metrics actually measure
it's all a political performance.
Remember when they made the decision for return to office and just...decided to post no support data on why they need to do it?
How about we actually try to solve problems and measure that instead of measuring what we believe are the steps that resemble solving problems?
So I have seen people using Claude to change variables’ names. Is it tokenmaxxing?
gaming the metrics instead of using the tools well. this'll show up in the capex ROI question at the next earnings call
Old and busted: Hey, whoever amongst you wants to try out AI in the business? Check your idea with legal to make sure we don't get busted for violating HIPAA/GDPR/PCI-DSS and you get 1 k$ worth of tokens at your favorite AI service to play around, if it's any good you get a week extra PTO.
New hotness: USE AI NO MATTER WHAT AND WE WILL MONITOR EVERYTHING, THOSE WHO REFUSE TO USE AI WILL GET FIRED.
And that's how you slop yourself into (at least) two major downtimes and burn millions upon millions of dollars for zero ROI - but the stonk markets don't care about lost ROI as long as you go along with the AI hype train.