I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
We solve it 2 ways in the Bazel ecosystem: for the intermediate artifacts, we only fetch the digest (hash + size) of the blobs to calculate the merkle tree forward. The blob itself can stay on the remote cache server.
For the bigger final artifacts, we support using Content Defined Chunking (rolling gear hashing) to only fetch the missing chunks between incremental builds. Binaries executable with stable layout benefits from this quite a lot.
We are definitely not done with all of the improvements here. But since all the major AI labs are using Bazel, we know that the tools can support “Agent Scale”. https://webazel.dev/
calculate the merkle tree forward. he blob itself can stay on the remote cache server.
Not sure how that would work with building say a docker image, reproducible builds are pretty hard problem to solve, and caching intermediate layers is not always simple or even doable, we typically still need to publish to a registry which is not the cache server.
The Bazel ecosystem builds container images not by using Dockerfile, which contains non-reproducible primitives such as RUN and others. We do it by actually constructing the file trees and tarballs manually, then using them to compose the JSON manifest and indices. This is done via smaller hermetic and reproducible Bazel actions and thus enables the ecosystem to scale way beyond what alternative BuildKit-based solutions can.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Depends on which targets and how granular the caching is.
- if you touch package.json that might invalidate everything
- touch core and you'll invalidate everything
- touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
What i have seen on my end is that it’s pretty easy to setup a cloud coding agent with a warm Bazel cache. “Warm” here can means multiple layers: same disk to CoW/hardlink, different disk, network disk/block devices, same rack/datacenter, same AZ, etc…
And yes, there are a ton of investments going toward Bazel recently to unlock these newer use cases.
Agentic Coding has been a huge strain on CI,
I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
It used to be a standard expectation, but that was before google or AI could have answered your question in less time and typing that your complaint on HN required. Especially when the author is writing a blog post for an audience that knows what CI is.
I don't think it would occur to me to write out Continuous Integration because it's basically a nonsense term if you don't know what it means anyway, it's not any less opaque than just CI. The problem isn't the shorthand here, but yeah it's probably good to spell it out anyway so when someone googles it they have more confidence they've found the right thing.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
> where sub agent orchestration is done through agent to agent messaging
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
QA is not automatable. QA is about finding the things that you didn't think of and so didn't write a test for it. That many people think qa is not important shows in the bad software we have.
QA is about finding the things that you didn't think of and so didn't write a test for it
QA is about quality. That it has been increasingly used to mean "repetitive manual testing only" is part of the very trend you decry of qa being undervalued.
Fundamentally QA is about two groups of people collaborating in an adversarial process ("you build it we break it") to achieve a common goal: a better product. QA by definition requires a relationship of equals, or the process ceases to be adversarial and becomes useless ceremony. If you're not able to assemble two distinct groups of people, one person can wear both hats (ie you test your own software) but you'll have more blind spots.
How much of QA is done by a human or automated, is and has always been an implementation detail.
If you're in charge of QA, you're in charge not just of finding new problems, but preventing regressions as well. And you're responsible for embedding as much of that work into the devloop itself, so that builders can find and remediate problems earlier and with less pressure on your limited time. How do you achieve that? automation.
I started my career doing QA and 99% of what I was doing could be automated by an AI today. Today I still do QA on my own product- I just spend all my time on the remaining 1% and my product is better as a result.
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
Walter Reuther: Henry, how are you going to get them to buy your cars?
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
Which is a bit terrifying and removes the ‘engineering’ component of a critical part of the job, in exchange for slot machine / pray to ai gods…which comes with disconnect from the code, something I assume will adversely compound over time.
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
I was just discussing with my team that this exact thing maybe brings back to relevance the idea of "behaviour driven development" and I was reminded of this Cucumber/Gherkin lib & language that defines a kind of executable prose you can use to specify how the software should behave. It's an interpretable programming language but designed to be close to how a human might just write down their specs of what kind of actions and responses are expected from a software system.
The idea is to drive actual testing from this, but in this era, I think it's interesting as a way to use AI to generate tests, and to cross-check those tests with the natural language descriptions, in a bit of a cycle that helps refine the highest level definition of the software.
Once that's nailed down, the implementation is just details.. normal engineering concerns like maintainability etc notwithstanding of course, but you can trust more and more the AI agents to get it right. The design specs being natural enough for humans to deal with but interpretable/specific enough to actually generate tests is pretty interesting for the bottleneck you are talking about, I think.
Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
to be fair all CI providers I've had the pleasure of working with have gnarly performance profiles for the boxes they provide.
I don't think it's out of malice, but I do feel uncomfortable with the fact that the people who sell me the CI coordination software also sell me the minutes for the boxes that run the CI software. There's _some_ alignment of interests but not as much as I would want!
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
I wonder why larger companies don't use self hosted github runners. You can buy a pretty beefy machine (TB of RAM, 256 cores, fast NVMe disk) and tests will run faster than on any hosted platform. Plus you don't have to shard as aggressively because more fits into one machine, benefit of shared page cache, shared persistent disk, can easily preserve working directory (for example my pnpm install takes 0 seconds, because the node_modules folder is already present from the previous run).
Microsoft is pushing so hard to get companies far away from the on-premises world. Once they're in the Cloud, there's too much vendor lock-in for big enterprises to go away.
It's sad that even so many greenfield businesses still default to it. There is no longer a need to. The barrier to alternatives used to be technical, now that's gone. 99.9% of web apps' CI needs are solved by a $10/month box.
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
None of them are bad. It's like saying a bike vs a scooter vs a car is bad. Use them correctly and for what they're intended and they work fine. Use them improperly for the wrong things and suddenly people think the tools suck, when it's the humans who misused them that suck.
If we lived in a world where people or agents could define those service boundaries up front, correctly, with some reasonable foresight for change on the horizon ...then sure...I'd agree.
However, our world is not that world. Your agents may be happy in their tiny walled kingdom of toil and ineffectiveness, but you and your users will not. Poorly drawn service boundaries will drag you down more than almost any other architectural mistake.
If there is one universal amongst organizations, its that they love walls and silos. Be wary of putting them up ahead of time, cuz tearing them down once established is nearly impossible.
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
Personally I believe that this boost in productivity will not necessarily lead to greater revenue.
All the companies have the same access to AI, and AI is making them all better at doing what they were doing before (writing software). So some companies that use AI really well may be able to take market share from other companies that are slow. But I think this might just lead to a more intense competition for customers.
Kinda like what happened to music after digital recording became the thing - we have more music than before and the music is better, but being a musician became a much more intense competition to find an audience.
Well I don't really know what will happen in the future. My first thought is that what will happen depends on the company and industry. I can think of some non tech companies where AI might not matter for a while, like a car wash or a lumber mill. But long term likely yes for tech companies and companies where tech makes a difference.
People won't want to hear this, but for FAANG things are probably similar. I'm not very convinced that Meta, for example, is actually creating new revenue. It's just consolidating a lot of the existing global ad spend, it's just wrecked newspaper classifieds and the like.
If every software company saw a roughly equal improvement in productivity, they'd still be splitting the same customer base amongst each other - not clear whether revenue would actually go up for anyone.
Perhaps some new markets could be entered that weren't feasible before?
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
If you are product-driven, life is pretty good, and maybe has never been better
It has certainly been better for junior devs, mid devs, anyone looking for zirp era salary growth and supply demand ratios. For a senior code-adverse coaster looking forward to see his static salary get eaten by inflation, life has indeed never been better.
Yeah, at Corp, that makes sense. I was thinking more along the lines of scrappy startup silliness. "I am one person and I deal with any and all code to get my product built."
From that POV, has it ever been better, assuming you can rise to the top of an ever more crowded market?
"I am one person and I deal with code to get my product my built."
From that POV, has it ever been better, assuming you can rise to the top of an ever more crowded market?
Building a product is easier now, same way making a photograph is easier now. Making a reliable and high income living out of it though, it looks increasingly uncertain.
In a world with abundant music and photography, just saying that you made your music using Reason/ProTools or you made your photograph with a Leica isn't enough. I believe the same will happen with software. Sales, product design, customer service and operational aspects like uptime will matter more than the software itself. Ironically, this is the kind of stuff the HN type highly dislikes.
If you are product-driven why not go into Product Ownership roles? It certainly will be more satisfying compared to prompt "engineering" clerk "career".
Well, my life is currently both, as I am on the smallest possible tech and product team. To me, extrapolating on having done this since Sonnet 3.5 to now, there will be no tech aspect to the job soon enough. There are many hundreds of billions invested in making that a reality, and they are succeeding.
Until ~9 months ago, I spent most of my time being a "prompt engineer." Today I join a meeting, ten minutes later I receive the transcript. Then, I run a custom skill in my project, and I get Jira epics and stories to triage that are nearly perfect. Then, I run the second skill orchestration skill... some babysitting... and ~85% of the time that is all I need to do. Docs, code, unit and e2e, great UX... all there after every meeting, and basically two commands on my part.
I see two to three to maybe five years before anything I have to offer, in any capacity, is completely cut out of the picture.
I honestly don't understand how everyone is not on this same page. The labs are going to eat it all. The only reason I see for them to talk about "pausing," is because they finally realized that they are going to collapse the entire service economy around themselves at this rate: aka, the USA.
Cost is stable (or rising?), user-facing delivery is sameish, (some) engineers seem happier because they are allowed to gold-plate and prepare for future “human development of the hard-things”.
Engineers are happier because they can spend more time on that extra round of polish that was just filed into low-priority "fix when time" Linear tickets.
QA is happier because the stupid bulk operations are handled with AI and they can focus on the hard to find stuff that tickles the QA mind in a special way =)
Same stuff, but with better (in-house) tooling and (most) people are happier. The Code Artesans aren't, but it's mostly their problem and they'll find companies that only do fully manual programming for specific niches.
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
products with a real userbase that couldn't exist or be scaled pre-AI?
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
I'm pretty sure we would find an overall increase in actually useful apps
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
more code faster is competing against some of the largest tax hikes of the last ~70 years, economic uncertainty on what the taxes tomorrow will look like, combined with an energy crisis.
Are you claiming productivity should be growing much more slowly then it is over the past few months or years? Can you show me any analysis that supports that claim?
Because I have seen plenty of analysis published in the past 30 years that puzzle over the productivity growth stagnation in the face of an exponential growth in capital expenditure in computer technology.
So by all means share with me some of the groundbreaking results showing that we can finally see more productivity growth than would be otherwise expected by the conjecture.
There's a difference between task-level productivity, which AI has been shown to increase, and effects on the macro. You're talking about the latter while most people here are engineers considering the former.
It's still unclear if task-level productivity gains bubble up, but it's also still early and I'm not sure we should expect to see immediate results there.
Once the previous goalpost gets demolished on HN, there's always someone new to put a new one. We aren't quite there yet but eventually the goalpost moves to something that is just impossible to even measure. That'll be the end game surely
It’s interesting that people claimed the exact same thing when computers were being introduced to the work place int the 80s and 90s. Lots of papers showing how productivity didn’t go up at all and using a paper and pen seems to be just as efficient as using a PC.
Even in the 19th century, when electricity became widely available, there was no productivity gain for 30 years at least. This is a well understood phenomenon.
This is just ~2 - 4 years after ChatGPT launched, and despite very shallow adoption (only ~6% of all work hours.) As a sibling comment indicates, it took almost 2 decades for the Computer Revolution to be visible in national level statistics.
Also note this study was originally published in 2024, then revised in 2025, but this preliminary evidence has been around for a while, if people wanted to find it. It's even been posted to HN a couple of times, somehow it just doesn't get the attention you'd think it should get, even if it was just to poke holes in the conclusions.
When we feed these estimates into a standard aggregate production model, this suggests that generative AI may have increased labor productivity by up to 1.3% since the introduction of ChatGPT. This is consistent with recent estimates of aggregate labor productivity in the U.S. nonfarm business sector. For example, productivity increased at an average rate of 1.43% per year from 2015-2019, before the COVID-19 pandemic. By contrast, from the fourth quarter of 2022 through the second quarter of 2025, aggregate labor productivity increased by 2.16% on an annualized basis. Relative to its prepandemic trend, this corresponds to excess cumulative productivity growth of 1.89 percentage points since ChatGPT was publicly released
These long term data suggest that this "excess" remains bellow historical productivity growth
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
App store revenue actually decreased! That suggests that the apps are largely not useful, unless you make the implausible assumption that nearly none of them are charging money.
If they were actually useful though, demand shouldn’t stay the same — more niches should be getting filled. If they’re just clones of each other, then yes
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
I don’t think this is happening, at least not on any scale that makes any of this actually useful. One off success stories do not make a revolution.
It absolutely has generated a massive amount of slop, though. The problem is that you still need intent and while I think LLMs can help you free up and push through annoying boilerplate or tedious spots where known solutions exist, you still need to design these things and that’s something I’ve not seen an LLM be too helpful with.
In the gaming space, retro console PC ports have absolutely exploded in the last year. People are modding old games with 4k textures, ray tracing, DLSS 5, widescreen support, uncapped framerates, dual-screen (e.g. Ayn Thor), etc. PS5 emulation has gone from almost nothing to AAA titles in-game in like 2 months, and apparently now runs on xbox (lol).
The other day someone posted a reverse engineered GPU driver for their Mac[0].
Anyway, what makes you think those pushes aren't individuals meeting their own needs? I expect it will take a while for this to really sink in, but the future is people asking the computer to build exactly the app they want.
Also, English localizations of even super-obscure foreign retro games.
Most of those might be not as good as fan translations from dedicated people with good knowledge of English and the original language (especially for very context-dependent languages like Japanese) but they are definitely good enough to follow the story and play through the game.
Has anyone said that AI is fixing user research, actual idea generation, marketing and audience reach? Or are you assuming that better code should be making all of that irrelevant?
It either delivers better outcomes or fails to do so for not being as general and useful as touted.
The original sales pitch from the major player in this technology hype cycle is "cure cancer, fix global warming, take over the economy". Not "more LOC".
whats the large hits that have happened in the past couple years?
I think ai has raised the floor quite heavily on what it takes to be a large hit.
all those vibecoded pushes are people building bits and bobs and variations on each other, instead of buying it or using a common service. If it could be done by ai, it will be written off as unimportant.
Not OP, but I ported a legacy .NET WebForms application to Blazor. The actual code migration was completed over roughly a 48-hour period, followed by fairly extensive testing.
We were fortunate to already have a strong end-to-end test suite written in Python, so we could run the new application against the same tests and verify that the existing functionality was preserved. QA found around 20 bugs, which we fixed pretty quickly before launching.
After the migration, we also moved the application's authentication from Shibboleth to Entra OAuth and deployed it to our OpenShift infrastructure. We couldn't do that with the old WebForms application because our cluster doesn't have Windows worker nodes, so the legacy app had been stuck running on VMs. Getting it onto OpenShift gave us another operational and cost-saving benefit beyond simply modernizing the codebase.
I did this back in January, when the models finally became capable enough for this kind of work. I believe I used GPT-5.2 through Codex. Successfully completing this project is what finally got me fully on the AI bandwagon. I had used AI before, but mostly for smaller tasks like writing code, refactoring individual methods, or making isolated changes.
The application is now modernized, more stable, faster, and more functional than it was before.
I think the project worked as well as it did for two important reasons. First, I had deep domain expertise in both the application and its surrounding systems because I was the original developer. Second, our QA team's test suite was comprehensive enough to validate the functionality that actually mattered. AI dramatically accelerated the work, but there still needed to be someone who understood what the application was supposed to do and a reliable way to verify that the new implementation behaved correctly.
None of this means we couldn't have done the migration without AI. We absolutely could have. The difference is that AI made it possible to do it at practically zero cost in terms of engineering time and money compared with the alternatives.
We had wanted to move away from WebForms for years, but with the size of our team and the constant stream of new feature requests, there was never a realistic opportunity to stop development for months and focus on a rewrite. I estimate that doing the migration myself without AI would have taken at least six months to do properly. The other option would have been hiring a contractor, which I would estimate at $80,000 or more over six to nine months.
Based on my token usage, those roughly 48 hours of working with Codex to port the application cost about $150. Those were January prices, so I don't know what the equivalent cost would be today.
For me, that was the project that changed AI from something useful for assisting with individual coding tasks into something I saw as capable of fundamentally changing the economics of software engineering work.
I'm running several. They're not "large hits" but they're providing value to thousands of people. They couldn't exist pre-AI because core functionality depends on LLMs.
What you're missing is that slop has increased 20x but quality new releases have also increased 5x. Unless there's something in your niche though you'll mostly see the 80% slop so I get why you feel that way
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
People mean where are the good new apps AI made possible. Of course it has a 84% year-over-year slop app submission.
If the cost to produce goes down, unless the demand is elastic, you'd expect fewer big hits in favour of a more fragmented, more competitive space all earning less.
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
That's exactly the opposite of what I want to see. As an app store user, I want to see fewer, higher quality apps, not more shovelware or slop.
Last time I looked the other app stores didn’t show anywhere near that much of a bump (that was a several months ago though).
And a few months ago I went through a random sample of new apps on the App Store and the vast majority were just wrappers for AI APIs. So it was more of a new gold rush situation than a productivity bump.
I have this theory that the economics of AI development would make it so it's much more viable to make in-house, custom, apps rather than paying subscriptions for apps. I have even seen this play out on a personal level. A friend of mine who has no dev experience vibe-coded something custom that evite does. So it's possible measuring the number of apps miscounts ones that are not public or shared. I'm not negating anything you said, just adding something about metrics on apps.
Maybe one day but right now in order to get any meaningfully usable app with AIs you need to pay way more in AI sub than monthly sub of a bunch of apps. It doesn’t make sense right now to build especially your own if you’re not a company. Also it’s not clear how long we’ll have this heavily subsidized subscriptions.
At the 60 person smb (travel industry) I work we've taken web development in house for the first time, created new user experiences that sat on our roadmap for years, created custom AI solutions on top of our Fresh desk tooling, and (predictably) created a number of internal productivity tools.
The actual bottom line result is very fuzzy. Cost are shifting, by spending less on agencies but in return opening up new positions or shuffling people internally.
We don't see a clear increase in revenue, nor can we accurately tie retaining revenue to our AI initiatives.
Workplace satisfaction scores remain similar, with a few outliers (e.g. some people's work got incredibly exciting, while a few others are terrified by all the change).
Time will tell if this all leads to anything meaningful for the business.
We know it's easy to produce working code now, what's missing is useful software that actually solves a problem. What problem does that toy OS solve? The main point of toy OSes has been education, but you learn nothing when using an LLM. So all of this is just pointless energy consumption, it's not solving any real problems that people have.
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
It’s hard to imagine a company more all in on ai than Google. The pressure to use it is intense and they are throwing ai at basically every problem. If a proposal doesn’t have ai in the title somewhere it has no chance of funding.
I don’t really think it has made Google products better, but it is absolutely the expectation.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
Why do you assume "we not seen an improvements in products"? Who is "we"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
When did previous releases stop meeting your criteria?
Ages ago. It was really noticeable when they prioritized features over quality.
Such as?
HN is full of them. I don’t have great have specific example at the top of my head, but lots of hard reverse engineering/low level projects credit AI and that they would never have happened without it these days.
I mean, looking at the release notes for things like say the Linux kernel it is clear that the improvements are there at a massive rate of change. But it's small stuff, it's performance improvements, it's quality of life. If you do things right people will think you've done nothing at all. What did you expect? Businesses to come up with whole new business lines, or is it just that backlogs of work is getting burned down that was going to be gotten to eventually? From what I can tell from the PM forums, the next bottleneck is the Product Org coming up with ideas worth implementing as fast as the developers can deliver them.
Why have we not seen an improvements in products? [..] Is everyone just running full speed in circles or something?
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
Why not? If you let the LLM run amok, you get 4 one line functions calling each other instead of one 4 line function. Run that for 24 hours and you'll get more than 2000 tests.
I've been bootstrapping for two years now, just me and AI tooling.
Code moves much faster, product taste doesn't. Once you have your marching orders you can make features 10x faster. Shots off target still miss though. It doesn't matter if your code gets generated 10x faster when it doesn't resonate with users.
"This" is lazy but yes, this. Especially in the app space, 90% of the stuff coming out has barely any "do people who aren't me actually need this, does this deliver something new and useful" thought put into it. It's a skill most people lack, they can't judge it.
Bootstrapping a solo software business has multiple core required skills. Technical skills are now largely unnecessary, but it turns out most people lack multiple of the other required skills too.
I've definitely noticing buggier and buggier software, that's for sure. Even our internal tooling and CI pipelines and all that kinda stuff has taken nosedive thanks to the deluge of garbage
PS5 emulation has gone from barely working to running Dark Souls at 10+ FPS with virtually no graphical glitches… in 6 weeks. If you’re not familiar with normal emulator development time, this is… quite extraordinary. There have been insane progress on decompilations and many other things in the emulator space.
Everyone’s trying to figure out how to convert this speed to product features at scale, but enterprises are like container ships. Lots of might but slow to turn. The littler companies can actually take advantage of this and produce higher quality products at much faster speed. I think you’re expecting too much in the short term and too little in the long term. AI-native companies are gonna eat everyone’s lunch, once they figure out how to actually do it reliably.
Also, in 12 months, we went from seeing game dissassembly and decompilations projects be slow and annoying, to suddenly having enough mapower where people are taking the liberty to CHOOSE which decompilations to support. DK64's release was 80% anti-AI marketing, not because they were strictly anti-AI, but because they were telling the OTHER decompilation to fuck off and learn some goddamned standards. Completely unthinkable before.
There was a video by a layperson 2 years ago where they wanted to know the mechanics of some elusive pokemon pinball spawns where there was a whole bunch of missinformation about it online. The people behind that project had the correct addresses but they were completely unlabeled and the layman had to essentially figure out and do some of the work themselves to sort it out and figure it out. Nowadays that would never happen. An LLM will just do it for you within the day.
The fact that this is an account with a username specifically focused on this specific concern that was created 5 hours ago is real weird/suspicious.
Also, are these even actual decompiles? Sure, playing Wheel of Fortune with a LLM model that at some undetermined point spits out what seems to be a working example is sorta useful, but also…how exactly do you debug this nonsense? Don’t tell me we’ll just have the LLM thrash around a bunch again. What I’m asking is does anyone understand what is going on here other than “lol, it works, shut up”?
LLMs work great at decompilation, I'm not sure what you are trying to suggest. Outside of game compilation disassembly is useful for debugging various crashes in 3p code where you don't have the source. And yes LLMs do the debugging here.
I’m suggesting that having a model wander around stochastically is not necessarily a great way to gain any insight or understanding. LLMs do not think, they cannot really debug anything. Attaching intelligence or intent to them is a non sequitur, it’s not happening. A cloud that looks like a toaster does not imply anything beyond the human obsession with pattern matching, why do we keep doing this with LLMs? They can randomly wander around and sometimes produce useful output.
I don’t doubt they help by randomly permuting around in a way that people cannot do at scale, but I question the economics of this approach and I think it needs to be used judiciously. Sometimes you just have to do that especially like you said, when you do not have the source or the code is intentionally obfuscated, this can be useful, but a lot of people seem like they’re just vibe coding crap and blowing money on tokens until they get a useful result and I have no idea what the value of that is.
You must be new here. Lurkers posting under an apt username then disappearing is a feature, not a bug. He doesn't need to argue with anyone online - he said his piece and we're free to take it or leave it.
Usually, green username are a sign of "leave it", I'll agree. But this one seems very informative.
AI-native companies are gonna eat everyone’s lunch, once they figure out how to actually do it reliably.
Couldn't you say this about any company? What isn't just a matter of "figuring it out"?
Why can't they use AI to crack this nut? :D
If you’re not familiar with normal emulator development time...
Emulators have historically been a shitshow until someone figures out missing pieces here and there. By their nature, that tide lifts all ships. It has very little to do with using AI. I'd argue the real reason emulators aren't what they used to be is the hardware is more complex now and the scene is no longer attracting the most talented devs. Video games used to be on the cutting edge and carried a lot more cultural weight. It's pretty underwhelming to get cred for working on lame x86 hardware that plays yet another franchise reboot.
I did the same thing with a old multiplayer game called Wulfram, we got it online and a mostly working server (most of the game logic was server side with no surviving code or binary) with the help of AI, Mostly opus and astra. https://wulfram3.com is our efforts in this.
The real question is why are people too lazy to spin up their own phorge. It's a form of learned helplessness supported by a supposed business case to prefer buying services rather than bootstrapping your organization with proven technology.
the important thing to microsoft is still finding new ways to make money, not to make your clicks faster. they might even be using their new velocity to make clicks slower with an ad in between.
there's alao much less incentive to build things to share. people can just make their own widget app and not need yours
Agreed, but give it time. The tipping point of good models only just arrived with Fable 5, GLM 5.3, Grok 4.6, Muse Spark 1.3, and the like. Those are the only models that do a better job than me, and I'm happy to hang up my IDE—and that was barely a month ago.
I've been fixing everything that has been sitting there—not blocking us, but slowing us down—all the things we never had the bandwidth for. We're now squeezing more out of development, CI, and production.
I'm not sure why, but I'm concerned this may be the heyday, and we might not get this again at this price or speed. So I figured we should clear the backlog while we can still afford it and still have the ability to do so.
But now things have changed, and the bigger, grander ideas are starting to brew.
Probably for the same reason that SV companies hiring thousands of developers struggled to improve their product much past the original product, that was built by a handful of people.
Scale in headcount was a tactic to get investment, then you had to find stuff for everyone to work on. Suddenly people have the time to engineer so hard that we get runtime JSON defined CSS rendering engines to produce the same buttons we've had since 1995, instead of just writing a stylesheet and html.
Code output velocity from AI threatens to be useful, except it's also prone to over-engineering and burning tokens on the unnecessary. It's learned from the best after all. My suspicion is there is a lot getting done, but it's just not that impactful to flagship products.
As others have noted, there is a lot of new work going into passion projects that would have never happened otherwise, and that is cool. But I wouldn't hold my breath for SV tech to become super pragmatic and effective.
With pay-per-token, there’s also an incentive for over-engineered but functionally harmless architectures. A json based css engine is probably something that can be test-cased really well in the training set.
those thousands of developers aren't working on the FB feed or whatever...
they're working on the ad tech business and other business-related systems involved including internal tools.
and all the other platform engineering shit under the hood that makes much of the web scaleable... and lots of it is open source and contributed to by various engineers from these companies.
end of the day, these are businesses. they're not charities or whatever casual shit.
nobody is stopping you from building a competing product that's lean or whatever.
why don't you do something like that? i'm sure you're a genius.
No seriously, companies hired developers just to starve the competition of talent, raise more investment etc. It was a genuine thing, that left a lot of developers doing pretty meaningless work. A similar thing happens for managers in big orgs, who want bigger headcounts to inflate their power in the org. You get thumb twiddling and initiatives to justify the headcount.
It's probably changing now as VC cash floods into AI instead of web app startups, but I am not talking out of my ass, it's a well known phenomenon.
The industry does not incentivise or reward lean software, the software input for a lot of "software" companies doesn't require it, that's fine. Exactly as you said they're businesses, they do what makes money and go where incentives take them. That cuts both ways, they are not disincentived to have a lot of wasted dev hours, at least not historically.
I think it's more about money and scaling, bus factor is pretty big if run very lean organization eg whatsapp 2014 and that point all those developers are kinda your cofounders and probably start asking much bigger piece of pie. With small teams you kinda trade scaling and availability to velocity. It's much easier to ship but running oncall 24/7 with small team is just nightmare.
well people are spending tokens like crazy for sure, Anthropic and all other labs and everything related to AI seem to be making tons of money.
On a serious note, it might take a while to realize the actual benefits or losses. Its clearly not a good signal when people whose job is to manage other people start writing their own pet AI projects. This just indicates that AI has created this big job insecurity among everyone. At the end of the day being an engineer, eventually the job is relatively safer when there is so much code being written out there.
Similar to what others have said, we have had big changes in our systems, thanks to LLMs. Some examples of what we have done:
- Rewrote a data extraction and PDF bounding box algorithm - LLM provided the tooling to visualize the output of algorithm and find the right rules for our needs.
- Migrated an old SciBERT model that was bundled into a 6GB docker container that needed GPU, with to an ONNX based inference container about 900MB running on CPU.
- Setup a K3S cluster to replace our Nomad cluster.
- Optimized an algorithm that used SQLite with better indexes and optimized querying with some 60-80% performance gains.
We have reduce resource usage in specific areas of the application drastically. These were all possible before, but LLMs provided the tooling to iterate and deliver it in time and cost that seemed prohibitive before.
Now are the end users going to see the benefit and is the product magically better? Well no. The chicken answer is I am not involved in that side to know. But a more realistic answer is, user experience and product fit is not something LLMs can solve. That's still upto the humans to figure out and I think that's where this "nothing has improved" feeling comes from.
AI makes a lot of drudgery type coding tasks a lot easier (think code migrations, etc.)
But actually coming up with, testing, and rolling out loved new user features. AI isn't that good at that and we can't really "prompt" that out of it as easily as we can prompt a py2 to py3 conversion of old code
Is everyone just running full speed in circles or something?
Yes. The lack of actual tangible results is how you know that the claims of increased productivity are false. We haven't seen a bunch of new useful apps (or anything else for that matter), which we would have if LLMs actually worked.
Ours is moving faster than ever in terms of feature delivery, and we've used AI to really hammer at the security aspects and clear a load of backlog stuff.
I wanted to review 600+ YouTube streams and made a custom Tinder-like interface in less than a minute with Antigravity. I would've just slogged through using a spreadsheet before.
Using the custom interface instead save a lot of time.
AI doesn't fix bad ideas. As computer industry has become the cash cow for erstwhile bankers and stockmarket brokers, expect decline of general software and rise of a small niche (zig is a recent example but also suckless / plan 9 crowd, gentoo linux, some parts of FP / PLT crowd) which try to preserve their own little world despite economic incentives to do otherwise.
I agree, look at the top token maxxer: Microsoft, Facebook, Twitter, are there observable improvement ? Like the performance of Windows is still...not ideal, every Windows update still has some issues. If AI is indeed 10x, shouldn't we(end user) see at least 2x improvement ?
Which OS is this on? I have a Windows 10 machine I keep alive for development and actual weekly use - is this issue a Windows 11 thing? I don't use the Windows 11 machine I have at all.
Yes, this is Windows 11 (forced by company). I believe the first right-click is blocked by loading something, though I'm not sure what, as after opening the menu it shows "Loading..." in place of "Open With Terminal" for a second. Subsequent right-clicks are ~1000ms and sometimes 2000ms to open. Probably also varies by equipment, as my company machine is loaded with a lot of bloatware, though it's not a weak machine by any means.
If you believe the explosion of high quality 1-3 people indie games on Steam has nothing to do with AI I don't know what to tell you. The amount of slop has increased 20x but the amount of good quality ones has also increased 4x. And it's not that hard to distinguish them.
Personally I’m leveling up on using AI to develop a project. My speed at knocking out PRs has increased a lot, but much of that is massaging the process of using AI, including managing CI bottlenecks and costs. I’ve never had to deal with 15+ PRs being developed in parallel before. It’s interesting work, but there’s a lot of doing stuff to get to the stuff I want to do.
Examples of things I’ve tackled this week:
- GitHub Actions costs exploding and looking into self hosting options (gonna try a box in Hetnzer)
- figuring out how to handle unattended builds
- tinkering a lot of email, from how to manage domain reputation and worrying about bots spamming from my server because they started filling out forms
- figuring out how to manage AI development from my phone
- setting up a process to prevent unintended destructive database migrations
- finding a way to know have to babysit the AI but approving a constant stream of questions and permission requests
- all the random little issues that the AI files to fix an endless amount of little things
- finding the right model(s) to use to most efficiently make use of the limits of my subscription
The blind community is benefiting enormously from coding agents. Game accessibility mods for everything under the sun (the big names in the last month or two are Civ V and Witcher, although there's plenty more), accessible 3rd party clients for annoying sites, people's favorite speech synthesizers ported to platforms they never ran on natively (or just straight down turned into portable C), plenty of small but nifty utilities and apps.
This is because that community's needs are amenable to what AI can do. Accessibility work (on somebody else's product) is a lot of demotivating and extremely difficult reverse engineering drudge work with a verifiable success criterion, and this is what AI excels at. Large-scale software dev is all about judgment and taste, and here, AI is not doing so well.
You see the same things with mathematics versus medicine. In math, the bottleneck is basically human attention, formal proofs in Lean are, again, drudge work with verifiable success. In medicine, the bottleneck is patients, paperwork and the lab environment, so even an omniscient LLM without the ability to pour fluid into a beaker wouldn't be that much of a productivity improvement.
Perhaps it is because the last 20 years has seen a shift in developer culture away from lower-level understanding and native applications so that all the vibe-coded apps you use on the desktop are Chromium-based web "applications".
If you apply AI to low-level applications, it is quite good and you avoid the runtime bottleneck of webness, I have found. Eg. https://github.com/Redrum624/Vitrine is very good but has a horrible UI due to the webby nature of its rendering - all vibe coded. If you used a native toolkit it'd feel completely different.
So, my 2 cents: we are mostly using AI to analyze and fix things in our codebase that were hard to fix manually. Rare race conditions. Extra unit tests. Things that just didn't pass the effort+cost/gain ratio before.
The quality definitely improved, but our products were usable before, only not perfect. The improvement is mostly on the margin. Instead of crashing twice a week, the app crashes twice a month. A communication session won't fail in two hours, but maybe in two days. (But the average session in the real world was < 20 minutes anyway.) Some UI elements that were not necessary, but are nice to have.
I built a debian based OS focused on self-hosting: github com/onmoose/os
It's not really Linux size but I couldn't have built this if it wasn't for coding agents.
Still a ton of work needed on UX but the main functionality works so well.
Linux also has decades of work on it, we can't really expect new products to equal that in 2 years of agentic coding
deploy and rollback, which do not scale the same way
Why not? A good chunk of software engineering problems can be solved if we re-structured our definitions. Even just a year ago, people did not believe that a good chunk of prod-running code would be completely AI generated. Now we're at the point where some companies try to split "what code has to be PR-reviewed, and what doesn't have to be" to remove that bottleneck and so on.
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
We are working on something here, pushgate.dev. It forces the agent to run tests; we are also working on adding support for static analysis of bundles using Jev. Onboarding is a bit of a mess right now, but send me a message cole@testifysec.com if you are interested.
My biggest takeaway (as a one man band 20 leagues below this spending time optimizing Gitlab free plan and self hosted runners): they did not worry about any of this until 100M ARR and more 1B valuation[0].
With the widespread increase in speed/commit cadence/content added to repos, I wonder if there's an angle here for companies with a lot of CI/CD needs to simply start self-hosting their CI/CD machines on premises.
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
Despite our test suites almost quadrupling since the start of the year
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
I suspect a substantial part of this is an avalanche of useless testing.
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
I actually look at the tests first. But for this to work, you have to have good hygiene. If the test makes sense, fails without the PR but passed with the PR, I'm almost happy.
I think to solve this problem properly, we have to stop treating "build" and "test" as separate buckets of work, to be designed and scaled separately. It's all CI. As soon as you try seriously scaling out tests, you will run into build bottlenecks. To truly scale CI you need a scheduler that understands your build, test environment, and all the glue in between, well enough to schedule it in a way that actually speeds things up. That is very difficult and not something that even the best build tools can do - yes, even Bazel. Bazel can run tests but it's not nearly as good at it than at building.
Yes. Bazel is half of the equation. The other half which other enterprises rely on is the server side of Bazel's Remote Build protocol. There, the scheduler can be implemented with logic that accounts for sandbox sizing as well as other requirements/constraints.
Our CI has thousands of tests and is done in 2 minutes. Everytime it grows above 3 I add more parallelization and keep it ultra fast. Been working great for our team
How are you speeding up the provisioning? Most of my tests run pretty quick, but spinning up a new runner takes probably most of the time. I haven’t tried optimizing this yet.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
Unknowable. How much did a bug that was caught by those tests, and did not reach production save the company? What if there were many such bugs? Or none? How much extra time was lost to developers waiting around for tests to run?
The next bottleneck is customers. There better be a lot of them, they better have loads of money, and they better be willing to install updates for every fucking release that goes out whether they want it or not because how else are we going to pay for all this?
It's strange that they did not mention reworking the tests themselves. In my experience AI-generated tests are absolute garbage - unless you specifically prompt to reason about coverage to avoid duplication and to merge some.
Self-hosted is the way. Small potatoes in comparison, but I also made the change this year as my CI usage went from <3k mins/$0/mo -> $100+/mo.
Migrated actions to a spare M2 MacBook Air at home, and when offline, uses hosted Actions as a backup. Bonus: has simulators/emulators for ease of running Maestro tests for mobile QA.
Another approach is to redesign CI pipelines. Most checks should be done within the agent loop for quicker feedback (using hooks, skills). Linting, unit tests etc.
CI should handle only stuff which cannot be run locally due to resource or setup reasons.
Also CI should only receive pre-validated change candidates with passed checks inside the agent loop.
Because the third-party runners sit outside GitHub’s network, they rely on a direct IP link to reach GitHub. The provider traced the hangs to intermittent degradation on that link.
This is interesting. If I read that correctly this means the CI host had a direct peering connection with Github or Azure network.
Why were they not able solve the issue?
the blog mentions that they use a local cache to "reduce the time" but the problem is not solved right?
If you can pay the setup cost, Bazel will get you build times ~10s with a warm cache for even massive projects.
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
Were any of those using JS/TS? I've seen Bazel perform wonders with many compiled languages, but I'm not impressed with the JS ecosystem.
Nothing substantial, no. I've never personally experienced builds to be so slow to begin investigating bazel as an option for JS/TS.
Tsgo, oxlint, caching dependencies etc. what linear outlined in their blog post would be more impactful for the average TS project I've worked on.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Bazel can even skip the final artefact!
well for test runs you kinda need the binary to run, but you're correct, if the job is just "does this build" no download necessary.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
We solve it 2 ways in the Bazel ecosystem: for the intermediate artifacts, we only fetch the digest (hash + size) of the blobs to calculate the merkle tree forward. The blob itself can stay on the remote cache server.
For the bigger final artifacts, we support using Content Defined Chunking (rolling gear hashing) to only fetch the missing chunks between incremental builds. Binaries executable with stable layout benefits from this quite a lot.
We are definitely not done with all of the improvements here. But since all the major AI labs are using Bazel, we know that the tools can support “Agent Scale”. https://webazel.dev/
Not sure how that would work with building say a docker image, reproducible builds are pretty hard problem to solve, and caching intermediate layers is not always simple or even doable, we typically still need to publish to a registry which is not the cache server.
The Bazel ecosystem builds container images not by using Dockerfile, which contains non-reproducible primitives such as RUN and others. We do it by actually constructing the file trees and tarballs manually, then using them to compose the JSON manifest and indices. This is done via smaller hermetic and reproducible Bazel actions and thus enables the ecosystem to scale way beyond what alternative BuildKit-based solutions can.
https://www.youtube.com/watch?v=biYXmAv4Ppk&t=314s should be a good talk to study up on the matter. The speaker is now working at Apple.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Most PRs only touch a handful of targets so cache hits are extremely high in practice.
Depends on which targets and how granular the caching is. - if you touch package.json that might invalidate everything - touch core and you'll invalidate everything - touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
What i have seen on my end is that it’s pretty easy to setup a cloud coding agent with a warm Bazel cache. “Warm” here can means multiple layers: same disk to CoW/hardlink, different disk, network disk/block devices, same rack/datacenter, same AZ, etc…
And yes, there are a ton of investments going toward Bazel recently to unlock these newer use cases.
Is this really easy in the AI era? It seems like the repetitive, verifiable work that the agents should be good at.
vitest load balancing on test duration times instead of by file would be a nice performance win
We actually tried this and (for us) it doesn't lead to enough of a gain that's worth the extra complexity.
Agentic Coding has been a huge strain on CI, I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
This still surprises me.
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
CI -> Continuous Integration
Its best thought of as the testing systems that are run as part of pull request review / merge to main / build processes.
Thank you
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
CI is really only significant on the programming side of things. Networking, systems administration, hardware, etc often don’t have that workflow.
sysadmins CI is hoping the machine still boots after an update. Unless you are lucky you got two identical machines to have a test env.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I see you have not discovered infrastructure-as-code (IaC). We have CI for our infra.
I’m familiar, the issue is IaC only works on a subset of system administration jobs.
Maintaining systems in a tunnel boring machine is a very different than an internet startup.
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
What do you think about this? https://claude.ai/share/54eef6b8-b702-4f3e-968c-b19184e951f4
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
If expecting more from others is what gives someone a sense of superiority, then what is that you're doing with your expectations?
Next time, Google it or ask ChatGPT. That's what elementary kids do when they don't know something.
I mean, "CI" is a complete misnomer. There is usually no "integration" happening while a CI pipeline runs, it's mostly just running a couple of tests.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
It's wild to me that people think asking for
"Continuous Integration (CI)"
in the first line of a paragraph of a long article is "going out of your way". It used to be the standard expectation for using shorthand.
It used to be a standard expectation, but that was before google or AI could have answered your question in less time and typing that your complaint on HN required. Especially when the author is writing a blog post for an audience that knows what CI is.
Do you want to be right, or do you want to learn?
I don't think it would occur to me to write out Continuous Integration because it's basically a nonsense term if you don't know what it means anyway, it's not any less opaque than just CI. The problem isn't the shorthand here, but yeah it's probably good to spell it out anyway so when someone googles it they have more confidence they've found the right thing.
Man, what a weird thing to complain about.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
"AI is in the cultural zeitgeist"
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
I know what a CD (Compact Disk) is.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
TLA or TLA+?
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
Question here:
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
Good thing Linear has been finished for a long time doesn't need more features, so AI coding can go slow. Oh damn, it's busy becoming the next Jira :(
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
To be fair, the forced cloud migration has (unfortunately) been pretty good for performance.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
Uh, no. If stage X is the bottleneck, it's immaterial how much you speed up stage Y.
You're assuming QA reviews won't be fully automated, and triggered from a CI pipeline.
We can assume lots of things, but at some point the rubber must hit the road.
What exactly are you using to back up this claim?
QA is not automatable. QA is about finding the things that you didn't think of and so didn't write a test for it. That many people think qa is not important shows in the bad software we have.
QA is about quality. That it has been increasingly used to mean "repetitive manual testing only" is part of the very trend you decry of qa being undervalued.
Fundamentally QA is about two groups of people collaborating in an adversarial process ("you build it we break it") to achieve a common goal: a better product. QA by definition requires a relationship of equals, or the process ceases to be adversarial and becomes useless ceremony. If you're not able to assemble two distinct groups of people, one person can wear both hats (ie you test your own software) but you'll have more blind spots.
How much of QA is done by a human or automated, is and has always been an implementation detail.
If you're in charge of QA, you're in charge not just of finding new problems, but preventing regressions as well. And you're responsible for embedding as much of that work into the devloop itself, so that builders can find and remediate problems earlier and with less pressure on your limited time. How do you achieve that? automation.
I started my career doing QA and 99% of what I was doing could be automated by an AI today. Today I still do QA on my own product- I just spend all my time on the remaining 1% and my product is better as a result.
It's true because it usually makes no sense to QA test a build where the test suite has not yet passed.
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
Maybe we can also replace the customers with agentic consumers.
https://quoteinvestigator.com/2011/11/16/robots-buy-cars/
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
TLDR: it's not all bad :)
That’s what I’m doing! Results yet to be measured.
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
hasn't improved software either. it's made the overall quality worse if anything.
Which is a bit terrifying and removes the ‘engineering’ component of a critical part of the job, in exchange for slot machine / pray to ai gods…which comes with disconnect from the code, something I assume will adversely compound over time.
I feel no one talks about this, and yet it is glaringly obvious. Are we even solving the right problem? No one cares. Push code. Number go up.
That problem isn’t new though. It’s not the biz people that are enthusiastic over AI, it’s the devs.
You're so lucky to work for a company where PMs don't vibecode and tell how easy it is to do things....!
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
Amdahl’s law.
I was just discussing with my team that this exact thing maybe brings back to relevance the idea of "behaviour driven development" and I was reminded of this Cucumber/Gherkin lib & language that defines a kind of executable prose you can use to specify how the software should behave. It's an interpretable programming language but designed to be close to how a human might just write down their specs of what kind of actions and responses are expected from a software system.
The idea is to drive actual testing from this, but in this era, I think it's interesting as a way to use AI to generate tests, and to cross-check those tests with the natural language descriptions, in a bit of a cycle that helps refine the highest level definition of the software.
Once that's nailed down, the implementation is just details.. normal engineering concerns like maintainability etc notwithstanding of course, but you can trust more and more the AI agents to get it right. The design specs being natural enough for humans to deal with but interpretable/specific enough to actually generate tests is pretty interesting for the bottleneck you are talking about, I think.
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I wonder how much faster GitHub actions could be if they weren’t running on Azure. Because Azure is either slow or very expensive.
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
It's not Azure as much as self-hosted runners are basement bin servers on clearly massively oversubscribed machines.
to be fair all CI providers I've had the pleasure of working with have gnarly performance profiles for the boxes they provide.
I don't think it's out of malice, but I do feel uncomfortable with the fact that the people who sell me the CI coordination software also sell me the minutes for the boxes that run the CI software. There's _some_ alignment of interests but not as much as I would want!
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
I wonder why larger companies don't use self hosted github runners. You can buy a pretty beefy machine (TB of RAM, 256 cores, fast NVMe disk) and tests will run faster than on any hosted platform. Plus you don't have to shard as aggressively because more fits into one machine, benefit of shared page cache, shared persistent disk, can easily preserve working directory (for example my pnpm install takes 0 seconds, because the node_modules folder is already present from the previous run).
Because they're running away from on-premise and the associated Ops teams.
Microsoft is pushing so hard to get companies far away from the on-premises world. Once they're in the Cloud, there's too much vendor lock-in for big enterprises to go away.
It's sad that even so many greenfield businesses still default to it. There is no longer a need to. The barrier to alternatives used to be technical, now that's gone. 99.9% of web apps' CI needs are solved by a $10/month box.
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Microservices are still bad. You want either a modular monolith or FaaS with most of the work in a domain library.
None of them are bad. It's like saying a bike vs a scooter vs a car is bad. Use them correctly and for what they're intended and they work fine. Use them improperly for the wrong things and suddenly people think the tools suck, when it's the humans who misused them that suck.
If we lived in a world where people or agents could define those service boundaries up front, correctly, with some reasonable foresight for change on the horizon ...then sure...I'd agree.
However, our world is not that world. Your agents may be happy in their tiny walled kingdom of toil and ineffectiveness, but you and your users will not. Poorly drawn service boundaries will drag you down more than almost any other architectural mistake.
If there is one universal amongst organizations, its that they love walls and silos. Be wary of putting them up ahead of time, cuz tearing them down once established is nearly impossible.
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
I call this moment "the Great CI Bottleneck of 2026". Here are my thoughts on it: https://dagger.io/blog/the-great-ci-bottleneck-of-2026/
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
Here's my constant question:
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
The increased revenue is in startups, like the recent couple YC batches.
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
I think you could make that argument, yeah.
https://au.finance.yahoo.com/quote/NET/financials/
Interesting to look at, a decent example for sure.
Is that because more people are signing up to use Cloudflare as a MITM to block the increasing avalanche of AI-generated traffic though??
It may not have to do with increased revenue, but it does have everything to do with reduced cost, especially developer's cost.
I bet every company is finding up how to level up their employees via AI, so that they can use less of them in the future.
So even without increased revenue, AI has its (mis)uses.
Personally I believe that this boost in productivity will not necessarily lead to greater revenue.
All the companies have the same access to AI, and AI is making them all better at doing what they were doing before (writing software). So some companies that use AI really well may be able to take market share from other companies that are slow. But I think this might just lead to a more intense competition for customers.
Kinda like what happened to music after digital recording became the thing - we have more music than before and the music is better, but being a musician became a much more intense competition to find an audience.
probably a naive question but then would we expect companies not using it to lose market share to competitors that are using it?
Well I don't really know what will happen in the future. My first thought is that what will happen depends on the company and industry. I can think of some non tech companies where AI might not matter for a while, like a car wash or a lumber mill. But long term likely yes for tech companies and companies where tech makes a difference.
People won't want to hear this, but for FAANG things are probably similar. I'm not very convinced that Meta, for example, is actually creating new revenue. It's just consolidating a lot of the existing global ad spend, it's just wrecked newspaper classifieds and the like.
If every software company saw a roughly equal improvement in productivity, they'd still be splitting the same customer base amongst each other - not clear whether revenue would actually go up for anyone.
Perhaps some new markets could be entered that weren't feasible before?
But there have been trillions invested in infra for AI. Surely those investors are going to need to see a return at some point?
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
How is dev satisfaction?
doesn't really matter now, does it? Someone's gonna do the job, whatever it looks like.
A luxury of the past, for >90% of devs?
I want to add to this, because it's certainly not all doom and gloom for me, yet.
If you are product-driven, life is pretty good, and maybe has never been better? (For the moment)
It has certainly been better for junior devs, mid devs, anyone looking for zirp era salary growth and supply demand ratios. For a senior code-adverse coaster looking forward to see his static salary get eaten by inflation, life has indeed never been better.
Yeah, at Corp, that makes sense. I was thinking more along the lines of scrappy startup silliness. "I am one person and I deal with any and all code to get my product built."
From that POV, has it ever been better, assuming you can rise to the top of an ever more crowded market?
Building a product is easier now, same way making a photograph is easier now. Making a reliable and high income living out of it though, it looks increasingly uncertain.
In a world with abundant music and photography, just saying that you made your music using Reason/ProTools or you made your photograph with a Leica isn't enough. I believe the same will happen with software. Sales, product design, customer service and operational aspects like uptime will matter more than the software itself. Ironically, this is the kind of stuff the HN type highly dislikes.
If you are product-driven why not go into Product Ownership roles? It certainly will be more satisfying compared to prompt "engineering" clerk "career".
Well, my life is currently both, as I am on the smallest possible tech and product team. To me, extrapolating on having done this since Sonnet 3.5 to now, there will be no tech aspect to the job soon enough. There are many hundreds of billions invested in making that a reality, and they are succeeding.
Until ~9 months ago, I spent most of my time being a "prompt engineer." Today I join a meeting, ten minutes later I receive the transcript. Then, I run a custom skill in my project, and I get Jira epics and stories to triage that are nearly perfect. Then, I run the second skill orchestration skill... some babysitting... and ~85% of the time that is all I need to do. Docs, code, unit and e2e, great UX... all there after every meeting, and basically two commands on my part.
I see two to three to maybe five years before anything I have to offer, in any capacity, is completely cut out of the picture.
I honestly don't understand how everyone is not on this same page. The labs are going to eat it all. The only reason I see for them to talk about "pausing," is because they finally realized that they are going to collapse the entire service economy around themselves at this rate: aka, the USA.
Not sure how long the current subsidized pricing would last. Also, the vast majority of software shops can't afford those subsidized prices anyway.
Which devs are you asking about, the humans or the AI agents?
Start a few dozen agents, and take a long lunch, right? No?
I assume they are very satisfied with their paycheck.
No better:
If we cared about the satisfaction of the worker, we’d never have fast food.
This is the way!
Cost is stable (or rising?), user-facing delivery is sameish, (some) engineers seem happier because they are allowed to gold-plate and prepare for future “human development of the hard-things”.
Is that a fair summary?
Not op but no.
Cost is stable, what users see is "the same".
Engineers are happier because they can spend more time on that extra round of polish that was just filed into low-priority "fix when time" Linear tickets.
QA is happier because the stupid bulk operations are handled with AI and they can focus on the hard to find stuff that tickles the QA mind in a special way =)
Same stuff, but with better (in-house) tooling and (most) people are happier. The Code Artesans aren't, but it's mostly their problem and they'll find companies that only do fully manual programming for specific niches.
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For the question where are the alternative OSes? Here is one that I've seen. There's probably more - https://www.reddit.com/r/ClaudeAI/comments/1wfpydl/i_asked_c...
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
So let’s stop making new apps then?? What is your point? Looks like you’re just complaining for the sake of it.
Has economic growth accelerated? TFP, labor productivity?
Where is the net benefit people keep talking about when they claim productivity gains are obvious because "more code faster, can't you see?"
more code faster is competing against some of the largest tax hikes of the last ~70 years, economic uncertainty on what the taxes tomorrow will look like, combined with an energy crisis.
Are you claiming productivity should be growing much more slowly then it is over the past few months or years? Can you show me any analysis that supports that claim?
Because I have seen plenty of analysis published in the past 30 years that puzzle over the productivity growth stagnation in the face of an exponential growth in capital expenditure in computer technology.
So by all means share with me some of the groundbreaking results showing that we can finally see more productivity growth than would be otherwise expected by the conjecture.
And can you show analysis supporting your claim? Sounds like some kind of piketty bullshit.
There's a difference between task-level productivity, which AI has been shown to increase, and effects on the macro. You're talking about the latter while most people here are engineers considering the former.
It's still unclear if task-level productivity gains bubble up, but it's also still early and I'm not sure we should expect to see immediate results there.
Once the previous goalpost gets demolished on HN, there's always someone new to put a new one. We aren't quite there yet but eventually the goalpost moves to something that is just impossible to even measure. That'll be the end game surely
Was the goal of any investment ever not 'increase productivity'?
It’s interesting that people claimed the exact same thing when computers were being introduced to the work place int the 80s and 90s. Lots of papers showing how productivity didn’t go up at all and using a paper and pen seems to be just as efficient as using a PC.
Even in the 19th century, when electricity became widely available, there was no productivity gain for 30 years at least. This is a well understood phenomenon.
Google for “the Solow Paradox”.
Labor productivity, even at a national level, possibly yes:
https://www.stlouisfed.org/on-the-economy/2025/nov/state-gen...
This is just ~2 - 4 years after ChatGPT launched, and despite very shallow adoption (only ~6% of all work hours.) As a sibling comment indicates, it took almost 2 decades for the Computer Revolution to be visible in national level statistics.
Also note this study was originally published in 2024, then revised in 2025, but this preliminary evidence has been around for a while, if people wanted to find it. It's even been posted to HN a couple of times, somehow it just doesn't get the attention you'd think it should get, even if it was just to poke holes in the conclusions.
These long term data suggest that this "excess" remains bellow historical productivity growth
https://www.bls.gov/productivity/
https://www.bls.gov/productivity/images/pfei.png
App store revenue actually decreased! That suggests that the apps are largely not useful, unless you make the implausible assumption that nearly none of them are charging money.
https://www.ft.com/content/e5533f32-e4c2-4ce1-9d2e-a831e9654...
I mean in a market where production cost goes down & competition increases while demand stays the same... That's exactly what I would expect.
If they were actually useful though, demand shouldn’t stay the same — more niches should be getting filled. If they’re just clones of each other, then yes
i mean, i had multiple gaps filled in the last few month for things i was longing for.. but all for free.
If.
The labour of finding better goods has become harder with more content showing up.
I don’t think this is happening, at least not on any scale that makes any of this actually useful. One off success stories do not make a revolution.
It absolutely has generated a massive amount of slop, though. The problem is that you still need intent and while I think LLMs can help you free up and push through annoying boilerplate or tedious spots where known solutions exist, you still need to design these things and that’s something I’ve not seen an LLM be too helpful with.
In the gaming space, retro console PC ports have absolutely exploded in the last year. People are modding old games with 4k textures, ray tracing, DLSS 5, widescreen support, uncapped framerates, dual-screen (e.g. Ayn Thor), etc. PS5 emulation has gone from almost nothing to AAA titles in-game in like 2 months, and apparently now runs on xbox (lol).
The other day someone posted a reverse engineered GPU driver for their Mac[0].
Anyway, what makes you think those pushes aren't individuals meeting their own needs? I expect it will take a while for this to really sink in, but the future is people asking the computer to build exactly the app they want.
[0] https://news.ycombinator.com/item?id=49717638
Also, English localizations of even super-obscure foreign retro games.
Most of those might be not as good as fan translations from dedicated people with good knowledge of English and the original language (especially for very context-dependent languages like Japanese) but they are definitely good enough to follow the story and play through the game.
Has anyone said that AI is fixing user research, actual idea generation, marketing and audience reach? Or are you assuming that better code should be making all of that irrelevant?
It either delivers better outcomes or fails to do so for not being as general and useful as touted.
The original sales pitch from the major player in this technology hype cycle is "cure cancer, fix global warming, take over the economy". Not "more LOC".
AGI any second now
whats the large hits that have happened in the past couple years?
I think ai has raised the floor quite heavily on what it takes to be a large hit.
all those vibecoded pushes are people building bits and bobs and variations on each other, instead of buying it or using a common service. If it could be done by ai, it will be written off as unimportant.
the userbase of the future is 1, maybe 10
Not OP, but I ported a legacy .NET WebForms application to Blazor. The actual code migration was completed over roughly a 48-hour period, followed by fairly extensive testing.
We were fortunate to already have a strong end-to-end test suite written in Python, so we could run the new application against the same tests and verify that the existing functionality was preserved. QA found around 20 bugs, which we fixed pretty quickly before launching.
After the migration, we also moved the application's authentication from Shibboleth to Entra OAuth and deployed it to our OpenShift infrastructure. We couldn't do that with the old WebForms application because our cluster doesn't have Windows worker nodes, so the legacy app had been stuck running on VMs. Getting it onto OpenShift gave us another operational and cost-saving benefit beyond simply modernizing the codebase.
I did this back in January, when the models finally became capable enough for this kind of work. I believe I used GPT-5.2 through Codex. Successfully completing this project is what finally got me fully on the AI bandwagon. I had used AI before, but mostly for smaller tasks like writing code, refactoring individual methods, or making isolated changes.
The application is now modernized, more stable, faster, and more functional than it was before.
I think the project worked as well as it did for two important reasons. First, I had deep domain expertise in both the application and its surrounding systems because I was the original developer. Second, our QA team's test suite was comprehensive enough to validate the functionality that actually mattered. AI dramatically accelerated the work, but there still needed to be someone who understood what the application was supposed to do and a reliable way to verify that the new implementation behaved correctly.
None of this means we couldn't have done the migration without AI. We absolutely could have. The difference is that AI made it possible to do it at practically zero cost in terms of engineering time and money compared with the alternatives.
We had wanted to move away from WebForms for years, but with the size of our team and the constant stream of new feature requests, there was never a realistic opportunity to stop development for months and focus on a rewrite. I estimate that doing the migration myself without AI would have taken at least six months to do properly. The other option would have been hiring a contractor, which I would estimate at $80,000 or more over six to nine months.
Based on my token usage, those roughly 48 hours of working with Codex to port the application cost about $150. Those were January prices, so I don't know what the equivalent cost would be today.
For me, that was the project that changed AI from something useful for assisting with individual coding tasks into something I saw as capable of fundamentally changing the economics of software engineering work.
People don't know what they want.
OK but you talked about people going fast. 84% YOY increase in volume shows people are going a lot faster.
You can move the goalposts to "yeah but it's not any good" but 90% of everything is crap anyway, and that was true long before AI.
I'm running several. They're not "large hits" but they're providing value to thousands of people. They couldn't exist pre-AI because core functionality depends on LLMs.
What you're missing is that slop has increased 20x but quality new releases have also increased 5x. Unless there's something in your niche though you'll mostly see the 80% slop so I get why you feel that way
Any notable breakthroughs in the top 100 apps on the app store that could attribute their success to AI?
People mean where are the good new apps AI made possible. Of course it has a 84% year-over-year slop app submission.
Any real "killer apps" though?
If the cost to produce goes down, unless the demand is elastic, you'd expect fewer big hits in favour of a more fragmented, more competitive space all earning less.
That's exactly the opposite of what I want to see. As an app store user, I want to see fewer, higher quality apps, not more shovelware or slop.
Your prejudice is showing.
The OS one is funny to me, it's been hard to keep osdev.org online due to the insane amount AI bot traffic.
Thank you for your site dude! I had so much fun and learned so much building my own toy OS over a decade ago.
osdev is golden, truly. hope you survive the slopocalypse without too much difficulty
Last time I looked the other app stores didn’t show anywhere near that much of a bump (that was a several months ago though).
And a few months ago I went through a random sample of new apps on the App Store and the vast majority were just wrappers for AI APIs. So it was more of a new gold rush situation than a productivity bump.
I have this theory that the economics of AI development would make it so it's much more viable to make in-house, custom, apps rather than paying subscriptions for apps. I have even seen this play out on a personal level. A friend of mine who has no dev experience vibe-coded something custom that evite does. So it's possible measuring the number of apps miscounts ones that are not public or shared. I'm not negating anything you said, just adding something about metrics on apps.
Maybe one day but right now in order to get any meaningfully usable app with AIs you need to pay way more in AI sub than monthly sub of a bunch of apps. It doesn’t make sense right now to build especially your own if you’re not a company. Also it’s not clear how long we’ll have this heavily subsidized subscriptions.
At the 60 person smb (travel industry) I work we've taken web development in house for the first time, created new user experiences that sat on our roadmap for years, created custom AI solutions on top of our Fresh desk tooling, and (predictably) created a number of internal productivity tools.
The actual bottom line result is very fuzzy. Cost are shifting, by spending less on agencies but in return opening up new positions or shuffling people internally.
We don't see a clear increase in revenue, nor can we accurately tie retaining revenue to our AI initiatives.
Workplace satisfaction scores remain similar, with a few outliers (e.g. some people's work got incredibly exciting, while a few others are terrified by all the change).
Time will tell if this all leads to anything meaningful for the business.
And that's just the people operating in public instead of private :P
We know it's easy to produce working code now, what's missing is useful software that actually solves a problem. What problem does that toy OS solve? The main point of toy OSes has been education, but you learn nothing when using an LLM. So all of this is just pointless energy consumption, it's not solving any real problems that people have.
App submissions are up but what about revenue?
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
It’s hard to imagine a company more all in on ai than Google. The pressure to use it is intense and they are throwing ai at basically every problem. If a proposal doesn’t have ai in the title somewhere it has no chance of funding.
I don’t really think it has made Google products better, but it is absolutely the expectation.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
Because we haven't.
We 100% have. I've seen significant improvements in many of the services I use.
Drop a couple names, I'd like to know who is leveraging AI well
Not doubting you, care to share what those improvements were?
All of the work work at my company has been internal facing. People are rightly leery of exposing things made by an LLM to the public.
I've been noticing much faster UX changes in the Youtube for Android App.
I don't know if it's AI or not, and I haven't liked all the changes, but there have been more of them than I remember before.
The new iOS has actually increased the quality for the first time in years. There are open source projects now that tackle problems nobody has before.
It’s not night and day (as you’d expect from the hype), but I see some differences.
In what ways? When did previous releases stop meeting your criteria?
Such as?
Ages ago. It was really noticeable when they prioritized features over quality.
HN is full of them. I don’t have great have specific example at the top of my head, but lots of hard reverse engineering/low level projects credit AI and that they would never have happened without it these days.
I mean, looking at the release notes for things like say the Linux kernel it is clear that the improvements are there at a massive rate of change. But it's small stuff, it's performance improvements, it's quality of life. If you do things right people will think you've done nothing at all. What did you expect? Businesses to come up with whole new business lines, or is it just that backlogs of work is getting burned down that was going to be gotten to eventually? From what I can tell from the PM forums, the next bottleneck is the Product Org coming up with ideas worth implementing as fast as the developers can deliver them.
For a $2tn industry, YES.
The simplest explanation is that they don't give a flying flamingo about what you or I consider "improvements to products".
This report is an example.
There are several changes that modify CI behaviour, where the article gives no corresponding quality measurement.
They replaced type aware custom lint rules with AST-only static analysis. They don't say anything about what those new rules detect, didn't do old-vs-new rule comparison. They switched the TypeScript check from tsc to tsgo. Again, they are very proud of the performance improvement, but don't seem to care about diagnostic equivalence. The list goes on. They don't even report pass/fail agreement between the old and the new CI. They have 4x more tests, but no idea whether this big test suite works any better than the smaller old one, or even whether it works at all.
Are they really adding 2,000 tests a week to their codebase?
Anecdotally, codex is very fond of checking strings are equal between UI and test.
They said all their tests are written by agents so probably, yeah.
Why not? If you let the LLM run amok, you get 4 one line functions calling each other instead of one 4 line function. Run that for 24 hours and you'll get more than 2000 tests.
What is Switch 2 security doing here? (independently of it having so few features added per update)
My guess they mean some kind of jailbreaking.
I've been bootstrapping for two years now, just me and AI tooling.
Code moves much faster, product taste doesn't. Once you have your marching orders you can make features 10x faster. Shots off target still miss though. It doesn't matter if your code gets generated 10x faster when it doesn't resonate with users.
"This" is lazy but yes, this. Especially in the app space, 90% of the stuff coming out has barely any "do people who aren't me actually need this, does this deliver something new and useful" thought put into it. It's a skill most people lack, they can't judge it.
Bootstrapping a solo software business has multiple core required skills. Technical skills are now largely unnecessary, but it turns out most people lack multiple of the other required skills too.
I've definitely noticing buggier and buggier software, that's for sure. Even our internal tooling and CI pipelines and all that kinda stuff has taken nosedive thanks to the deluge of garbage
PS5 emulation has gone from barely working to running Dark Souls at 10+ FPS with virtually no graphical glitches… in 6 weeks. If you’re not familiar with normal emulator development time, this is… quite extraordinary. There have been insane progress on decompilations and many other things in the emulator space.
Everyone’s trying to figure out how to convert this speed to product features at scale, but enterprises are like container ships. Lots of might but slow to turn. The littler companies can actually take advantage of this and produce higher quality products at much faster speed. I think you’re expecting too much in the short term and too little in the long term. AI-native companies are gonna eat everyone’s lunch, once they figure out how to actually do it reliably.
To further this point, I’ve picked up the torch and got Switch2 controllers working in Dolphin and Cemu
Wind Waker in 4K w/official Bluetooth GameCube controller
(Nintendo went out of their way to invent a new protocol so that it didn’t “just work” like OG switch controllers)
Also, in 12 months, we went from seeing game dissassembly and decompilations projects be slow and annoying, to suddenly having enough mapower where people are taking the liberty to CHOOSE which decompilations to support. DK64's release was 80% anti-AI marketing, not because they were strictly anti-AI, but because they were telling the OTHER decompilation to fuck off and learn some goddamned standards. Completely unthinkable before.
There was a video by a layperson 2 years ago where they wanted to know the mechanics of some elusive pokemon pinball spawns where there was a whole bunch of missinformation about it online. The people behind that project had the correct addresses but they were completely unlabeled and the layman had to essentially figure out and do some of the work themselves to sort it out and figure it out. Nowadays that would never happen. An LLM will just do it for you within the day.
The fact that this is an account with a username specifically focused on this specific concern that was created 5 hours ago is real weird/suspicious.
Also, are these even actual decompiles? Sure, playing Wheel of Fortune with a LLM model that at some undetermined point spits out what seems to be a working example is sorta useful, but also…how exactly do you debug this nonsense? Don’t tell me we’ll just have the LLM thrash around a bunch again. What I’m asking is does anyone understand what is going on here other than “lol, it works, shut up”?
LLMs work great at decompilation, I'm not sure what you are trying to suggest. Outside of game compilation disassembly is useful for debugging various crashes in 3p code where you don't have the source. And yes LLMs do the debugging here.
I’m suggesting that having a model wander around stochastically is not necessarily a great way to gain any insight or understanding. LLMs do not think, they cannot really debug anything. Attaching intelligence or intent to them is a non sequitur, it’s not happening. A cloud that looks like a toaster does not imply anything beyond the human obsession with pattern matching, why do we keep doing this with LLMs? They can randomly wander around and sometimes produce useful output.
I don’t doubt they help by randomly permuting around in a way that people cannot do at scale, but I question the economics of this approach and I think it needs to be used judiciously. Sometimes you just have to do that especially like you said, when you do not have the source or the code is intentionally obfuscated, this can be useful, but a lot of people seem like they’re just vibe coding crap and blowing money on tokens until they get a useful result and I have no idea what the value of that is.
You must be new here. Lurkers posting under an apt username then disappearing is a feature, not a bug. He doesn't need to argue with anyone online - he said his piece and we're free to take it or leave it.
Usually, green username are a sign of "leave it", I'll agree. But this one seems very informative.
Couldn't you say this about any company? What isn't just a matter of "figuring it out"?
Why can't they use AI to crack this nut? :D
Emulators have historically been a shitshow until someone figures out missing pieces here and there. By their nature, that tide lifts all ships. It has very little to do with using AI. I'd argue the real reason emulators aren't what they used to be is the hardware is more complex now and the scene is no longer attracting the most talented devs. Video games used to be on the cutting edge and carried a lot more cultural weight. It's pretty underwhelming to get cred for working on lame x86 hardware that plays yet another franchise reboot.
I did the same thing with a old multiplayer game called Wulfram, we got it online and a mostly working server (most of the game logic was server side with no surviving code or binary) with the help of AI, Mostly opus and astra. https://wulfram3.com is our efforts in this.
The problem, as ever, is figuring what's worth doing and what's not. LLMs are not as useful there.
The real question is why are people too lazy to spin up their own phorge. It's a form of learned helplessness supported by a supposed business case to prefer buying services rather than bootstrapping your organization with proven technology.
Do you mean "forge"? As in Git forge, like Github?
incentives are not aligned to improving products.
the important thing to microsoft is still finding new ways to make money, not to make your clicks faster. they might even be using their new velocity to make clicks slower with an ad in between.
there's alao much less incentive to build things to share. people can just make their own widget app and not need yours
The improvement isn't happening because company are also actively firing people in name of layoff and aren't actively hiring.
Agreed, but give it time. The tipping point of good models only just arrived with Fable 5, GLM 5.3, Grok 4.6, Muse Spark 1.3, and the like. Those are the only models that do a better job than me, and I'm happy to hang up my IDE—and that was barely a month ago.
I've been fixing everything that has been sitting there—not blocking us, but slowing us down—all the things we never had the bandwidth for. We're now squeezing more out of development, CI, and production.
I'm not sure why, but I'm concerned this may be the heyday, and we might not get this again at this price or speed. So I figured we should clear the backlog while we can still afford it and still have the ability to do so.
But now things have changed, and the bigger, grander ideas are starting to brew.
Probably for the same reason that SV companies hiring thousands of developers struggled to improve their product much past the original product, that was built by a handful of people.
Scale in headcount was a tactic to get investment, then you had to find stuff for everyone to work on. Suddenly people have the time to engineer so hard that we get runtime JSON defined CSS rendering engines to produce the same buttons we've had since 1995, instead of just writing a stylesheet and html.
Code output velocity from AI threatens to be useful, except it's also prone to over-engineering and burning tokens on the unnecessary. It's learned from the best after all. My suspicion is there is a lot getting done, but it's just not that impactful to flagship products.
As others have noted, there is a lot of new work going into passion projects that would have never happened otherwise, and that is cool. But I wouldn't hold my breath for SV tech to become super pragmatic and effective.
With pay-per-token, there’s also an incentive for over-engineered but functionally harmless architectures. A json based css engine is probably something that can be test-cased really well in the training set.
those thousands of developers aren't working on the FB feed or whatever...
they're working on the ad tech business and other business-related systems involved including internal tools.
and all the other platform engineering shit under the hood that makes much of the web scaleable... and lots of it is open source and contributed to by various engineers from these companies.
end of the day, these are businesses. they're not charities or whatever casual shit.
nobody is stopping you from building a competing product that's lean or whatever.
why don't you do something like that? i'm sure you're a genius.
No seriously, companies hired developers just to starve the competition of talent, raise more investment etc. It was a genuine thing, that left a lot of developers doing pretty meaningless work. A similar thing happens for managers in big orgs, who want bigger headcounts to inflate their power in the org. You get thumb twiddling and initiatives to justify the headcount.
It's probably changing now as VC cash floods into AI instead of web app startups, but I am not talking out of my ass, it's a well known phenomenon.
The industry does not incentivise or reward lean software, the software input for a lot of "software" companies doesn't require it, that's fine. Exactly as you said they're businesses, they do what makes money and go where incentives take them. That cuts both ways, they are not disincentived to have a lot of wasted dev hours, at least not historically.
I think it's more about money and scaling, bus factor is pretty big if run very lean organization eg whatsapp 2014 and that point all those developers are kinda your cofounders and probably start asking much bigger piece of pie. With small teams you kinda trade scaling and availability to velocity. It's much easier to ship but running oncall 24/7 with small team is just nightmare.
well people are spending tokens like crazy for sure, Anthropic and all other labs and everything related to AI seem to be making tons of money.
On a serious note, it might take a while to realize the actual benefits or losses. Its clearly not a good signal when people whose job is to manage other people start writing their own pet AI projects. This just indicates that AI has created this big job insecurity among everyone. At the end of the day being an engineer, eventually the job is relatively safer when there is so much code being written out there.
Similar to what others have said, we have had big changes in our systems, thanks to LLMs. Some examples of what we have done:
We have reduce resource usage in specific areas of the application drastically. These were all possible before, but LLMs provided the tooling to iterate and deliver it in time and cost that seemed prohibitive before.
Now are the end users going to see the benefit and is the product magically better? Well no. The chicken answer is I am not involved in that side to know. But a more realistic answer is, user experience and product fit is not something LLMs can solve. That's still upto the humans to figure out and I think that's where this "nothing has improved" feeling comes from.
AI makes a lot of drudgery type coding tasks a lot easier (think code migrations, etc.)
But actually coming up with, testing, and rolling out loved new user features. AI isn't that good at that and we can't really "prompt" that out of it as easily as we can prompt a py2 to py3 conversion of old code
AI is a rocket motor.
Doesn't matter where we're going, as long as we get there FAST!
It's a sickness.
Yes. The lack of actual tangible results is how you know that the claims of increased productivity are false. We haven't seen a bunch of new useful apps (or anything else for that matter), which we would have if LLMs actually worked.
What do you mean by llms not working? Did you use one recently?
Ours is moving faster than ever in terms of feature delivery, and we've used AI to really hammer at the security aspects and clear a load of backlog stuff.
"Why have we not seen an improvements in products?"
we literally do, we shipping feature faster than ever
I wanted to review 600+ YouTube streams and made a custom Tinder-like interface in less than a minute with Antigravity. I would've just slogged through using a spreadsheet before.
Using the custom interface instead save a lot of time.
AI doesn't fix bad ideas. As computer industry has become the cash cow for erstwhile bankers and stockmarket brokers, expect decline of general software and rise of a small niche (zig is a recent example but also suckless / plan 9 crowd, gentoo linux, some parts of FP / PLT crowd) which try to preserve their own little world despite economic incentives to do otherwise.
You can pretty much throw any PC game at Codex and get a playable VR mod with full 6dof and in most cases even motion controls
I agree, look at the top token maxxer: Microsoft, Facebook, Twitter, are there observable improvement ? Like the performance of Windows is still...not ideal, every Windows update still has some issues. If AI is indeed 10x, shouldn't we(end user) see at least 2x improvement ?
Wow. It actually does. I thought you were exaggerating.
Which OS is this on? I have a Windows 10 machine I keep alive for development and actual weekly use - is this issue a Windows 11 thing? I don't use the Windows 11 machine I have at all.
Yes, this is Windows 11 (forced by company). I believe the first right-click is blocked by loading something, though I'm not sure what, as after opening the menu it shows "Loading..." in place of "Open With Terminal" for a second. Subsequent right-clicks are ~1000ms and sometimes 2000ms to open. Probably also varies by equipment, as my company machine is loaded with a lot of bloatware, though it's not a weak machine by any means.
If you believe the explosion of high quality 1-3 people indie games on Steam has nothing to do with AI I don't know what to tell you. The amount of slop has increased 20x but the amount of good quality ones has also increased 4x. And it's not that hard to distinguish them.
Personally I’m leveling up on using AI to develop a project. My speed at knocking out PRs has increased a lot, but much of that is massaging the process of using AI, including managing CI bottlenecks and costs. I’ve never had to deal with 15+ PRs being developed in parallel before. It’s interesting work, but there’s a lot of doing stuff to get to the stuff I want to do.
Examples of things I’ve tackled this week:
- GitHub Actions costs exploding and looking into self hosting options (gonna try a box in Hetnzer)
- figuring out how to handle unattended builds
- tinkering a lot of email, from how to manage domain reputation and worrying about bots spamming from my server because they started filling out forms
- figuring out how to manage AI development from my phone
- setting up a process to prevent unintended destructive database migrations
- finding a way to know have to babysit the AI but approving a constant stream of questions and permission requests
- all the random little issues that the AI files to fix an endless amount of little things
- finding the right model(s) to use to most efficiently make use of the limits of my subscription
And so on.
It really depends on the industry.
The blind community is benefiting enormously from coding agents. Game accessibility mods for everything under the sun (the big names in the last month or two are Civ V and Witcher, although there's plenty more), accessible 3rd party clients for annoying sites, people's favorite speech synthesizers ported to platforms they never ran on natively (or just straight down turned into portable C), plenty of small but nifty utilities and apps.
This is because that community's needs are amenable to what AI can do. Accessibility work (on somebody else's product) is a lot of demotivating and extremely difficult reverse engineering drudge work with a verifiable success criterion, and this is what AI excels at. Large-scale software dev is all about judgment and taste, and here, AI is not doing so well.
You see the same things with mathematics versus medicine. In math, the bottleneck is basically human attention, formal proofs in Lean are, again, drudge work with verifiable success. In medicine, the bottleneck is patients, paperwork and the lab environment, so even an omniscient LLM without the ability to pour fluid into a beaker wouldn't be that much of a productivity improvement.
Yes, just rehashes of the same thing that was done before, in another language/framework/codebase/fork.
Perhaps it is because the last 20 years has seen a shift in developer culture away from lower-level understanding and native applications so that all the vibe-coded apps you use on the desktop are Chromium-based web "applications".
If you apply AI to low-level applications, it is quite good and you avoid the runtime bottleneck of webness, I have found. Eg. https://github.com/Redrum624/Vitrine is very good but has a horrible UI due to the webby nature of its rendering - all vibe coded. If you used a native toolkit it'd feel completely different.
So, my 2 cents: we are mostly using AI to analyze and fix things in our codebase that were hard to fix manually. Rare race conditions. Extra unit tests. Things that just didn't pass the effort+cost/gain ratio before.
The quality definitely improved, but our products were usable before, only not perfect. The improvement is mostly on the margin. Instead of crashing twice a week, the app crashes twice a month. A communication session won't fail in two hours, but maybe in two days. (But the average session in the real world was < 20 minutes anyway.) Some UI elements that were not necessary, but are nice to have.
I would guess most are working on improving their workflow/tools and keeping up with the rapid changes.
Moose has entered the room.
I built a debian based OS focused on self-hosting: github com/onmoose/os It's not really Linux size but I couldn't have built this if it wasn't for coding agents. Still a ton of work needed on UX but the main functionality works so well.
Linux also has decades of work on it, we can't really expect new products to equal that in 2 years of agentic coding
Speed up CI and the next bottleneck just moves to deploy and rollback, which do not scale the same way.
You are right, but one problem at a time :)
Why not? A good chunk of software engineering problems can be solved if we re-structured our definitions. Even just a year ago, people did not believe that a good chunk of prod-running code would be completely AI generated. Now we're at the point where some companies try to split "what code has to be PR-reviewed, and what doesn't have to be" to remove that bottleneck and so on.
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
We are working on something here, pushgate.dev. It forces the agent to run tests; we are also working on adding support for static analysis of bundles using Jev. Onboarding is a bit of a mess right now, but send me a message cole@testifysec.com if you are interested.
My biggest takeaway (as a one man band 20 leagues below this spending time optimizing Gitlab free plan and self hosted runners): they did not worry about any of this until 100M ARR and more 1B valuation[0].
[0] https://linear.app/now/sharing-growth-with-the-people-buildi...
With the widespread increase in speed/commit cadence/content added to repos, I wonder if there's an angle here for companies with a lot of CI/CD needs to simply start self-hosting their CI/CD machines on premises.
In my experience, self hosting your CI/CD runners had the biggest impact in cost savings throughout, while also allowing for more powerful machines, which directly means quicker CI/CD runs
I think the SDLC needs to be re-evaluated, tests, specifically unit tests, are just cosmetics. "wow its so easy to get high coverage now!" as opposed to why we're doing it in the first place
I don't find agents to be using tests any differently than a junior or mid level developer. I don't find humans to be using tests any differently than a junior or mid level developer either.
Basically, the tests are never guiding the features, they aren't highlighting regressions, the test is simply modified for the updated application. Unit tests especially.
I have found end to end tests to be useful, and also unblocked. They can tell when the actual user experience has changed, and go back to the feature's implementation to stop altering the user experience in unexpected ways.
additionally, I find this across every industry that was worried about AI. basically, whatever best practice was neglected due to lack of investment into your org is now being done without additional investment into your org. that's a good thing. now we need to look at the purpose of the best practice, and if that purpose itself was already solved in the process.
I suspect a substantial part of this is an avalanche of useless testing.
If you even review PRs still: when was the last time you didn't just skip over tests? And if you ever looked at tests in an LLM-heavy PR, how many of those tests tested something useful, and not just built-ins and trivial behaviours?
There's at least some awareness in the industry of how LLMs generate a lot of boilerplate in business logic. It feels like we're much less aware of how much of it is in tests.
The opposite. Reviewing PRs is now about just reviewing the tests, as the code is very likely to be fine if tests are relevant and they're passing.
The main enabler of agentic coding is heavy end-to-end/characterisation testing.
We can ask AI to do very large-scale refactors, for example, because if tests pass, it's 99% that everything is fine.
Probably half the generated tests in my app are for markdown files.
I actually look at the tests first. But for this to work, you have to have good hygiene. If the test makes sense, fails without the PR but passed with the PR, I'm almost happy.
I think to solve this problem properly, we have to stop treating "build" and "test" as separate buckets of work, to be designed and scaled separately. It's all CI. As soon as you try seriously scaling out tests, you will run into build bottlenecks. To truly scale CI you need a scheduler that understands your build, test environment, and all the glue in between, well enough to schedule it in a way that actually speeds things up. That is very difficult and not something that even the best build tools can do - yes, even Bazel. Bazel can run tests but it's not nearly as good at it than at building.
Yes. Bazel is half of the equation. The other half which other enterprises rely on is the server side of Bazel's Remote Build protocol. There, the scheduler can be implemented with logic that accounts for sandbox sizing as well as other requirements/constraints.
I gave a talk about how we do it at BuildBuddy at a recent BazelCon here: https://youtu.be/iQqLtuBzkKE?t=848
Our CI has thousands of tests and is done in 2 minutes. Everytime it grows above 3 I add more parallelization and keep it ultra fast. Been working great for our team
How are you speeding up the provisioning? Most of my tests run pretty quick, but spinning up a new runner takes probably most of the time. I haven’t tried optimizing this yet.
Heh. I fixed my CI/CD flows last year to be no more than 60 seconds: https://pics.ealex.net/share/2IpZSt2cxrKWzeUokNNmk4eRI1jjzF2...
And this includes _full_ linting and unit tests.
I did that by self-hosting Github Runners, using Podman to build images (it doesn't squirt your build context through a socket every time!), and using Docker images as content-addressable cache.
I will do a write-up about it this week...
Did the tests produce four times as much value, though?
Unknowable. How much did a bug that was caught by those tests, and did not reach production save the company? What if there were many such bugs? Or none? How much extra time was lost to developers waiting around for tests to run?
The next bottleneck is customers. There better be a lot of them, they better have loads of money, and they better be willing to install updates for every fucking release that goes out whether they want it or not because how else are we going to pay for all this?
It's strange that they did not mention reworking the tests themselves. In my experience AI-generated tests are absolute garbage - unless you specifically prompt to reason about coverage to avoid duplication and to merge some.
Self-hosted is the way. Small potatoes in comparison, but I also made the change this year as my CI usage went from <3k mins/$0/mo -> $100+/mo.
Migrated actions to a spare M2 MacBook Air at home, and when offline, uses hosted Actions as a backup. Bonus: has simulators/emulators for ease of running Maestro tests for mobile QA.
Another approach is to redesign CI pipelines. Most checks should be done within the agent loop for quicker feedback (using hooks, skills). Linting, unit tests etc.
CI should handle only stuff which cannot be run locally due to resource or setup reasons.
Also CI should only receive pre-validated change candidates with passed checks inside the agent loop.
This is interesting. If I read that correctly this means the CI host had a direct peering connection with Github or Azure network.
Why were they not able solve the issue?
the blog mentions that they use a local cache to "reduce the time" but the problem is not solved right?
Unless this impacts net income, it's just theater.
Does anyone do approaches where developers run CI locally on their machine and then somehow push up signed or authenticated artifacts?