My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircrafts).
It's simple: when a mistake happens, you run you CAPA process and determine the root cause and the correction to the process that allowed the mistake to happen in the first place. This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
Anyway, here's mine, still wip:
https://hale-lang.org/docs/dna/
https://github.com/hale-lang/hale/issues/690
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
fine-tuning may be a more scalable approach to LLM personalization than sending all the same context to two LLMs
I'm working towards both in my homelab to see which works better with little qwen
This article hints at it, but I think the blocker is environment.
How can you self drive an app on windows? There's no clear UI framework, design pattern, nothing which can bullet proof your app.
I bet you'd have better luck on Plan 9.
Remove the choice. Make frameworks which have limited options which are usable by default.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircrafts).
It's simple: when a mistake happens, you run you CAPA process and determine the root cause and the correction to the process that allowed the mistake to happen in the first place. This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
Author here, hello! Happy to expand on how we're thinking about this if any of it is unclear.
We wrote this post as part of a launch, which you can check out here: https://x.com/danlovesproofs/status/2095182189499711759
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
[0] https://github.com/tailscale/tailscale/pull/17843
We should publish some stats. The fix rates tend to be around 45% baseline, increasing over time as we learn from behavior within a given codebase.