You can tell it not to do this?

Something like:

"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)

... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.

If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.

I yolo vibe develop, but how one end up with 210 files change commits? Unless one shooting a project, there is no way to end up with such scope? I really don't get it.

Most PRs that are generated by agents today get YOLO merged

Not quite. Ironically, they get PR-reviewed by someone else's agent. The humans in between are meat-proxies, pressing OK buttons.

I'm going back to writing code by hand. AI can assist with debugging and research. I've had enough of slop. I've had enough of dealing with people wielding slop machines.

In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.

This is bananas. I've seen 25 year old software systems that haven't broken the 100k commits barrier. It feels like that ought to be enough for entire product lifecycles. What's going on that isn't simply wheelspinning?

I had to bisect over around a million commits to the monorepo to find the culprit.

Thank god bisect is O(log n) at least...

I can confirm that our code quality at work is regressing and we are shipping less product features than before with AI.

Sadly, the hype is real and that's what's happening in many places. Businesses are eager to bet on AI cause they'd been promised x100 productivity = fire 99 or 100 devs = huge profit.

I expect one day some slopware will succumb to one of these weird production bugs, no LLM will be able to fix it, and when the biz guys ask me for a fix estimate, I'm going to say "3 years".

It's funny, because in my experience, the process often looks like:

1. Human gives high-level design. 2. Agent generates wrong code with misleading comments. 3. In further iterations, agent get mislead by said code and comments, ends up generating insane workarounds.

It is.

Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).

The slop theory tells us that (slop * slop) = doubleslop.

confidence score really feels like real machine learning project which we test to check the confusion matrix

What's so revolutionary?

Because it's all "AI" now <insert Ancient Aliens meme>

Could you tell us? That would be simpler for everyone no?

I did this yesterday with user data to find duplicates and determine which conflicting fields should win in a merge. Took me about an hour-ish to run the experiment. I provided a good amount of context for each pair. It was underwhelming / OK at best and didn't add any value to my existing merge workflow, so I threw it out after the experiment (I kept the findings though). It was quite easy to setup and use via openrouter and extremely cheap and fast.

For fun I also tried using it at my local LLM router (all of my prompts and responses go through it for personal analytics) to decide which model to route tasks to based complexity etc. Again, underwhelming for my purposes, and so I'm not using it.

Not sure what real world use case it's best at, but I agree that it's simple enough to implement it yourself and see if it fits the type of work you are doing.

I don’t think its much cheaper like luna is double the cost. and you can easily use luna to generate a training set and train your own model. I think the main things its faster so a lot of people go the Wow moment I can classify my whole db at once. Or this model decides so quickly. Speed is the key differentiator, it’s a strong one, always.

But isnt the whole point of Jev that "we trained it so you dont have to"? Yet it make such a simple mistakes still.

Yes, I don’t think hallucinations will ever go away. But because it doesn’t work with language the type of hallucinations aren’t too comparable to LLMs. But it is still a risk of course, whatever system you design should take that in consideration

custom trained classifier

The "moat" is data. And compute. But mostly data.

It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.

One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.

There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.

(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)

The core feature is good quality out-of-the-box domain and cross-domain calibrated probability you can use as a threshold.

SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.

Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.

Like the classifiers we were building 15 years ago with random forrests and logistic regressions

I think no one mentioned here, but the obvious difference is that Jev can spit out decisions directly from natural language input. None of these ML models could do that, and other than a full-fledged LLM (which is optimized for conversation and agentic tasks) or some classical NLP models (which underperform compared to LLMs, AFAIK), there is nothing right now that rivals Jev-like models.

Of course it's not perfect, but seems like the right step forward for quick classification/decision tasks based on natural language.

I have a better idea, what about stop spamming slop PRs to projects?

Jev has good use cases that require speed. But speed is not all. Tasks like code review need quality, which costs time. If you just want speed, you might as well go with a non-reasoning model like Haiku.

i read this as: "you are generating code wrong! you need to generate code better, so you don't even need to think about what you are generating!"

jev is not good at reasoning, its a generic low-level classifier. for reviewing / classifying code, i would strongly suggest not using jev.

You don’t need classifier to filter noise. The noise is the same patterns repeated many times. I’ve got a tool does that to break down big diff and then feed to llm for grouping. https://github.com/thepartly/differential

Yey! Let's speedup project downfall to crap abyss even more!

You either care about project and do reviews. Or yolo vibe code. This middle ground has little sense.