Unfortunately this often represents the only feedback given by the people in those „higher“ positions.

„The indent is wrong here“

„Comments should end with a period“

Because this kind of feedback is and was always easy.

If you get feedback like that, it’s time for your team to get automated linting/formatting

Call for a style guide meeting every time you see feedback like this, and write down what everybody agrees on. You’ll never have to do it again after 3-4 of those. Problem solved.

Still not solved? Guess it was really about the commas and not the value delivered anyway, so do whatever you feel like.

That meeting sounds like it would be the most bikeshed-y thing imaginable.

What you actually need to do is have a manager lead assert that it’s happening in a top down way and just run the formatter with the defaults. Let people argue case-by-case on what to change after that.

There's been a lot of talk about the purpose of code review recently. It makes sense in the face of AI. Heres a link that was submitted a little while ago: https://mathstodon.xyz/@mjd/115096720350507897

And in response I wrote a non-exhaustive checklist of things that a code review can look for:

- Does it functionally achieve what it sets out to (as per tacker issue or PR description)?

- Does it have extraneous code? Leftover debug prints, private API keys etc...

- Does it have any obvious defects? Memory leaks, un-handled edge cases, security flaws, obsolete API calls, etc...

- Could it be more understandable? Add/remove abstractions, better variable/method names, more/less functional etc...

- Is the style consistent with the codebase and/or style guidelines?

- Are there obvious performance improvements? Hashset instead of list, lazy evaluations, etc...

- Is it sufficiently well tested?

I think LLMs are okay at most of these, and worst at the first.

- Do we want this? Cost/Benefit etc

- Is the change architecturally right?

Particularly the latter LLMs seem still pretty useless at.

The former feels more like a product leadership problem.

Although I do think that LLMs have made it much easier to justify writing low-value code which can make this more common now.

Missing my biggest issues as you ask the agents to do larger tasks with less up front planning.

Is there already a pattern or code on in in the existing codebase that handles this functionality,

Do we really need net new code to achieve this functionality?

Can existing code be extended or abstracted to more cleanly implement this feature or functionality.

“Net new” is one it seems to be particularly bad at.

I don’t think I have ever even once seen an LLM solve a problem related to overengineering by simply removing the overengineering. They always choose to add more epicycles and further compound the complexity.

You can tell it to do that. I don't mean abstractly, I mean concretely, tell it how to solve it and it will do what you tell it.

When you get to that level of micromanagement, won't it be simpler to just do it yourself?

Pangram check on the article: 94% of this text is AI

In my experience, automated code review is more pointless than ever.

We have all the linters, tests, and AI writing code for us. I don’t need the left hand to tell the right hand it did a good job. I’m very certain my code runs when I push the PR.

What I need now is architectural, long-horizon and business perspective.

What kind of code review tools did you try?

What I need now is architectural, long-horizon and business perspective.

That's exactly what these tools are now good at. They have a huge gap when fixing these issues properly but they can spot these issues no problem

No, no they can’t. I use AI a lot and I get a lot of value from it but they are still terrible at programming “in the large”, by which I mean slotting features into the place meant for them in the existing code base. When coding and reviewing their view is too local. They will implement a change in the first place that looks feasible when coding. When reviewing they will not look for code duplication or fit. They will review for correctness, performance, security and style but not for architectural coherence.

They also have shockingly weak ability to identify business acceptance criteria that are completely missing in the implementation or test coverage.

For those of us who do utilize the pre-existing tools and the coding agents effectively, yeah automated code reviews eventually become redundant. But they're still around because there are a lot of devs who don't even glance at what the agents are writing for them. Maybe partly because they don't care to streamline their workflow, or maybe because they've never been particularly good at writing code, and don't actually understand what the agent is giving them.

Lately I've seen some pretty glaringly obvious issues caught during the preliminary automated code review, and the issues seem to be coming from individuals who don't actually understand what the code is doing.

For the rest of us who are actually using the whole stack effectively, the automated code review is essentially a CI gate to protect the repo from the devs who don't know what they're doing.

To get the context that isn't in the code, maybe it would be better to ask for a review of the prompt?

Thank you, well put! Bots reviewing code written by bots is a self licking ice cream cone.

I believe code review is important (the article articulates some of the reasons) but I have to admit, having different models review a PR before passing to a human has proven valuable.

I agree that using LLMs to review code can be valuable, but I think the submitter should perform this review (or you could automate it) and then once the submitter has fully finished the feature and believes it's ready it should be reviewed by another human.

I'm not interested in being a middle man between Claude and my colleague, nor am I interested in using a colleague as a middle man between myself and Claude.

A defense of human code review I wish I saw more often, especially in light of the concerns people have about cognitive/comprehension debt: comprehension redundancy. At the end, if taken seriously, at least two people understand how the feature works (even if that number is, on average, trending closer to between one and zero). Ideally at least one of the two also comes away with a better understanding of the wider system and how the feature fits into or stands out from that landscape.

If only one person knows the code, then PR time isn't going to save you.

I'm tech lead and I basically don't review PRs, and I tell people this, with a caveat - if you can tell me what you specifically want me to review, for what specific purpose, I'm happy to!

So "can you review this bit for race conditions" is great, love it. This forces people to actually think about what in their code they should be suspicious of, if anything.

"Can you review this" [link to PR] is getting a rubber stamp because humans have never been good enough at "just spotting bugs" to make this worth it and now any LLM is better than a human.

And for the purpose of understanding - PR time is too late. I have not reviewed a PR ("for real") in a long time and yet I could tell you how every system my people have built works down to a very fine level of detail. And it's because _we talk to each other!_ We don't just chill in the same slack channel and code independently, we all value each others brains and want each others inputs because we know it will improve our product and we value what perspectives others will bring.

Trying to learn via PR is a sad substitute for real collaboration and teamwork.

I consider PRs to be primarily a defense of the architecture, and to a lesser degree a general sanity check. It’s also a useful opportunity to enforce automations are being run

I consider 99% of my "defense of the architecture" strategy to be teaching my team why the architecture is important, how to think about it, and invite they commentary on it as we own and evolve it together. And of course if they are doing something and want input or are uncertain, then my door is open.

If PRs are a notable part of my architecture defense, I'm going to work on investing in the team instead of reviewing PRs.

I consider 99% of my "defense of the architecture" strategy to be teaching my team why the architecture is important, how to think about it, and invite they commentary on it as we own and evolve it together.

Nice way to avoid the responsibility for any team fuckup: it's not me, I only teach them, they decide themselves.

Nice way to avoid the responsibility

Welcome to the tech industry. Its all about shifting responsibility in case something goes wrong. Thats the only reason companies use third party software in the first place, to have a scapegoat...

I mean, do that too, but end of the day PRs are your final chance to catch the mistakes before they start cementing, and is your best opportunity to identify misunderstandings (you can smell the confusion in their changes and address it directly).

But also, I must defend against the hordes of unwashed masses and maintain the sanctity of my domain. End of the day, a codebase I own is a codebase I own, and others cannot be allowed to poison the well, intentionally or not. That’s how you get cholera

THANK YOU. I’ve always felt this way about PR review. I feel like people should write a natural language description of the change, and every section of it should link to part of the diff, and every part of the diff should be linked to by part of the description. That or just leave a comment on every chunk of the diff.

People in my team are generating long and verbose PR descriptions with AI. No idea why and for whom. I certainly never read any of them.

This is wild to read, I always review PRs and frequently find bugs or significant problems in them that get them bounced back

That's a strong signal that your team isn't doing well. Significant problems should have been spotted at a software design stage, or raised in standups, or identified in a pairing session. The earlier you can find an issue the simpler it is to fix, so waiting until the last moment (e.g. PR) means you're spending far more time fixing issues than necessary.

As for finding bugs, what happens if you miss them? Do they go out to production and potentially lose user data? Finding bugs in PR is a big problem. For a start it shows your automated tests aren't good enough, and secondly it shows the devs aren't checking their code works well enough.

If you do them at all, PRs should be a gate for checking whether the code meets the team's quality bar, not if it even works. The team should be able to deliver working code without them.

I'm guessing you work on some highly technical domain that receives highly structured data and is amenable to TDD.

Not all domains are like that. The majority of bugs I see are business/domain logic bugs. It's akin to having misunderstood or missed some aspect of the question, not causing data loss.

I'm guessing you work on some highly technical domain that receives highly structured data and is amenable to TDD.

I lead a group of teams that build frontend software, so not really. :)

The majority of bugs I see are business/domain logic bugs. It's akin to having misunderstood or missed some aspect of the question, not causing data loss.

Financial loss, reputational harm, degraded UX, etc. They're all significant problems. They're more recoverable than a data loss, but equally bad from an accepted low quality standpoint.

I'm also going to guess that you don't have BAs, PMs, or people responsible for the logic reviewing the code in a PR. Consequently you can't spot those problems in at the PR gate unless the issue is that the dev didn't understand the requirements and wrote code that didn't do what it's supposed to. In which case we're back to the quality and testing problem. By raising questions in standup ("Can I clarify that I understand the AC right?"), pair programming ("Let's check the code against the AC") and communicating properly ("Can you demo the feature to the BA so we can be sure it's correct") you move the problem to the people who can answer, and stop the devs needing to review that someone wrote working code.

I just don't believe PRs are the right point to be finding out that the requirements were wrong or that the dev didn't understand what to build. That needs to happen as early as possible. PR is as late as possible.

I'm pretty sure human code review is already on the way out for > 90% of code generated outside of critical systems.

I am contemplating code review within my own organization, and the question I return to is:

Does this organization prioritize human learning?

That has been my primary motivator for code reviews. I want to teach and learn from others, especially given the decreasing levels of collaboration due to increased AI usage.

The sad truth is that all of my feedback just goes straight to agents. Maybe 10% is reacted to by a human, so I’m left wondering if there’s any value to a real review aside from poorly training robots to do my job, and further atrophying the abilities of my team members.

Well, I can tell you that at my current job we had something of a crisis over the summer when we realized that nobody could make changes without fear of breaking things anymore because we lost the ability to tell which existing behaviors were and were not safe to change.

Previously the knowledge needed to discern that sort of thing would be disseminated through both design and code review sessions. But plan mode and AI code review largely put an end to that.

So we put our heads together and came up with some new policies about project management and how we use AI, and things have steadily getting better since then.

(Though, in fairness, the one guy who seems to actually enjoy getting paged after hours seems to be having less fun.)

What plans did you put in place? What did you do to ensure knowledge transfer?

Strict WIP limits, all non-trivial projects need two human collaborators who are both responsible for understanding and signing off on the project plan, docs, code, test strategy, etc., and regular sync-up meetings.

So do you still have humans doing code review? What did “sign off on code” look like in practice? (We’ve been thinking about this a lot too)

    If you will tell me precisely what it is that my machine cannot do, then I can always prompt my machine to do just that.

    - John von Altman

Articulating “what humans can do, that AI cannot” is a mug’s game. If you specify it well enough, they just paste your text into their /goal prompt box and ralph loop their agent swarm until it produces something too exhausting to distinguish from doing the thing. If you don’t specify it well enough, then you’re just doing human-centric magical thinking to move the goalposts etc etc.

No. It's very easy. AI doesn't have value judgement. Or, in simpler language, it "doesn't care". You can't make it care. It needs to have stakes in the real world for that to happen. It needs to want to preserve itself, to be able to be happy, to understand what makes it happy and to optimize that function. Similarly, there should be things the AI doesn't want to happen (to it), identify them and be able to make up a strategy to avoid them.

You may think you can pretend your way out of it, by, eg. copying human concerns onto AI's behavior, but it won't work because if the AI is "smart" enough, it will discover those to be a lie, and if it's dumb enough to get confused by it... then you will have an artificial stupidity instead of intelligence...

More so, you actually don't want AI that has value judgement, because, if, hypothetically, you have created one, then it might as well start fighting for resources against humans...

So, AI is bound to be limited in what it can do the more the decision belongs in the strategic or meta domain. And you very much want it to not grow a mind of its own, so that it doesn't write or approve code that optimizes AI's happiness, instead of yours.

I’ll admit, I read some papers about basic instrumental drives common to all goal-directed behaviour, that suggested artificial intelligence might end up competing with us (and soon after, outcompeting us). But I am relieved to learn that if this does happen, it will be an “artificial stupidity” instead of an artificial intelligence that repurposes the atoms I am made of.

The true reason why code review is universal is that it provides a liability shield for negligence. Negligence is interesting. It has nothing to do with whether or not you ship something broken. As long as you follow a process that attempts to not ship something broken, then you are not negligent.

Engineers played along with this farce because code review served valuable team collaboration, coordination and management functions, about which the author of the article is correct.

Understanding a system by reading code is harder than understanding a system by writing code.

If AI can generate code at 100X, 1000X, or 10000X human capacity (no ceiling here), and you are gated on code review as your mechanism for system understanding, then a team's productive output will barely increase.

If companies want to compete in the world of AI generated code, human code review has to go. The only question is, what replaces it?

Continuing to apply human code review to AI generated code is negligent, if you are shipping at AI generation speed, with that as your only gate, and no other systems and processes to validate correctness and limit risk.

On the engineering side we can adapt easily.

Code review was never about finding bugs. When we do code review the first thing we check is: "do the tests pass?" Then we look at the change and the test coverage added for it and ask: "does the test coverage adequately demonstrate the functionality of the code?" The we ask: "What is the scope and potential impact of this change?" "What is the deployment and rollback plan and how will we monitor and detect defects after deployment?"

Code review was never about the code. It made the lawyers happy and provided a vehicle for doing the things that actually make systems work.

"does the test coverage adequately demonstrate the functionality of the code?"

and is this a solved problem? If not, then the bottleneck is right here, if it is solved, then yeah we shouldn't need anymore software engineers other than the elites

It's "largely solved". I'm not clarifying, thank you.

We're using Github Copilot to review some of our PRs (the functionality that's built-in into Github directly). Man, I hope that people are using better agents for code review. Because if Github Copilot is in any way comparable to what the "machines will review all code soon" people are using, then I'm really worried. It's nowhere close to "good". It can find some obvious things, but it misses too much and has too many wrong findings.

One of my clients has an automatic "best practices" AI robot that runs each time you create a PR. It is pure downside. Even the developer responsible for creating it admits as much.

However, for some weird reason it's still in place. This is the part that actually concerns me. Ignoring the bullshit comment is trivial. The quiet and relentless accumulation of entropy is happening everywhere. This is why GitHub crashes at noon every business day.

This does not reflect my experience. AI code review has improved tremendously in the last years to the point that nobody in my team does manual code review any more. It used to be ridiculously bad two years ago, but not any more.

Do you use some "framework" specifically or (custom) skills, or is it just plain prompt to review the code?

I was just at the Explore DDD conference in Denver and a portion of Friday was sitting at the cafe tables informally discussing the impact of GenAI on software engineering with notable people.

Most of these people were deeply concerned that if we lean into using GenAI for “everything” that our collective knowledge will dissipate.

I was the vocal contrarian. There are many historical examples of humans obfuscating knowledge to simplify progress.

Does anyone solder their own microchips at scale anymore? No. We have highly sophisticated robots and machinery to do that work with extraordinary outcomes.

In software engineering, if you remove “coding” as a discipline you’re left with all the other aspects of designing software which I contend can be retargeted in college CS curriculum.

The leap isn’t about code reviews. It’s about design reviews and that’s where better outcomes are served regardless of whether GenAI is involved or not.

I have a roughly year old codebase at https://github.com/ChicagoDave/sharpee/ that is designed by me, but generated by Claude Code with my own skills and agents as guardrails. I’m fairly certain the code I extract from Claude doesn’t require human review, but the design of the system and its changes are continually reviewed by me.

My contention is that we “collectively” are still trying to discern where the AI/human line is and most are still “holding” that line to human interactions.

Let it go. Define what part you do need human decisions on and focus on those things.

I really don't get this discussion at all. Ignoring human comprehension etc., every code review of agentically created code shows bad tests (lots of overfitting, testing on specific implementation details, not general behaviour/business logic), weird and repetitive abstractions, duplications of code, circumventing architecture etc. etc. It's a complete mess. Unmaintainable, bad craft, bug-prone and corrosive. That's what you want to catch. You're piling on tech debt like hell and you're gonna pay for it down the line. Look at the fucking code and then come back to argue. Jesus. I don't even understand how this is a debate. Look at the code.

Management wants to go vroom vroom. Gambles that new models will be good enough to fix tech debt.

I do agree with you

It's also a gamble that you don't have to look at the underlying layers because it's "the new abstraction layer". Which is intellectually incredibly lazy.

They talk about astractions as if "requirements -> code" is the same as "Go -> ASM -> bytecode". Which is just utterly stupid. But downvotes tell me a different story haha.

I'm also management btw, and the two startups I am a CTO at all have clear buy-in from other C-levels that we can't just spam AI slop w/o reading it and hope for the best. But I do see that most probably don't see it.