> We may create codebases that are not merely hard to maintain by humans, but that assume machine participation as part of their maintenance model... People more and more merge code they cannot fully explain. People lose their ability to create issue reports or discuss things in chat, without augmenting or rephrasing their messages with the context provided by a clanker. Too many people increasingly rely on a machine to summarize or contextualize it. More and more do I encounter people who converse with me through the indirection of an LLM.
I experience this daily now. It find it discouraging and concerning.
I believe we're merging more code we can't fully explain because we are now relying on code review to build the mental model that was previously built by writing code and collaborative technical planning. I don't think code review is fit for this purpose. I do think we can extend code review with structured exercises, informed by pedagogy, that strike a better balance between friction and understanding. (I'm looking for help testing these exercises).
And this code is often full of security vulnerabilities. It's just hacks on top of hacks on top of hacks. You end up with 100K lines of code full of weird fallbacks, doing something which could have been done more reliably with just 1K lines of code.
I think author's comment about preferring systems which make invalid edge cases impossible rather than implementing fallbacks is hugely important. With the fallback approach; you end up implementing fallback on top of fallback on top of fallback... Each fallback seems to increase the amount of code exponentially and somehow it always creates new problems. This should almost be a 'General law of system design.'
Fallbacks reduce the risk of failure but make failures more complicated and harmful when they do happen.
As a software engineer, like the new coding environment which is being created by AI.
Big tech companies have created infinite work for me. The human developer has become a critical component of code execution. The human needs to always be present to handle the nearly infinite number of difficult unhandled exception cases which are guaranteed to occur from time to time.
The software engineer is no longer like a laborer, but more like a security guard who sits at his desk drinking coffee most of the time and only steps in on rare occasions when something goes wrong.
My experience with LLM's is that the "fallback" issue is probably one of the most serious issues. I have seen zero talk about it outside of this thread. I'm not sure its even being worked on. The llms have this terrible drift towards always making something happen even if that thing is not even related to the task at hand. "failure" in the sense of simply throwing an error/exception is something models seem highly resistant to.
I cant tell you at this point how many times I've seen them do something like
Fail > make up values > maybe log it > keep working silently with increasingly corrupt data.
This isn't going to work because the LLM doesn't have enough context. Many security issues involve a failure mode which cuts across multiple parts of the code. A PR which seems perfectly valid on its own may be the missing piece which opens up a vulnerability. Each component may be fine on its own, but brought together, the system is vulnerable.
Think of a machine with interlocking gears; each gear may itself be perfect and may fit perfectly with each other, but then if a tiny pebble comes between them, the entire machine breaks. Maybe the problem here is that the final gear was too close to the ground and would catch stray pebbles kicked up by the wheel in front of it... The LLM couldn't know this unless it understood the full context in which the change occurred; not only the code, but the environment itself.
In a poorly designed codebase with hundreds of thousands of lines of code, it's impossible to have the full context of the code even. The architecture would lack proper separation of concerns to allow one to effectively establish an appropriate defense perimeter. In a poorly designed codebase, every part of the code can harbor a vulnerability.
It's like; if you don't have a proper access control layer which is automatically and declaratively enforced for all your endpoints, every endpoint will have to enforce security restrictions on their own; duplicating similar-looking code over and over. If one endpoint out of 1000 incorrectly enforces a security restriction, that could be a critical vulnerability.
Of course you don't just check the diff. Rather in your CI infra, it's important as part of every PR, it needs to be given the full repo to check if it introduces any issues. This works wonderfully on github, even with non SOTA tools like gemini-code-assist.
why do you think it's not possible to have full context of codebase? modern harnesses excel at finding all the right codepaths, even in a large codebase.
Not to mention that effectively reviewing code is a much more difficult skill than writing it. Without a good mental map of how it
affects other parts of the system, it’s basically a rubber stamping ceremony.
Github’s poor PR UI doesn’t help either, there’s limited tooling to navigate around the codebase not directly changed (but affected) to identify and highlight problems.
I agree that reading AI code all day is agonizing. We're relying on code review to develop parts of our mental model of the system that were previously developed through coding. We're having more difficulty comprehending and recall details of the system. This is probably unsurprising; people recall information better that they "generated" than information they read. I am applying some lessons from pedagogy to extend code review. If this resonates with you, I would like to talk.
Even when I write a detailed specification and review the resulting diff line-by-line, I am unsatisfied with my comprehension and recall of the changes. I don't understand my systems as well as I used to (I don't think this is surprising; see the "generation effect"). I have been experimenting with extending code review with some new exercises intended to improve comprehension, at the cost of a little friction.
When I work with Claude to plan a feature and then review Claude's implementation, I don't understand the feature as well those I developed without AI assistance. I don't recall details of the feature's behavior as well, even days later. I suspect that this is not surprising to anyone who has studied pedagogy. I've been working on applying some exercises during code review (including self-review of my own AI-assisted code) to improve comprehension and recall (https://bridgekeeper.io/). If this problem resonates with you, I would like to talk.
I'm working on a possibly-quixotic tool to mitigate the "cognitive debt" from AI-assisted development. Not everybody agrees that this is a problem. Maybe some teams that are only writing specs and reviewing plans still understand their products adequately. If you have an opinion either way, I'd appreciate hearing from you.
I'm working on a tool for mitigating the cognitive debt from AI-assisted development. Pedagogical research provides some techniques we can use in code review to improve comprehension and recall. If this resonates with you, I'd appreciate hearing more about your experience with cognitive debt.
I'm working on a tool for mitigating the cognitive debt from AI-assisted development. Even when I review Claude's diffs line-by-line, I don't understand them as well as code I wrote myself, and I have difficultly recalling them days later.
Pedagogical research provides some techniques we can use in code review to improve comprehension and recall.
This doesn't resonate as a problem with everyone, but I'd like to hear from you either way.
Also, the tool talks like the Monty Python bridgekeeper, for whatever reason.
I experience this daily now. It find it discouraging and concerning.
I believe we're merging more code we can't fully explain because we are now relying on code review to build the mental model that was previously built by writing code and collaborative technical planning. I don't think code review is fit for this purpose. I do think we can extend code review with structured exercises, informed by pedagogy, that strike a better balance between friction and understanding. (I'm looking for help testing these exercises).