I am an ethical hacker who specialises in custom software. Some hear ethical hacker and think HACKERMAN, but it mostly consists of reading code. A lot of code. In a busy month I review 2 to 4 custom codebases for security vulns. The more complex codebases can eat up the entire month.
That used to be slow. Clone the thing, read it, build a mental map. Chase the odd choices that even the original developers can no longer explain. Follow functions until I get why they exist. In that initial period you go from understanding nothing, to understanding something, and hopefully to understanding most of it. It is often said that code review is no one’s favourite activity and that most just want to develop, but I love it.
But LLMs have changed how people write code, and they have changed how it gets reviewed too. Plenty of software engineers have written about the sense of loss that comes with their field changing, and I feel it too. That initial period I love is one of the parts getting compressed. It is a bittersweet feeling though. The mental map that used to take me days to build now takes me an afternoon to have a rough equivalent.
The model reads faster than I do (I am dyslexic after all), it never gets bored (I have ADHD after all), and it will happily trace a call path across forty files while I go make coffee. It is the biggest shift in how I work that I can remember.
There are two halves to it though.
the fast part
The good half first.
It is very good at the monotonous work. Finding every place a function gets called, tracing what touches a piece of user input, checking which of forty near-identical handlers actually reach the database, the boring exhaustive coverage that used to cost me entire days. It churns through all of it and does not skip a file.
A recent one: a typical app without centralised authorisation logic. Every route handler should be handling access control by calling it locally, which means every route was one forgotten line away from being public. Verifying that by hand is a day of reading the same function shape on repeat. Now it is asking the model to walk every route and flag the ones missing the check, and then making that coffee. It came back with a short list. Most were public on purpose. One was not supposed to be.
And it is good at the thing I used to spend serious time on, building the shape of an unfamiliar system. It gets me to a working mental model in hours, and from there I can start asking the questions that actually matter.
That head start alone is worth it. I move into a codebase faster than I ever did.
confidently wrong
Now the other half.
The counterargument I hear the most is that these models (not picking favourites now, that is a discussion for another day) will hand you a wrong view of the code. And that is usually where the discussion ends. Not how wrong, not how often, not what a wrong model can still do for you. It is wrong, do not trust it, waste of time, done.
Sure, sometimes it fumbles. It makes truly bad conclusions, it misses the point of a module entirely, and it is confidently wrong.
The tricky part is that it does not tell you when it is guessing. A handler it traced properly and a handler it made up on the spot get explained the same way, clean, plausible, well structured, same confident tone.
My favourite example so far, from a recent review. Round one: it found a memory issue. Round two, asked it to verify itself: no, that was a mistake, there is no issue there. Round three, asked it to double check a final time just to make sure: “You are absolutely right! I see it clearly now, yes, the buffer is definitely overflowing here.” Same code, full confidence every round. It cannot even agree with itself.
Obviously some wrong answers are easy. I read them, they do not make sense, I go look for myself. The fluent ones are the sneaky ones, because they read exactly like correct ones. In most jobs that costs you some time. In an assessment it can cost you the finding. A suspicious path waved off because the summary said it was fine, but the summary was made up.
So I do hear the counterargument, it really is wrong a good chunk of the time. But so am I. Half of my early theories about a codebase die as I read the rest of it. That has always been the job.
And that does not have to be just a bad thing. Fast assumptions, mistakes, correcting on the fly, that is the core of my flow towards understanding a codebase. A confidently wrong model fits right into that flow, it just means some of the corrections are aimed at it instead of at me.
It also keeps me on my toes. The summaries sound right even when they are not, and it is tempting to move on as if I read the code myself when I only read a summary of it. So the job is to keep up with the model, disprove it when it needs disproving, and steer it around when it needs steering. That is not much different from working with a colleague, or a colleague working with me. For anything that matters I go read the actual code, not the summary of it. The tool tells me where to look, it does not get to tell me what is there.
Even the wrong answers are rarely wrong from the start. Usually it read the right files and traced a real path, it just took a wrong turn somewhere along the way. Everything before the turn can still be built on. Sometimes there is an original idea hiding in it, or a line of thinking that got 99% of the way there. In a security review that is useful input on its own, a brainstorm, a bit of interactive rubber ducking.
And the slow way did not go anywhere. Following a function by hand, getting confused, backtracking, carrying three half-wrong theories about a module until one of them survives contact with the code, that is still the process that builds the real intuition, and it is still exactly the kind of rabbit hole my brain loves to disappear into (the ADHD again). The difference is that I do not always have to spend that time anymore. Sometimes the model has already done it for me, which leaves more time to look at other things.
two halves make a whole
So both halves work out in my favour. The fast half saves me days. The wrong half lies to me with a straight face (if it had one) often enough that I can never switch off the part of my brain that checks. In this line of work that part should never be off anyway.
Mostly it has shifted the job. Less of my time goes into the slow first read, the part I said I love, and more of it goes into deciding what to actually verify. Turns out I love that too.
I wrote in the first post here that I like tools but that the thinking has to stay mine. That still holds. The model can do some of the work, the understanding is still mine to do, even if that means I have to work harder and faster these days to keep it that way. The truth? I would not hand that part over even if I could.