I am constantly improving how I build things around LLMs, but keeping up with what belongs in the harness is a job in itself. A shiny new model comes out and I have to work out how much of that code is still useful, what the model can now handle itself, and whether some of my carefully added instructions are just getting in its way.
Sometimes explaining why I have kept or changed something takes me longer than making the decision. Ask me why we start a new context window for one task but keep the existing one for another, or why I refrain from using certain words in my prompts, and often I have a perfectly reasonable answer. Other times I have a feeling that it will work better, built up from spending a lot of time with these things, and the explanation comes out as something like “the model likes it that way.”
Which is a wonderful thing to explain to technical peers when we have all spent years working with cold, hard, deterministic logic.
I get some of this when using an LLM to write ordinary software too. I develop preferences about what to ask and how much context to give it, without always being able to explain why one approach works better. But when the model is part of the software itself, those preferences also affect how I build the thing. I am designing around behaviour I can try out and observe, but cannot follow through a codebase in the way I am used to.
getting a feel for it
Some of these decisions are fairly easy to reason about. If the previous conversation is full of instructions for a different task, starting a fresh context can help keep those from getting mixed up with what I am asking for now. Or a particular word in the prompt suggests an interpretation I did not intend, so I use a different one. I do not need to understand everything happening inside the model to have a useful explanation for those choices.
Other times, two tasks look similar enough that I would expect to handle them the same way, but I have a feeling that one will be fine with a cheaper, faster, less capable model, while the other needs a big boy. I might have a theory about why, but I am less confident in it than I am in the decision itself.
I get the same feeling when deciding how much to plan out for the model. Do I give it the steps upfront, or will it do better if I let it find its own way? Maybe part of the task should go to a subagent, although now I have to work out what context that agent needs and what happens if it comes back with something useless. I can spend quite a while building this stuff, all because I looked at a task and thought “yeah, I think it needs a bit more hand-holding for this one.” Ask me why this one and not the other one, though, and we are back to the original problem.
None of that is entirely new. In a security review I can get suspicious about a function before I know what is wrong with it, and sometimes that feeling turns out to be nothing. The experience is useful because it gives me somewhere to look, but I still have to follow it through.
With code I can usually keep going until I have something concrete to point at. With a model I can end up with a version that behaves better and still be unsure what exactly helped. Maybe the instructions are clearer. Maybe I stopped giving it two things to pay attention to at once. Maybe I have just made it better at the particular examples I keep trying. Probably all of the above.
Coming up with a plausible explanation is apparently a skill the model and I share.
explaining the thing
This becomes awkward when someone asks why it is built that way, because that is a completely reasonable question. If I have added another step, they should be able to ask what we are getting for the extra complexity. “It feels better” does not give them much to work with, even if the feeling came from a lot of trial and error.
There is a difference between having no reason and struggling to explain the reason, but from the other side of that conversation they can sound very similar. I might be recognising a pattern without being able to describe it yet. I might also be remembering the three times a change helped and conveniently forgetting the times it did nothing. I would prefer to believe the first one, obviously.
The answer cannot just be that everything is subjective now, but it sure is a whole lot fuzzier. I can still test whether a change helps with the task I care about and look at where it fails. Having examples to compare makes the conversation much easier, because there is something for the other person to inspect and question. We can decide whether an improvement is worth the extra complexity even if we disagree about why it happened.
What I find harder is keeping those two things separate when I explain the decision. “This version did better on these examples” can be enough to justify using it, but it is very easy to attach a plausible explanation and start treating that as something I know too.
Then a new model comes along and I have to revisit it. If I know a step exists to work around a particular failure, I have something specific to check. If all I have is “it seemed happier when I did this,” working out whether I still need it is a bit more involved.
still interesting though
I wrote in the previous post about how much I enjoy building a mental map of unfamiliar code. I enjoy that part here too, even though the map is harder to check and I have to redraw bits of it whenever I change models. Working out what to ask of the model and what the surrounding code should handle has become a large part of the work, and I like that there is still so much to figure out.
If I am honest, “eeeeeeeh, it seems to work better this way” is still my default answer. There is often more behind it, but getting that out of my head in a way someone else can actually question takes a bit of work. I want to get better at that, even if it starts with pulling up a few examples and going “look, this is the thing I mean.”