In an earlier post, I described what the "magic" of software development really is - reducing the time and cost of the OODA loop for experimentation. In another recent post, I relayed our experience with regards to getting good results from LLMs. Taken together these point out the actual path to achieving the total factor productivity gains which I had pointed out in an earlier post that had yet to materialize.

Improving the production process is often an expensive process, with it's improvement loop being on the timescales of sprints or entire projects. Even then it has to overcome all the usual human failings of outright lying to simple ignorance and everything in-between. This process can be hyper-accelerated with simulations of the worst employee ever, e.g. clod the robot. You can do an entire year's worth of projects in a day's worth of tokens, and likely find all the places your production process would fail with people too.

Through this simulation of flawed, very human interactions you can cheaply shake out the most pathological failings in your production process via structural interventions. Even if you throw away the prototypes produced during such simulations entirely, the process changes you enact in the pursuit thereof are of immense value. The WWIC effect plaguing projects since time immemorial with expensive rework can be exterminated thereby. When you've already gone through twelve simulations of the project and how it goes you'll have a pretty good idea of what you want on the other end.

Managers of all sorts have to re-orient their thinking along these lines, or they will fail. If they are not capable of understanding this core loop is the most important thing they will be replaced by those that do. This is the closest thing to actually predicting the future we will get in our lifetimes, and we'd be fools not to use it.

No, not that PCP. Nor "Praise, Correct Praise". Perl::Critic::Policy.

The core issue when dealing with LLM tooling to generate programs is the same problem working with other people. Unstated preferences means you get output which is unacceptable, but understandable because mind-reading is beyond both. Encoding this in things like skills is still not enough. Just like with your own internal wiki documenting how you "do things around here", you can be sure it will be routinely ignored.

No, you need something tighter in the OODA loop. This is the same reason that LSPs and IDEs are so popular with people too. Programming languages have so much to remember that "teacher, my brain is full" is a serious problem, only remediated by near-instant feedback. Given LLM agents are dumber and more forgetful than the any fauna you have ever worked with (it is a computer after all), the rule applies.

Sometimes even this isn't enough, and you need hooks. The earlier in the process, the better (as always). This also will save you tokens, as it doesn't waste work on things that won't pass muster.

The process that works (which I tentatively call "rudra" - "he who eliminates problems by their roots") is like this:

1. Ask LLM to do thing, with existing LSP/Critic/Tidy hooked up.
2. Review diff. If there is any issue, no matter how trivial, the change-set and LLM session gets the death penalty.
3. Ask LLM to write critic policy or a hook which will prevent it from ever making such errors in judgement again.
4. Repeat process from step 1

This is essentially simulated annealing at a layer of abstraction you don't normally encounter it. To be fair, these "agents" are actually a declarative data retrieval language, much like SQL. They simply have a query planner which is a Markov model. Most do not recognize this because it is at a higher level of abstraction than they are accustomed to.

The usage model here has similar limitations to SA. You will quickly find there are points at which you reach diminishing returns as regards encoding your preferences. This is no problem for most people, because "good enough" is.