I’m not assuming he’s not competent, and I’ve looked him up - he’s by no means incompetent. But he himself said he’s not qualified to write tests for that. If you cannot write tests for whatever you’re doing, you shouldn’t be doing that. Someone with his knowledge, or at least the knowledge he should have given his CV, should know that. In this specific case he is incompetent, because what he’s doing is simply wrong on every level.

You don’t need to be an expert on what you’re doing to use LLMs efficiently. You can also have solid prompts and ideas to use a LLM to cancel out your personal lack of knowledge in a specific domain. In any case, expecting LLMs to produce correct output when you’re actively guiding it to do something wrong is simply stupid.

Any claim of actual intelligence in a LLM is simply not true. Never been, never will be. Artificial intelligence is an umbrella term for ANI, AGI and ASI, artificial narrow, general and super intelligence respectively. A narrow intelligence is not even close to human intelligence, and is hyper-specialized in a single task. All and any LLMs are and always will be ANIs, and their hyper-specialization is basically a stochastic word (well, token) completion on steroids. An AGI is mostly defined as “close to” or “approaching” human intelligence, as in general knowledge and transfer of it into unrelated fields.

This, reasoning and capabilities will help you nothing when you guide it in the wrong direction. You need to keep in mind the absolutely mind blowing amount of money involved around LLMs. The bubble is too big to fail. Any LLM is a product, and their first and foremost goal is to make you use it, so you pay for it - therefore the primary directive of the AI is to give you what you ordered, to glaze you, and to be your best, obedient buddy. You want a video of the bug, of course! Here you have a video of how that bug looks like - stochastically that’s the answer to the prompt.

Naturally, this code didn’t have tests

Codebase with no tests, check.

it was a UI interaction bug for which I’m not even really qualified to write a test for

What the hell are they doing in bugfixing an UI bug, when they are “not qualified” to write a test for it. Anyhow, not competent enough for the codebase you’re working on - check.

so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug.

So, instead of asking the LLM to e.g. create a proper reproduction as a test case, asking it to bisect, which the author claimed that I wasn’t possible, for some reason. So, also adding can’t bisect on his own, and can’t prompt properly, check and check.

[Waffling about hallucinations] I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. […] The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.

So, the author realized it hallucinates. The author asks for video proof (instead of a fucking test, again). The author is surprised it generated him a video of exactly what they wanted to see, only creating it in a different way than they wanted to.

This reads like “I have close to zero clue what I’m doing, I also don’t really know how to achieve what I want properly, and now I’m making a salty blog post that my magical text microwave didn’t fix my half-assed description of a problem”. Like, honestly, what the hell was the expectation here?

Personally I see LLMs as a tool like any other. You can use it to mass produce low quality slop, just as you can use it to help you produce a higher quality output.

You’re perfectly right about architecture and guardrails, that’s how it has always been with any other tool or piece of software. It depends on how you use it. Remember the no-code hype train? It’s literally the same, people have been shoving it into everything, no matter whether it made sense. It worked for some, and it made development costs explode for others.

Guardrails are especially important for LLMs because you do not have deterministic outputs and potentially exploding costs.

So analyze, measure, and think about where and how it makes sense to integrate, and build it incrementally, again, just like with any other piece of software. Start slow, keep humans in the loop, measure and analyze, and improve incrementally. When you achieve confidence, potentially start automating going into an agentic direction, when it makes sense and the risks have been considered, but always keep provenance. You do not want blind decisions by the magical AI box.

And just to repeat, because I’ve seen heads roll because of dumb decisions: keep cost under control and always have limits set, and always consider which data flows into the AI and what happens with it afterwards.

Producing a half a million bill in a month by accident or neglect or suddenly having your customer database queryable on a public model is a surefire way to drive the company or at least your career to the ground in seconds of wrong decisions.

Also, read into all the stuff built around LLMs, protocols like MCP, attacks and defenses on LLMs, get knowledge about the inner workings, experiment and learn. When you’re the head of AI, you’re supposed to be the person who knows. And when you know what it does, how it works, and how to use it, you’ll find actually good and appropriate use-cases naturally.

Since there’s zero information about what kind of company you’re working at, the following is extremely generalized.

• integrating with monitoring systems, analytics DBs, ticket systems, whatever is used by management, allowing them to ask questions in natural language
• process automation using agentic workflows, e.g. pre-analysis of incoming email queue summarizing / sentiment analysis before the customer support sees it
• provide access to models and model APIs for development workflows and integration into git / ci, allowing to use llm in local development and e.g. setting up something like automated code reviews (not a replacement for human review, only as an addition)
• set up coaching, responsible use, hallucinations, etc.

Whatever you do, take security and data security especially into consideration first, not after:

• consider whether your used provider reuses your data for learning
• consider whether it’s relevant where it’s located (GDPR customers?)
• always set spending limits
• consider your local and your customers data protection laws and regulations that apply to your company (especially in health and financing)