Tacit knowlede and domain expertise in a world of LLMs

Obvious, but worth saying: LLMs, and the people prompting them, still require tacit knowledge and domain expertise to produce something useful and worthwhile. That's why I've been building stuff that solves problems for me – because I know what's 'correct' for my situation.

What I’ve watched happen... is more specific and more interesting: the binding constraint has moved from can you build it to can you tell whether it’s right.

Think about who can actually use one of these tools well. Picture two people.

The first is a domain expert with no real software background. A logistics dispatcher, a clinical coder, an actuary. They can’t read a stack trace and they couldn’t tell you the difference between a hash map and a list. But they can look at a schedule the agent generated and know instantly that no driver can legally work that shift, or that a claim with those codes would never pay. They know the correct outputs for a given set of inputs because they’ve spent ten years living in those inputs and outputs. Hand them an agent and they are startlingly effective, because the thing they’re missing, the ability to produce code, is exactly the thing the agent supplies. What they bring is the thing the agent can’t: the ground truth.

The second is a strong generalist engineer who has never worked in the domain. They can architect anything, they know reliability and testing and how to keep a system from falling over at 2am. But drop them into clinical coding and they cannot tell a plausible-looking wrong answer from a right one. The agent will happily generate a billing rule that compiles, passes the tests the engineer thought to write, and is subtly, expensively incorrect. The engineer has no oracle. They can verify that the software is well-built. They cannot verify that it’s correct, because correctness here is defined entirely by a domain they don’t hold in their head.

Source: Aaron Brethorst