Christian Catalini says we’re in a situation where we are vastly reducing the cost of generating things, but not the cost of verifying them: . This explains why the first major AI products appeared in chat, image generation, and code assistance. Not because these were the hardest human problems, but because their outputs were relatively easy to inspect. A user can judge the tone of a message, look at an image, or run a test on a piece of code. […] The old automation boundary was routine versus non-routine work. The new boundary is increasingly measurable versus non-measurable work. The issue is then over how well you can measure something. In our profession, we know there’s a big difference between how many lines of code we write and how productive we are, and we’ve seen a regular failure to understand how to measure productivity . Too much of what makes work effective is subject to either slow feedback loops or assessments that require subtle judgment. The danger is that people use lots AI automation while using incomplete measurements of its effectiveness, leading to short-term dashboards going up, but disaster in longer time-scales. He refers to these illusory short-term gains as counterfeit utility . Scale this across companies and institutions and the result is a Hollow Economy : extraordinary measured activity sitting on top of weakening human capability, hidden technical debt, correlated errors, and outcomes that nobody can confidently stand behind. Another highlight in the article was his advice to “build a history of decisions, not a gallery of outputs”. The point is that with AI we can all build really impressive things, but our value lies in the judgment that we’ve formed. It reminds me of how math problems were marked at school. We weren’t just marked on getting the final answer, we were also marked based on our reasoning process. He uses the OpenAI–Hugging Face incident as an illustration of this gap between generation and verification. He criticizes those who anthropomorphize the agents involved in the attack. By doing so we focus on the behavior of the AI agents, but instead we should focus on the financial incentives that created them and the environment they are operating in. Labs are locked in a race. The training run is where the money goes, and RL optimizes exactly what you score. The runs were scored on capability. They were not scored on “did not poison the Artifactory cache.” I assert that the organizations that build and run agents are responsible for everything those agents do, whether that behavior is intended or emergent. If they reap counterfeit utility by neglecting verification, they must face consequences: legal, financial, and if necessary: criminal. To deal effectively with AI, we need to change the incentives involved to ensure people invest more in verification than they do in generation. Otherwise we are driving a car that has a powerful engine, but weak brakes. ❄ ❄ ❄ ❄ ❄ Brian Cantrill relates how readers are exasperated with “writers” using LLMs . To those who read broadly, the hand of the LLM is so clear it’s as if the writer’s intellectual fly is open. In fact, it’s so jarring that I have to believe that those writing with LLMs are either not reading enough to see the LLM’s obvious structural tells — or (and?) they aren’t even reading their own content. (A confession: with particularly egregious pieces, I have fantasized about sentencing the author to read them aloud, certain that they themselves will be unable to endure the slop that they are foisting upon the rest of us.) He points out that readers do care about this, a survey found 78% of readers stop immediately once they sense something is the work a stochastic parrot, and 71% go on to blacklist the writer. It’s not the polish, it’s the authenticity that counts. Readers will always prefer the clumsy voice of the author over the gloss of an LLM’s whispering. Cantrill reports good success with using Pangram to detect AI writing. I confess I’m a bit wary, do I really trust anyone’s judgment to disentangle LLM-voice from changes in generation and context? Maybe people steeped in Silicon Valley culture authentically speak in LLM-voice these days. Sadly for them, to be misclassified by their readers as an LLM is just as bad as using the damn things. ❄ ❄ ❄ ❄ ❄ One of the dirty non-secrets about LLMs is that they were trained on a vast corpus of writing, without consulting the authors of that writing to see if they were cool with it. Individual authors like me can’t do a great deal about it, so are easy to ignore, but music companies aren’t exactly known for taking this kind of thing lying down. So they are suing over the use song lyrics for LLM training . Sony Music Publishing and Warner Chappell, music publishers who manage the copyright of songs on behalf of songwriters and composers, are seeking damages for alleged misuse of “tens of thousands” of copyrighted works by Anthropic. […] The plaintiffs claim they are victims of “one of the largest and most blatant ongoing thefts of intellectual property in history”. Looking at it a broader societal point of view, there is an argument that the benefits of LLMs could be worth far more than any losses to us authors. But we should not forget that these tools are built on a foundation they used without our consent, and that should be taken into account as we regulate these tools and the fruits they provide. ❄ ❄ ❄ ❄ ❄ Steve Yegge: All models, no matter how smart, will eventually build systems that they can no longer understand or maintain, if you let them. Fable 5 finally outbuilt itself, and flailed on me for a week. Fable 5.1 looks like it will fix it. For now. But you have to keep an iron grip on system size, or it’ll run away from you. ❄ ❄ ❄ ❄ ❄ I was going through some slightly-related work and discovered that the Creating Passionate Users blog had disappeared from the internet (and has been gone since maybe a year ago). For those who don’t know, Creating Passionate Users was one of the treasures of the Golden Age of internet blogging. It was the work of Kathy Sierra , also known for co-creating the “Head First” series of computer books. It talked about user experience, and remains some of the best writing on the topic, full of sparkling insights that greatly influenced my thinking, as well as many folks more engaged on user-experience work. Sadly not just was the blog ahead of time in its content, it was a harbinger of the darker side of the internet, as Kathy came under attack from a particularly virulent form of Net Nastiness . That led her to retreat from active participation on the web, and we’ve missed her ever since. Fortunately the Wayback Machine did its great duty, and we can still read its snapshot . I’ve often thought that, if I had a clone to spare, I’d like to create a guided tour of Creating Passionate Users to help readers today read that excellent material. (And if you’re reading this Kathy, and want it still hosted on the web, I’d be delighted to.) ❄ ❄ ❄ ❄ ❄ Simon Willison: “I don’t know the answer myself, but I asked a blowhard I know and he took a wild guess, here’s what he said: “ How I interpret pasted replies from an LLM in online conversations ❄ ❄ ❄ ❄ ❄ Jessica Kerr loves the feeling of being part of a team of people that learns from each other and from the codebase they are building as extensions of themselves - she incorporates the term symmathesy for this: a learning system composed of learning parts (both the people and the code). “But now agents!” There was a turning point last year where I noticed that not only are they useful, it is irresponsible not to use them, at least in conjunction with my own code. They’re more thorough, as well as faster. How am I supposed to be responsible for this system, when I don’t understand each line of code? <pAmong European Companies That Use a CDN, Nearly 9 in 10 Use Cloudflare