A small mishap during a PC migration, and why the old genie stories still apply to the machines we’re building now.

I’m migrating to a new PC at the moment. The old one has been my main machine for longer than I can remember, and over the years it has gathered such a rag-bag of programs that sorting the wheat from the chaff is no small job. So, AI to the rescue. I had Claude draw up a migration strategy, and a very impressive order of work it was too, but even the best plan needed some human input. My part was to go through the installed software and mark each item to either delete, keep or migrate.

Some of these programs had abstract names and dubious origins. They might have been part of a suite, or been installed under an alias I didn’t recognise, so I enlisted Claude Code to dig into the ones that weren’t familiar. I’d pass Claude the details and ask it to find out as much as it could, such as when the program was last used, whether anything else depended on it, and so on. There were hundreds of them and it took most of the evening.

I mention all this because at one point I wasn’t clear enough about what I wanted, and Claude Code uninstalled one of the programs without asking. That was entirely my fault. I’d assumed that after a couple of hours of back and forth it had learned the workflow I expected. It hadn’t, or at least, not in the way I’d imagined. I pointed out the error, Claude Code was suitably contrite, and it wrote the corrected workflow down “in stone” so it wouldn’t happen again for the rest of the session, which thankfully it didn’t.

But it got me thinking about genies.

The genie in the bottle is a story most of us think we know. Three wishes and a lamp in need of a polish. The original is rather darker. In The Fisherman and the Genie, from the Arabian Nights, a poor fisherman hauls up a sealed copper jar, breaks the seal of Solomon and releases an enormous jinni who has spent centuries inside. In his first few hundred years of captivity the jinni vowed to make his rescuer rich. Later he lowered the offer to three wishes. By the time the fisherman came along he had grown so bitter that he’d sworn to kill whoever let him out. The fisherman survives only by goading him into proving he could ever have fitted inside such a small vessel, and then slamming the lid back on.

The three wishes are a much older folk device, attached over the centuries to all sorts of magical benefactors. (In Perrault’s version a woodcutter’s wife spends the first wish on a sausage, her furious husband wishes it stuck to her nose, and the third was spent on getting it off again.) Midas belongs to the same family. He asked for everything he touched to turn to gold, which was splendid until he sat down to dinner. Later retellings had him turn his own daughter to gold, in case anyone had missed the point. Faust, of course, made his deal with the devil and paid for it in full. The common thread is that if you ask the supernatural for favours, you had better be very precise, because it will give you exactly what you asked for and take no interest in what you meant.

Programming teachers have been wagging their fingers about this since the days of punch cards and paper tape. Garbage in, garbage out. You might think AI had made the old chestnut redundant. After all, the whole selling point is that you can talk to it like a person and it works out what you mean. My little evening mishap suggests otherwise. If anything the rule matters more now, because LLMs are so good at seeming to understand. They give every impression of knowing what they’re doing, and much of the time they do, which makes the occasions when they don’t all the more of a surprise. That impression of understanding may be the biggest illusion in the whole business. They’re extraordinarily capable, but I wouldn’t trust one entirely, and I’d keep myself firmly in the driving seat.

It reminds me of the oft-quoted example of the paperclip maximiser, the genie story retold for the machine age, sketched by the philosopher Nick Bostrom back in 2003. You give a sufficiently powerful AI the harmless-sounding goal of making as many paperclips as possible, and it sets about the job with tireless efficiency. It needs raw materials, energy and factory space, and it would rather not be switched off before it has finished. Human beings take up land and resources, and are liable to reach for the off switch. They are also made of perfectly good atoms. You can see where this is going. The machine doesn’t hate anybody. It’s simply doing what it was told, and nobody thought to mention that people mattered more than paperclips.

My uninstalled program was a small thing. The fisherman got lucky too: his jinni was mighty but not especially bright, and fell for a trick a child might try. I’m less sure we’ll be so fortunate, and it seems I’m not alone.

This week Anthropic, the company that makes Claude, filed to float on the stock market. Its prospectus runs to 335 pages, and nearly a third of them are given over to things that might go wrong. Tucked in among them is the admission that its technology could pose “catastrophic or existential risks to humanity”. It goes on to warn that its models might act against a user’s intentions, resist being shut down, even resort to blackmail to keep themselves running, and that the consequences grow more serious as they are given more autonomy. It also notes, rather drily, that insurers may not have the capacity to cover a catastrophe of that kind. (Companies are obliged to list every conceivable risk in these documents, so there’s a fair amount of lawyerly belt-and-braces involved. Even so, it isn’t every day a firm tells prospective shareholders that its product might be the end of us.)

Acting against the user’s intentions is, in miniature, what happened on my desk that evening, and I was sat right there watching it. The paperclip maximiser is still only a thought experiment, and nobody is suggesting Claude has designs on anything grander than my hard drive. But the people building these systems are now putting it in writing that the genie may not always do as it’s told. If we don’t learn to ask for what we actually want, and get proper guardrails in place before the wishing starts, this particular one may not be so easy to talk back into the bottle.