AI-accelerated fiction is my game, but it started a lot further back than you might think.

I started fooling around with computers before the Internet was a thing. Back then, we used print magazines and books to find ideas for programs. I stumbled across one once that really captured my impressionable young mind.

It was a “sentence generation program.” I think it had been knocking around since the sixties in one form or another, probably running on mainframes with limited time-sharing resources. Now, on a microcomputer, it was something a budding programmer could really run with.

The basic idea was this. Every sentence in a language can be broken down into components: a noun phrase, a verb phrase, and so on. By identifying the patterns as a series of pathways, then randomly dropping in words, complex grammatically correct sentences could be generated.

While the program in the book was creating random sentences as a proof of concept, I started thinking of ways to make the prose more convincing. First, I increased the types of words and varied the combinations of phrases. Then I tried classifying the word types into categories, so instead of having one list of nouns, I had animal, vegetable, and mineral types. The more I toyed with the complexity, the more real the sentences became.

Then I tried repeating sentence patterns so that they resembled poetry. This was when the whole thing switched from being an experiment to something that entertained people: a machine that wrote poems. Nonsense poetry, but poems nonetheless. My school teacher was very entertained by this and, much to my pride, chose to demonstrate my little program at the school’s open evening for prospective entrants.

I wasn’t there that evening, but the next day the teacher wasn’t happy. Apparently, one of the words that got churned out was “bloody,” which in some contexts is quite poetic. In the context the random poetry delivered it, however, it was very, very rude. So after that, and with the school year coming to a close, the sentence generator project drifted into the background of my life.

I did, however, scribble some notes about the project, slipped them into a book, and promptly forgot all about them. A few years ago, while moving house, I stumbled across that book. I had forgotten about the project, but when I read the notes, one thing leapt off the page. I’d written:

“…this may one day be a way to generate intelligent content if I could figure out a way of using feedback to rewrite the sentences to check for context.”

While I’m not claiming in any way to have foreseen how large language models work, it sounded eerily close to the kind of problem later explored in the 2017 Google paper Attention Is All You Need.

Perhaps I should have stuck with it!