AI is bad at writing — removing em dashes won’t fix it

And neither will the next model.

If you’re worried about the state of AI writing now, I have sobering news. AI slop is flooding our inboxes, newsfeeds, and publications, fattening our brains with empty calories, and it’s only going to get worse.

Some argue this is just temporary — improvements are coming in GPT-6 or GPT-7, we’re told. But, unlike fields where AI has shown marked improvement over the last few years, writing has, if anything, gotten worse. When we and others compare the latest models writing with earlier versions, we find the newer ones use bigger words and more complicated sentence structures, yet communicate far less clearly with fewer original ideas. One researcher described the new models to me as “very educated and very, very drunk.”

I worked at one of the major AI labs and run a company that helps writers; I’ve spoken to researchers across the industry and academia. Those conversations revealed many issues, but primarily four intractable obstacles that practically guarantee AI writing will just continue to get worse: vanishing training data, limited visibility into the process of writing, regression towards the average, and the inherent subjectivity of writing.

Illustration by Edward Ardizzone, as found on the 200 cover of Books and Company, Issue Eight

1. We’re running out of good data to train on

During the pre-training stage of development, Large Language Models (LLMs) read all the info they can find (the internet and beyond), try to understand the relationship between every single word they process, and an LLM is born. A literal Large Language Model.

With each new model, AI labs seek out new sources of data to facilitate more advanced training. But, even if you’re inhaling the entire internet, new writing is scarce at the scale needed for model training. In fact, the majority of ‘new’ things on the internet are now slop, which creates wild model training challenges of its own.

That’s why human writing is often called the new oil. The glut of fresh writing to train on is why major AI labs are buying up old out-of-print books, butchering them so they can process the tender cuts of words within. But where will GPT-10 get its new writing from?

2. Most of writing is revision

Training data for writing, contrary to areas like coding and mathematics, only has ‘finished’ pieces and lacks the ‘process’ data — the drafting and editing and revision steps that lead to a compelling piece of writing. So while we often focus on the tactical ‘stylistic’ tells of LLM writing, LLMs struggle even more deeply with the strategic elements — bland ideas, generic structure and flow, everything too consistent and unsurprising.

An LLM can’t understand the iterations that lead to a strongly-framed thesis — because that isn’t documented anywhere (yet). To evoke Joan Didion’s ‘writing is thinking’ truism, as much as we want to anthropomorphise models, LLMs simply don’t have the data to ‘think’ through writing the way a human writer would. That’s what makes reading generative text feel like feasting on empty calories.

3. Models are trained to give the most likely response

Compounding that, LLMs are trained with a tremendous amount of reinforcement learning. Meaning the model might run through thousands of answers to a prompt, rank each one, and do it all over again (billions of times). When you then ask that model to write something, it serves up the highest scoring sequence. That’s why when you look at studies of writing, LLMs use more sophisticated language and employ more complex grammatical structures, but the breadth of ideas espoused is exceptionally narrow. This works to their advantage in things like providing factual answers (though they still are known to hallucinate), but that prioritisation dramatically degrades their ability to showcase critical writing skills.

And a weird thing happens when we (people!) read the output. It anchors us to its narrow set of ideas. Studies have shown that we become less creative when we’re exposed to an LLMs output; an LLM first draft means less thinking and new ideas go into each subsequent one. So, after seeing this predictable probabilistic writing, we think (and write) more predictably as well.

4. Writing isn’t binary

Writing is not code. It’s not verifiable. Even though generative code or mathematics is often clumsy or inelegant, it is binary. Code reliably runs or it doesn’t. A proof is true or it’s not. The same is not true of a book or essay or email.

Which gets us to what pretty much everyone agrees is the biggest single issue — what even is ‘good writing’?

Part of this is the ineffable quality of ‘taste’ (I think most beautifully explored by Celine Nguyen here). But a further profound challenge is that an important part of good writing is variety, and you can’t evaluate variety when assessing a single output.

Training LLMs is like training a pet — they get a reward when they do something good. At the post-training stage, thousands of people are employed to rate sample output from new models. This is used to sharpen or correct the models before you use them. Some things are easy to grade, like factuality. Writing is not.

If you’re one of the thousands of post-trainers judging the output of a model one paragraph at a time, how do you assess that? How do you balance the models’ bias for clarity and conformity against the vibrant and varied needs of writing outputs for a variety of contexts? Ocean Vuong wrote that, as a writer, your job is to thwart convention. How does a probabilistic model do that, when it’s graded one paragraph at a time?

Where do we go from here?

Even with LLMs’ growing context windows, increasingly-intrusive datasets and ever-more-powerful models, AIs still won’t become good writers. Because writing is not an output to be produced; it’s thinking, it’s connection, and it’s trust.

I was speaking to an expert recently who said his greatest fear wasn’t that LLMs would get better at writing, but that people would stop caring that they suck (when we write, and when we read). He sees this in his inbox, and on the sites he visits. He feels it himself, in the temptation to generate something rather than go through the process of crafting his thoughts into words.

Writing can be hard, but that’s what makes it valuable. It’s a process of putting your thoughts into words for other people. Choosing to write, and seeking out things that other people wrote, isn’t masochistic; it’s a recognition that it’s worth it, and the labour we and others are ‘saving’ with generative AI carries a broader cost that is disastrously high.

But there are reasons for hope. We see New York City banning AI in schools, and everyone satisfyingly shaming a billionaire (and the Wall Street Journal) for publishing unlabelled slop. We see the rise of the trust-based newsletter economy, and a society-level discussion of punctuation tells. We can recognise our own value in the words we put on a page, and the choices we make when reading.

LLMs wouldn’t exist if humans hadn’t written down everything we thought for the last 5000 years. We can’t afford to stop now.

Acknowledgements

This was the results of so so so many conversations and shared drafts, starting with a super-detailed 3000-word 10-item list. Thank you especially to Steve, Ryan, Jani, Hannah, Katy, Amy, Robbie, Lydia, two Jons, and many others. All were helpful in, I think, helping make this a clearer representation of my thoughts. And, of course, written in Marker.