Publish or Perish: AI vs Human

Are frontier AI models beginning to write papers better than humans? I ran a comparison, and it's left me more uncertain than ever.

Image: Midjourney

Despite attempts to drag academia kicking and screaming into the 21st century, peer reviewed publications and citation are still a gold standard against which many of us are judged. And so it’s no surprise therefore that there’s been a surge in academics using AI to churn out papers by the dozen in the hope of gaming the system — or at least giving them more of a competitive edge.

But are emerging AI models actually any good at writing academic papers?

I’ve been working extensively with various AI models as writing partners for over a year now, and have been on something of a roller coaster of a ride when it comes to my thinking around what they can and cannot do. A year ago for instance, I was being blown away by the seeming-eloquence of models like Anthropic’s Claude 4.5. More recently though, I’ve found the prose of even the most advanced models grating, while being superficially profound yet substantively hollow.

And yet, I’m seeing a growing wave of opinion that suggests that I’m an outlier here — especially when it comes to writing as academic papers. And so I thought I’d try an experiment (admittedly with an “n” of one), and pit myself against Anthropic’s latest model — Fable 5.

If you’ve been reading my recent articles you’ll be familiar with the genesis of the resulting comparison. After an initial foray into using Fable, I sat down to see how good it was at writing a paper that built on an extended my own work.

By focusing on an area that I’m deeply familiar with (and am responsible for developing the underlying concepts in) I was able to assess both Fable’s ability to make a genuine contribution to the field, and to write it up in a paper that could conceivably pass peer review.

I was impressed with the results — very impressed in fact. But I still found the paper somewhat lacking. Even though it was in an area I am intimately familiar with, I found it hard work reading it and increasingly frustrating — not so much because of what Fable was trying to say, but how it said it. It just felt like really poorly executed academic writing, and more like something a machine would write that has studied the form of the “academic paper,” but has no idea what the experience of reading one is like to a real person.

And so I thought I’d “improve” on Fable’s paper by editing it, line by line, and transform it from what I though was a jarring “cargo-cult” of a paper to something much more palatable.¹

The resulting “Maynard” version of Fable’s paper was not my best work I have to admit. But it was, in my estimation, OK. And I was pretty happy that it represented a substantial improvement on the AI version. Until, that is, I made the mistake of asking Fable for its opinion. And then went on to ask a whole lot of other AIs — all of which pretty much told me the same thing: that my version was far and away the worst of the two.

To make matters worse, the more I explored emerging attitudes toward AI-generated academic writing online, the more I began to wonder if it was me who was completely out of touch with what is considered today to constitutes good writing — an anachronism from a previous age, lost in his own myopic hubris.

OK, so I actually don’t believe this. Hubris though it may be, I’d like to think that nearly 40 years of experience has taught me something. But I am fascinated by how divergent opinions are becoming on what constitutes “good” academic writing, given the increasing prevalence of AI-generated content.

And so, rather than rely on my own assessment, I thought I’d seek the opinion of others (including you, the reader) on the two versions of the paper — the “Fable” one and the “Maynard” edit.

Each version of the paper is downloadable below, and in two formats — a markdown file that is easy to give directly to an AI, and a PDF, which is easier on the human eye.

Each paper is also anonymized, so that any AI you pass these to will have minimal clues as to which is which (and please do feel free to do this).

It is, admittedly, not that hard to work out which is mine and which is Fables if you read my previous posts. But for the sake of this article they are “Rabbit” and “Marmoset,” and I’m not saying here which is which.

The Challenge

So here’s the challenge: Download each and either read them, skim them, or feed them to your favorite AI. Then, let me know (in the comments or otherwise) which one you think is better — and why (especially as “better” is such a subjective term).

• The Marmoset version: PDF. Markdown.

• The Rabbit version: PDF. Markdown.

And if you are interested in my own thoughts here, read on:

My thoughts …

I was in two minds whether to leave this article here and wait to see what the reactions were, or whether to add my own initial thoughts on the comparison and what it reveals. On balance I decided to go ahead anyway as this has been occupying my thoughts.

To start with, I wanted to acknowledge that each of these papers represents considerable human effort. The original Fable paper builds on decades of my own work, and the process of working with the LLM and providing feedback on several iterations of the paper took a couple of days. Then the second, extensively edited version of the paper, took a further three days to complete as I went through it line by line. As a result, his is very much an exercise in human-AI collaboration, where the human bit isn’t insubstantial in either case.

In my version of the paper, I changed relatively little of the substance. There were areas where I had a better grasp of the material than Fable (not surprisingly as this is the result of my own work), and I edited the paper accordingly. But overall the ideas, analysis and insights that Fable generated remain intact.

What I did change though was how the information is presented, and especially the narrative form of the paper. This is where I have (or thought I had) a pretty decent sense of where a sentence or paragraph can be technically accurate but narratively ineffective, and how this can be remedied. As a result, the “Maynard” version of the paper reads in a way that makes far more sense to me as a writer.

Still, as I noted above, my version of the paper isn’t great. But it is something that I’d be happy to put my name to (it is, in fact, available as a preprint, as I believe it makes a useful contribution to the field of AI safety/risk independently to this experiment).

But then I fed both anonymized versions to Fable 5. And then to ChatGPT, Gemini, and Grok.

The verdict in each case: Fable’s version is the sharper, better, more refined, and more publishable version of the two.

Ouch!

I asked Fable to explain why. It claimed that the preferred version (it didn’t know it was its own):

• Was more compressed and therefore concise (it considered this a good thing);

• Had far less metaphorical “throat-clearing” (something else it thought was good — and a bit awkward for me);

• Didn’t over-labor the arguments (which meant, I discovered, that it felt the reader should fully understand its arguments from single well crafted sentences augmented with citations, and without any further contextualization or explanation); and

• Ensured there was a “mic drop” moment (Fable’s own words) at the end of each paragraph. In fact there was a note of simulated admiration at just how jaw-droppingly profound each of these revelations was — as if the reader would immediately feel the scales dropping from their eyes, paragraph after paragraph after paragraph …

What I find interesting here is that every one of these attributes that Fable applauded is one that, in my mind, leads to ineffective writing. They may make sense technically if you don’t know much about what it feels like to experience reading something as a human. But each one has a tendency to hinder the process of enabling the reader to get a glimpse into the mind of the writer.

And my sense here is that, while an LLM like Fable can study what it believes to be examples of good human writing, it cannot replicate faithfully a process that draws on our lived experience, our biological heritage, and our emotional and cognitive responses, as much as the technical meaning behind the words. Rather, because it cannot understand what it feels like to read as a human, all it can do is emulate what it infers from very incomplete data that reflect something that lies beyond an LLMs ken — at least for now.

And yet, it seems that many AI models are secure in their own understanding of what they think is good writing. And it’s this standard that they are using to assess what they generate themselves, and how they assess writing that’s passed to them for critique.

If this is true, one likely upshot is that, the more that people rely on LLMs to provide critical feedback on their own work — or to actually edit it — the more we’ll see papers emerging that match an LLM-view of what good writing is, rather than a human one.

This, to me, is worrying. But it gets worse.

I’m now seeing growing evidence that people are believing the LLMs rather than what people claim is effective writing.

I am encountering an increasing number of people who will claim a paper is good because an AI agent said so. And they get even more certain of this after they have unleashed a whole army of AI agent reviewers using advanced models, that also tell them it’s a solid paper.

the implication, of course, is that the AIs we have trained to “think” like us are now beginning to train us to think like them.

This worries me. But it also rises a niggling doubt in my mind. And that’s where we get back to me wondering if I am, in fact, an anachronism. Because what if I’m wrong and the LLMs are right? What if my idea of effective writing belongs to another era, and frontier AIs, with their access to more data than I could possibly hope to assimilate, are learning the true essence of great communication?

There’s a change of course that the LLMs are right and I’m wrong here.

At the same time, if our ideas of what constitutes effective writing are being informed by machines that do not know what it feels like to read as a human, and are incapable of emulating the full human experience of communication through the written word, what does that mean for the future — and the future of being human?

And this is why I’m so interested to see how others respond to the two papers above.

---

Update: I had meant to add this before pressing publish but it completely slipped the net:

I’m sure many people reading this will say that the problem here is using the wrong writing skill with Fable, or not training it sufficiently on my own voice. However I do not believe that this is the issue I was facing, and for two reasons.

The first is that I have been training AI models on my writing style and voice for a long time now - and successfully. Yet in this case the training and feedback always led to disappointingly performative and hollow results.

The second is that I quizzed Fable on its performance quite deeply, asking why it could not learn from my feedback and up its game, despite me trying every trick I knew. It eventually admitted that it has internal “templates” for want of a better word on what good academic writing is, and no matter how much fine tuning I attempted, these would always dominate how it wrote. This is where the traits I mention in the article above reside.

Fable also noted that it had trained on papers that were prominent and highly cited, and not necessarily papers that are well written - it’s training relies on a proxy for what is good that may have been misleading.

The result is that Fable, and, I suspect, other models, seem hard coded to write academic papers in ways that they think are appropriate, but that do not necessarily connect with human readers.

This may well change over time. But my sense is that this goes far more deeper than being fixable through fine tuning. In the meantime. If you use AI to extensively write for you and critically review its work or yours, you may want to be aware that the machine may not be your best guide. Unless, of course, LLM-style becomes the de facto standard!

1. “Cargo-cult” of a paper referring to something that follows the form of an academic paper without an underlying understanding of how the paper works in practice, and thus seems to be based on the assumption that, if the form is right, meaning will automatically flow.