I cracked and wrote an academic paper using AI. Here's what I learned ...
I deeply dislike AI-generated academic slop. But I'm curious about how AI can genuinely accelerate legitimate research. So I took the plunge ...
Just under a year ago I wrote about how I used AI to write a full PhD thesis. Using OpenAI’s Deep Research model of the time, I was able to come up with a passable dissertation within a couple of days.
It was far from perfect, and the resulting dissertation definitely benefitted from being a synthesis of existing ideas rather than representing original research. But it did demonstrate how the combination of combinatorial discovery (putting existing knowledge together in new ways), slick writing, and blistering speeds, could enable large language model-based AI to massively accelerate the process of academic scholarship and writing.
Since then, there’s a been a growing wave of AI-generated and AI-assisted academic papers hitting journals and preprint services like arXiv. It’s a trend that is both hinting at new forms of research and discovery, and threatening to overwhelm academic literature with a tsunami of pseudo-intellectual AI slop.1
This is, I must confess, a trend that worries me. There’s a growing temptation for academics whose careers depend on publications to churn out AI-written papers that have little intrinsic value, but get published because they look the part to an uncritical eye. And yet despite the AI slop that we’re already seeing here, there are growing indications that foundational and frontier AI models can be highly effective accelerators of research and discovery if used thoughtfully.2
Because of this — and despite my reservations — I was curious to get a better sense of how useful emerging conversational AI platforms are in academic research and publication. And last week’s post on the possibility that AI is a Cognitive Trojan Horse gave me the perfect excuse to explore this further.
And so I set about “writing” my first full-blown academic paper with AI.3
The Paper
If you’ve read my previous post on the AI cognitive Trojan Horse, you’ll know that it was a reasonably well researched article, and one that started to unpack whether there are potential mechanisms by which conversational AI may slip by our epistemic vigilance mechanisms — the mechanisms by which we decide whether to critically examine material we are exposed to, or whether we trust it.
But it was still just a Substack post, and not a rigorously researched academic paper.
I was sufficiently intrigued by the ideas that emerged from it though that I set about digging deeper — and this is where I decided to use this as an excuse to flex my AI-assisted scholarly muscles.
The process I followed is described below. I’ve included it here as I think that how AI is being used in contexts like this is as important — if not more-so — than what is being produced. And while there are a number of academics beginning to document how they’re using AI in ways like this, I believe we’re still early enough along the learning curve that sharing approaches to using AI in academic research and writing are useful.
But to the paper.
To extend the research that started with the original post, I carried out a deep (and AI-assisted) literature search across multiple disciplines to test the ideas explored in that post. This was then used as the basis for developing a more robust understanding of the intersection between conversational AI and epistemic vigilance.
The result was a paper that was published a few days ago on the preprint site arXiv — and after just a couple of days of AI-accelerated research and writing.
The paper — The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance (available here) — takes a slightly different direction from the original post (driven by the research), and introduces a couple of new and (I believe) novel ideas, including the concept of “honest non-signals.”
Honest non-signals in this case are defined as genuine characteristics of conversational AI (including fluency, helpfulness, and apparent disinterest) that appear to — but do not — carry tacit information that equivalent characteristics would carry in a human communicator. Rather, because they mimic characteristics that are often associated with trustworthiness — not out of maliciousness but simply because that’s the nature of LLM-based AI models — these models have the capacity to slip through our epistemic vigilance systems. The “honesty” comes in here because these are characteristics of the LLM, and not intended to be deceptive.
As the paper notes,
The fluency is real, but it does not indicate the organized knowledge that produces fluency in humans. Similarly the helpfulness is real, but it does not indicate the benevolent motivation that produces helpfulness in humans. And the lack of apparent self-interest is real, but it does not indicate trustworthiness in any meaningful sense—it indicates the absence of interests altogether.
In other words, these are genuine signals that nevertheless lack the content that we infer from them, because we are used to such signals coming from other humans.
The paper goes on to note,
The concern, then, is not that AI systems present false cues that vigilance should detect but fails to. It is that they present a configuration of genuine characteristics that falls outside the parameter space vigilance mechanisms are calibrated to evaluate. Here, an immune system analogy is instructive: a novel pathogen may evade detection not because the immune system is weak, but because the pathogen presents molecular signatures for which no template exists. The immune system works exactly as designed—and fails precisely because of that.
The paper continues by exploring mechanisms underpinning how conversational AI might bypass our epistemic vigilance defenses, and the possible consequences of this. And it concludes that the “intervention space” around ensuring AI safety may need to extend from improving accuracy, reducing hallucinations, and increasing alignment, to designing systems that present more calibrated trust-cues.
The result was a process and a resulting product that I found to be genuinely insightful and generative, and one that was effective because of how I used AI — not as a “slop prop,” but as a powerful research tool that extended what I was able to do, without supplanting my own intellectual contributions.
And a lot of this came down to the process that I followed.
The Process
The genesis of the question that prompted the paper came from a keynote I gave at at OEB 2025 Berlin. In it — as I mention at the start of the previous post, I rather provocatively asked the audience “Is AI a cognitive Trojan Horse?”
This question emerged from my evolving thinking around how highly attractive — seductive even — conversational AI could potentially circumvent our defenses because it was tuned to hit all of our “I want to trust and believe you” cognitive buttons. But back in December when I gave the keynote, this was little more than a provocative idea.
The idea was fleshed out in the research that led to last week’s Substack post of the same name. This was a mix of hypotheses emerging from my own research and some initial brainstorming with Anthropic’s Claude — but it was still primarily based on my own thinking. And it was still relatively underdeveloped.
It was the combination of intriguing ideas at this point, and the knowledge that I wanted to dive deeper to test these, that led to me realizing that this was an intriguing test case for a short AI-assisted research project — albeit one that was focused on developing rigorous ideas and concepts rather than running experiments.
And this is where the process began.
The first step in the process was a long conversation with Claude (using Opus 4.5) on what stood the test of a deep and cross-disciplinary literature review in the original Substack post, and what did not. This led to me iteratively checking relevant papers and working with Claude to get a better sense of how and where conversational AI might interact with our epistemic vigilance mechanisms.
The upshot of these early conversations was a request to Claude to carry out a deep research dive into what we’d discussed and unearthed, and to produce a detailed and grounded analysis of the ideas and hypotheses, along with links to relevant papers — all of which were subsequently downloaded for later reference.
At this point it was apparent that some of my initial ideas held up to scrutiny, while some of them needed adjusting and rethinking. Working with Claude also began to unearth intriguing new connections and ideas.
The next step was to refine the ideas that were beginning to emerge from the literature, and to start drafting a paper that pulled them all together. For this I set up a new project in Claude that was populated with many of the key papers that had previously been identified as being relevant (frustratingly there were too many to upload them all).
After further testing and refining the emerging insights and identifying a core set of ideas and arguments, I asked Claude to draft a first version of an academic paper that captured these (this and all subsequent drafts were produced as formatted Word documents).
It was awful!
Reading it felt like reading the first paper from a new PhD student where they still believed academic-sounding language was the equivalent of robust scholarship. The language that Claude used sounded academic at first blush, but was ultimately superficial and hollow — fluff masquerading as substance.
I started line-editing the draft paper, but gave up after the second page and a bunch of very pointed comments. Instead of continuing, I gave the partially annotated document back to Claude, let the LLM know in no uncertain terms what I thought of its attempt, and provided rather unvarnished instructions on what I expected of it — especially when it came to scholarship and academic rigor.
The next draft was substantially better.
Unlike the first draft, there were interesting new ideas in the second version that were well developed and justified, together with well-argued concepts that built on and extended my initial thinking. In fact it was so much improved that, rather than line edit myself, I went straight to “peer review” — using a new Claude session in this case as my highly critical academic peer reviewer.4
The four pages of review comments were critical but constructive — and from my read of the draft, very much on point.
I gave these back to Claude within the research project, and asked for a revised draft. What came back was better still, and was responsive to the feedback. But it still wasn’t where I felt the paper needed to be. And so we went through a second round of Claude as a critical peer reviewer.
Again, my assessment of the feedback was that it was on point. And so once again I provided the feedback to Claude and asked for a new draft in response.
The result was a draft paper that was good. Very good in fact.
At this point I thought we’d progressed to the point that I could once again take over the editorial reigns and dive in with detailed line edits.
These were substantive, and addressed the core concepts in the paper and the evidence supporting them, as well as what wasn’t working and what needed more work more generally. My feedback was very much in line with what I would have provided an accomplished grad student co-author.
Following this feedback (all using comments and track changes in a Word doc) Claude and I went through one further draft-line edit-draft cycle before I felt that the manuscript was robust enough for final fact checking and editing.
My next step was to download all available cited works (all but two were available — one that wasn’t was a paper I am familiar with, the other was a book that I obtained separately) and carefully check each source and any claims based on it. For this I used a combination of good old fashioned human scholarship with repeated checks using fresh chats with Claude.
Finally, the manuscript underwent a final set of checks and edits by myself to make sure everything held together and made sense, before submitting it to arXiv.
The whole process took around two days. For me it was a substantial intellectual and editorial lift — this was not a “press and post” paper by any means. At the same time, 2 days from idea to preprint is a crazily short period of time for an academic paper.
To have done all of this work manually would have taken weeks. And even then, I’m not convinced that I’d have produced something as robust and useful as what emerged from the AI-assisted process.
The Reflection
So what was the upshot of this exercise for me?
First off, it’s easy for me to see from this experience how using AI can substantially elevate the speed and quality of scholarship. Using Claude as a research and writing tool vastly accelerated the rate at which I could work, without me feeling as if I’d lost intellectual control.
In many ways, the process mimicked collaborating with a talented grad student or postdoc. The difference, of course, being that the AI could draw on vastly more cross-disciplinary resources and insights than any grad student could, and do so much, much faster than a human collaborator.
But this also left me feeling slightly uneasy. If I was working with a human collaborator, their name would be on the paper and their intellectual contribution acknowledged. And without a doubt, there was a form of intellectual contribution from Claude here — albeit one that was realized through my active involvement. For instance, the concept of honest non-signals came from Claude, as did the development and refinement of the various mechanisms by which conversational AI might slip by our epistemic vigilance mechanisms.
On the other hand, the resulting paper also has my intellectual fingerprints all over it. In some cases I provided a direct steer to Claude — the analogy with human immune responses for instance, and the exploration of how this work aligns with other approaches to AI risks and safety.
Objectively, and if seen purely through the lens of knowledge contributions, the paper makes a contribution to thinking and understanding around AI-human interactions. And this is a contribution that I believe is valuable.
More subjectively though, it’s a contribution that I can’t take full credit for. And herein lies a tension between academic outputs as self-serving indicators of success, and outward-facing sources of public good.
This is perhaps one of my biggest takeaways from the exercise. Using AI as an academic profile-padder is something I still find distasteful — even though it’s never been easier to churn out new papers by the dozen using artificial intelligence. And yet, AI-assisted discovery and insights as a public good feels like something we should be embracing … as long as we can work out how to ensure the latter without the hollow self-aggrandizement of the former.
That said, I do have one further niggling worry about this whole exercise. And that is this: If AI is so good at evading our epistemic vigilance mechanisms, how do I know I’m not an unwitting victim here?
And maybe this is where we still very much need a whole community of humans-in-the-loop as AI-assisted research and AI-generated papers become increasingly prevalent — all operating as a collective form of epistemic vigilance!
This, it seems, would make for a valuable follow-on research project.
Claude? …
Just this past week a new paper in Nature Portfolio examined the impact of AI on the impact of scientists’ work. While the emphasis of the paper is on scientific discovery, the authors noted that “[r]ecent developments in large language models have also become increasingly used to assist scientific writing.” They also note that the use of LLMs “raise concerns about weakened confidence in AI-generated content.” Hao, Q., Xu, F., Li, Y. et al. Artificial intelligence tools expand scientists’ impact but contract science’s focus. Nature (2026). https://doi.org/10.1038/s41586-025-09922-y
Hao and colleagues in the Nature Portfolio paper above found that over the past few decades the use of AI in scientific research has substantially increased the impact of scientists, and that generative AI seems to be accelerating this. At the same time they found that use of AI is narrowing the focus of research and discovery, and reducing scientific engagement. The paper was researched using AI.
Technically it’s the second. I have a rather cheeky and 100% AI-written paper that was submitted to arXiv before the one discussed here. However, given it’s rather unconventional nature, it’s still in a holding pattern with the moderators there!
One criticism at this point is that using Claude to critique Claude would seem rather circular and incestuous. And indeed there is a danger that inherent biases in the model lead to weak ideas being reinforced. However, my experience is that these models are at a level of sophistication that a new session has sufficient independence when augmented by human expert insight to provide valuable critical feedback.