My (Tentative) Solution to the AI Writing Problem

I asked a farmer

(I originally wrote this post back in January, and then for some reason totally forgot about it. I was reminded of it recently with the whole Pangram-Substack collaboration, and have lightly edited it to make it more relevant to today.)

One of my friends works in agriculture, and he and I were discussing the problem with identifying organic food. The thing is that (to hear him tell it), organic food is hard to distinguish from non-organic food at the point of reception, especially for a consumer. If you take an organic and a non-organic apple, it is pretty hard to tell the difference (you can test them for pesticide traces, but this is fallible). This is because whether a food is organic or not is mostly to do with the process it took to grow it, rather than something specifically to do with the final product. Even if there were good signs at the output stage, it still would not be a dead cert that a given food is organic. In order to avoid this, farms are subject to inspections, so that the actual process can be examiend as well.

It struck me over the course of this chat that we are in a similar situation with AI writing. The fact there are so many challenges online that are essentially “can you guess which of these is AI-generated?” is testament to that. And, not to be cliche, but I suspect these things are only going to get harder to detect. The most recent example of this has been Substack’s partnering with Pangram. I don’t know whether Pangram is reliable - maybe it is and maybe it isn’t. But I suspect that there will be ways of getting around it, and I also do sympathise with the paranoia of non-AI authors worrying about being falsely accused.

So how do we tell “organic” writing from AI writing? I suspect the most promising avenue is at the process level. And I think we have a lot of the tools we need in order to make this happen. I used to tutor students in maths and philosophy (as well as personal statement writing), around the time when AI was first emerging, and I asked them to write any written assignments in Google Docs. The reason for this was because Google Docs automatically saves your version history when you are writing, meaning that you can sort of “watch back” the process of the student writing their essay. I would give explicit instructions to write the entire thing on Google Doc, and not paste it from another word processor. This made it completely trivial to check whether they had used AI, because I could see them write it in the version history, with all the hallmarks of a fallible human typing something out. This became even easier when I had a program that would automatically play me back their writing process. I also write everything in Google Docs (originally because I did not want to pay for a word-processing program) so this is pretty easy to demonstrate. Here is the script to a video I wrote a while ago on The Great Gatsby (obviously sped up a lot).

As you can see, I make stupid typos, and go back to change or rephrase things. There are points where it takes me an embarrassing number of rewrites for a single sentence. There are natural human mistakes that an AI wouldn’t make. You can see me make notes at the bottom of the page for things I am going to write, but had not fully developed yet. You can see me jump around the document to edit things I had written earlier (such as when I rather embarrassingly forgot Jordan Baker’s story about Daisy and Tom’s wedding day, and had to go back and add it in). These are very human things to do. Large Language Models (at least at the moment) cannot imitate this process - they just write things out.

The problem is that you cannot share your version history with another person without also giving them edit access to your Google Doc. This is a pain in the arse, as it means writers cannot just throw up their version history online as proof that AI did not write it. Additionally, if you are anything like me, then the version history itself is full of half-completed thoughts, silly ideas, unfinished threads that later get cut, and, above all, stupid mistakes. All of this means that I don’t massively want to share my version history with the entire world, and I think this would be even more pronounced for people like journalists. Imagine if, in a first draft, a journalist forgot the name of the home secretary. This is the kind of trivial memory slip I have in my writing all the time (and I suspect many of us do), but it would be easy for a rival paper to use this as evidence of incompetence, if the journalist were forced to make the entire messy drafting process public.

So here is what I propose. We need some kind of independent service that is willing to check the version history of these documents, and can give it a “human-certified” stamp in the same way that we stamp things as organic. Then we could feel confident we were reading something that is human written. I am sure there are ways of automating this checking process (for example, alongside showing me a video of my students’ writing process, my checker highlights any suspicious copy-pastes, or other odd behaviour). But personally, I would prefer that there also be a human element to the review, if for nothing other than the fact that in uploading such version histories, we would be creating a data set on which you presumably could train an AI to imitate the human writing process.

This would not be perfect. It would not, for example, prevent someone from generating a piece of writing with an LLM, and then re-tying it out into the Google Doc. But I am working on a key assumption in this proposal: AI-generated writing is largely done to avoid expending time and effort, and the trouble it would take to mindlessly re-type something generated by a machine is the hurdle someone would then need to jump over to pass their writing off as human-written. I suspect this would be more of an issue for shorter pieces of writing, since re-typing out a thousand words may not be that much of a deterrent, but re-typing out 5,000, 10,000, or 60-odd-thousand in the case of a whole book, sounds so mind-numbing that I think it would put most people off. At that point, they may as well write the thing themselves. To get around this checker, I think you would have to (at the very least) be a very unusual kind of person. Also, we shouldn’t let the perfect be the enemy of the good here. I think this has the potential to be more reliable than AI-detectors that check the end-product, since people can work around those with other LLMs whose job it is to “humanize” AI-generated text, and as LLMs become less distinguishable from human writing, I suspect these will become more difficult to design (not to mention the fact that until very recently a lot of them said that the first page of Pride and Prejudice is 98% AI-generated - Austen you swine!).

The other thing is cost. I don’t know how costly this would be to implement at scale, either in terms of tools or manpower. It used to take me around a minute to check that a student’s essay (normally between 1,000 and 1,500 words) was not AI-generated. But it would take a lot longer to check a whole book draft. To give an idea, my script for The Great Gatsby video was around 13,000 words, and it took about 10 minutes for this program I have to play back the writing process at super-high speed. Ideally, we would want it to be cheap enough so that your average amateur writer can access it, because I wouldn’t want this to turn into something so inaccessible that only already established writers can prove their writing isn’t LLM-generated. This could put us in a “two-tier system” where poorer writers suffer from the additional suspicion that would come from not having a piece of writing be “verified as human”. Nonetheless, I hope these details can be worked out at some point by people who actually know about these things, and not a philosophy nerd with no technical expertise.

I am sure there are lots of problems with this idea (it might turn out to be much easier to train AI to imitate the process of writing than I thought - as I said I know nothing about computers), but I had not seen many people talk about it, despite the widespread worry about not being able to tell AI writing from human writing, so I wanted to throw its hat in the ring as a contender.