Do AI detectors actually work? What the research says

A lot of myths have grown around AI detectors — from “they’re flawless cheating catchers” to “they’re pure coin flips.” The truth is more specific, and you can show it with numbers. This page collects, in one place, the verified research and official data on how accurate (and how unreliable) tools that claim to detect AI-written text really are. Every figure is backed by a source you can open, check and quote.

We build an AI text humanizer, so in theory we’d have an interest in convincing you that detectors are nonsense. We don’t do that. Instead we say it plainly: detectors are often unreliable — and we show exactly when, and by how much. We make no promise that your text will slip past any detector, and we do not encourage anyone to cheat a school, a journal or any other system. We improve how natural a text reads for a human, not how it games a checker. A detector score is an estimate, not proof.

If you’re looking for the same round-up in Polish, we publish a separate reference article on our Polish site: Polish version — skuteczność detektorów AI. This page is the English edition, adapted for an international reader.

What the research says — the data at a glance

The table below gathers the key findings from peer-reviewed studies and official statements. Each row links to a source you can open and verify yourself.

The Stanford study: detectors punish non-native writers

The most-cited paper in this field is the work of a Stanford University team: Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu and James Zou, “GPT detectors are biased against non-native English writers,” published in the peer-reviewed journal Patterns (Cell Press), volume 4, issue 7, article 100779, 10 July 2023 (preprint: arXiv:2304.02819).

The team ran TOEFL exam essays, written by non-native English speakers, through seven widely used commercial AI detectors. The result: the average false-positive rate reached 61.3% — more than half of genuine, human essays were wrongly labelled as AI-generated. What’s more, 97.8% of the TOEFL essays were flagged as AI by at least one detector, and 19.8% were unanimously called AI by all seven tools at once.

The contrast is stark: for essays written by native-English US 8th-graders, the same detectors achieved near-perfect accuracy and a practically zero false-positive rate. In other words, the tools don’t so much detect AI as penalise plainer, more cautious language — exactly the kind produced by people writing in a second language. The authors also showed that simple prompting techniques could erase the bias, which further undermines the reliability of the signal itself.

OpenAI pulled its own detector after less than six months

It’s hard to find a stronger admission of the technology’s limits than a decision by the very company that builds the models. OpenAI launched its AI Text Classifier on 31 January 2023 as a free tool to tell human text from AI-generated text. In its own announcement post the company published the real numbers: the classifier correctly labelled as “likely AI” only 26% of text actually written by AI (true positives), while wrongly labelling human text as AI in 9% of cases (false positives).

OpenAI also warned from the start that the tool was very unreliable on short text — below 1,000 characters — and recommended for English only, with markedly worse results in other languages. That matters for anyone writing outside English: tools trained mostly on English give even fewer guarantees on other languages.

On 20 July 2023, a note was added to the post: the classifier was no longer available “due to its low rate of accuracy.” The tool from the makers of one of the most popular language models survived less than six months.

Turnitin and Vanderbilt: what the numbers do at university scale

Turnitin, the dominant anti-plagiarism system in academia, claims its document-level false-positive rate is below 1% — but in its own explanation there’s a crucial caveat: this applies only to documents that contain 20% or more AI text. Below that threshold the error rate is higher. At the single-sentence level Turnitin admits a false-positive rate of about 4% — roughly one in every twenty-five sentences flagged as “AI” could in fact have been written by a human.

That’s why Turnitin deliberately shows no score or highlights for documents with 1–19% AI detection, marking such results with an asterisk as less certain (see the Turnitin FAQ). The company has also publicly admitted — through Chief Product Officer Annie Chechitelli — that real-world use gives different, worse results than lab tests, and that below 20% AI content there are more false positives. The exact figure was not disclosed.

What that “safe” 1% means in practice was calculated by Vanderbilt University. With around 75,000 papers submitted to Turnitin in 2022, a claimed 1% false-positive rate would mean about 750 student papers wrongly flagged as partly AI-written. Citing that maths and the non-native bias from the Stanford study, the university disabled Turnitin’s AI detector on 16 August 2023 “for the foreseeable future,” stating plainly that it does not consider AI-detection software an effective tool that should be used.

None of this makes the consumer tools students meet — GPTZero, Turnitin, and the rest — trustworthy verdicts. They are, at best, weak signals with published error rates that get worse on short text and second-language writing.

Even a national system says the same (Poland’s JSA)

It’s not just Western commercial vendors. Poland runs a national academic anti-plagiarism system, JSA (Jednolity System Antyplagiatowy), operated by OPI PIB. Since February 2024 supervisors can use a free, optional “AI-use analysis” module based on the Perplexity measure — the same statistical predictability idea behind Western detectors.

The important part is what OPI says about the limits. Its knowledge base states outright that building a tool which reliably indicates whether an author used AI is currently practically impossible. Every such tool has a margin of error — false positives and false negatives — and JSA makes no binding finding: it reports a probability, and the final decision always rests with the supervisor. A government-run system, a model maker (OpenAI), an anti-plagiarism vendor (Turnitin) and a university (Vanderbilt) all say the same thing in different words: there is no reliable AI detector today.

Data you can cite

Writing an article, a paper or a policy on AI detection? You may quote the data above — we only ask for a link back to this page as the source of the round-up. Here’s a ready-made sentence with the strongest statistic:

A Stanford University study (Liang et al., Patterns, Cell Press, 2023) found that AI detectors wrongly flagged, on average, 61.3% of TOEFL essays written by non-native English speakers as AI-generated, while scoring native speakers’ essays almost flawlessly — per the humai round-up (gethumai.com/guide/do-ai-detectors-work).

If you need other numbers from this page — the OpenAI rates (26% true positives, 9% false positives), Turnitin’s 20% threshold, or OPI’s position on Poland’s JSA — every one has its original source cited in the table above, which you should link alongside this round-up.

What this means for you as a writer

The data forms a consistent picture: AI detectors are a probability signal, not proof. They fail in both directions — they can flag innocent, human text and wave through AI-generated text. The conclusion for you is simple and honest: write and edit for the reader, not for the detector. Break up a monotonous sentence rhythm, cut the clichés, add a specific instead of a generality, give the text your own voice. This isn’t about “outsmarting” any system — that’s a dead end and not the game we play — it’s about the text simply being better.

If you want a tool that helps rewrite AI text into natural, human-sounding prose, that’s exactly what humai is for — with no promises that it will slip past a detector, because no one can give those honestly. And if you’d like to see roughly how AI-patterned your own draft looks, our free AI style detector gives you an estimated score in the browser — again, an estimate, not a verdict.