There is a wide range of software available which has been designed to classify whether text is machine or human generated, with providers claiming high levels of accuracy in being able to identify whether text is written by a human or by a GenAI tool (GPTZero, n.d.; Turnitin, 2023). While some of these tools are free and others require either registration or payment, research by Walters (2023) has identified that the accuracy of paid-for tools is only slightly higher than that of free versions. However, claims of accuracy are contradicted by studies which demonstrate the varied levels of the detectors' ability to distinguish accurately between AI and human-generated content. (Chaka, 2023a; Gao et al., 2022; Krishna et al., 2023; Orenstrakh et al., 2023; Perkins, Roe, et al., 2023; Walters, 2023; Weber-Wulff et al., 2023).

Overall, our results demonstrate the challenges of current AI text detection tools being able to accurately determine whether a given piece of text was created by a human or a GenAI tool. This ability is further reduced when adversarial techniques are used to obscure the nature of a sample. If the goal of any given HEI was to use AI text detectors solely to determine whether a student has breached academic integrity guidelines, we would caution that the accuracy levels we have identified, coupled with the risks inherent in false accusations, means that we cannot recommend them for this purpose. This is not because of the demonstrated abilities of any one tool tested, as we recognise that developers are continuously updating these tools, and the detection of AI-generated content when subject to adversarial techniques is likely to improve. However, simultaneously, advances are being made in the development of more capable FMs that can produce more human-like content, resulting in a constant arms race between FMs and AI text detectors, with student inclusivity paying the price.

Do AI detectors work?

• In short, no, not in our experience. Our research into detectors didn't show them to be reliable enough given that educators could be making judgments about students with potentially lasting consequences.While other developers have released detection tools, we cannot comment on their utility.
• Additionally, ChatGPT has no “knowledge” of what content could be AI-generated. It will sometimes make up responses to questions like “did you write this [essay]?” or “could this have been written by AI?” These responses are random and have no basis in fact.
• To elaborate on our research into the shortcomings of detectors, one of our key findings was that these tools sometimes suggest that human-written content was generated by AI.
• When we at OpenAI tried to train an AI-generated content detector, we found that it labeled human-written text like Shakespeare and the Declaration of Independence as AI-generated.
• There were also indications that it could disproportionately impact students who had learned or were learning English as a second language and students whose writing was particularly formulaic or concise.
• Even if these tools could accurately identify AI-generated content (which they cannot yet), students can make small edits to evade detection.

In our baseline testing protocol of both non-manipulated AI-generated samples tested alongside the human-written control samples, we see an initially lower than expected average accuracy rating for the detection of AI-generated content, coupled with a substantial rate of false accusations in the human-written control samples. When the AI-generated samples were subjected to manipulation, significant vulnerabilities in accurately detecting text were observed. If the goal of implementing AI detection tools as part of an overall academic integrity strategy is to support academic staff in identifying where machine-generated content has been used and has not been declared, these inaccuracies may lead to a false sense of security and a broader reduction in assessment security. As assessment security is a key component in ensuring inclusive, equitable, and fair opportunities for learners, this is problematic. The varying degrees of reduction in accuracy following the application of adversarial techniques also point to the broader issue of inconsistency and unpredictability in the current AI detection capabilities. The effectiveness of these techniques varies dramatically across detectors, suggesting that the internal algorithms and heuristics of these detectors are tuned differently and react distinctively to similar inputs. Therefore, the results even within an institution may differ depending on the tool being employed and how it is being used.