Full Report
Two of the seven AI detectors I tested correctly identified AI-generated content 100% of the time. This is up from zero during my early rounds, but down from my last round of tests.
Analysis Summary
# Main Topic
Assessment of the efficacy and consistency of commercially available AI content detection tools against AI-generated text.
## Key Points
- Two out of seven tested AI detectors correctly identified AI-generated content 100% of the time in the latest round of testing.
- This performance metric represents an increase from zero successful identifications in early testing rounds but a decrease compared to the previous test round.
- Results across different AI checker tools are highly inconsistent, ranging from perfect detection to labeling human-written text with high confidence as AI-generated (e.g., one tool scored human text at 22% AI).
- The article frames the use of uncredited AI content as meeting the dictionary definition of plagiarism.
- Detection tools tested included GPT-2 Output Detector, Writer.com AI Content Detector, BrandWell AI Content Detection, GPTZero, ZeroGPT, Writefull's GPT Detector, Originality.ai, QuillBot, and Grammarly.
## Threat Actors
- Not explicitly named as malicious threat actors, but the context relates to users generating content via large language models (LLMs) such as **ChatGPT** and attempting to pass it off as original work (plagiarism concern).
## TTPs
- **Content Generation:** Utilizing Generative Pre-trained Transformer (GPT) models (e.g., ChatGPT) to produce human-like text.
- **Evasion:** Submitting AI-generated text without proper attribution, attempting to pass it off as human work.
## Affected Systems
- AI Content Detection Software (including GPT-2 Output Detector, Writer.com, GPTZero, ZeroGPT, Originality.ai, Grammarly, etc.).
- Educational and editorial systems where authorship verification is critical.
## Mitigations
- The primary mitigation discussed is the use of AI detection tools, though their unreliability necessitates caution.
- **Human Judgment:** The author notes that some tools, even when scoring human text poorly, are ultimately relying on subjective judgment (e.g., scores above 80% confidence are considered "correct" generously).
- **Tracking:** Monitoring the performance drift of specific detectors over time (e.g., Writer.com failed in earlier tests but appeared functional in the current test).
## Conclusion
The landscape of AI content detection is volatile and inconsistent. While a small subset of tools demonstrated perfect accuracy in the latest evaluation, the overall lack of dependable uniformity across checkers poses a significant challenge for individuals (teachers, editors) attempting to verify content authenticity. Organizations relying on these tools must recognize that no single vendor currently provides reliable, consistent detection across the board.