Full Report
There’s new benchmark measuring AI’s ability to perform mathematical cryptanalysis. Anthropic’s frontier model actually found new attacks. The benchmark: “CryptanalysisBench: Can LLMs do Cryptanalysis?” The idea is to benchmark the ability of LLMs to discover new mathematical cryptanalytic attacks against a series of historical algorithms. Abstract: Cryptanalysis—the task of finding attacks against cryptographic schemes—its at the intersection of mathematical reasoning and cybersecurity, two areas where LLMs have advanced fastest. Cryptanalysis represents both a clean testbed for frontier reasoning (as practical attacks can be automatically verified) and a domain with unusually high stakes, since the primitives under study underpin our digital security. In this paper we ask whether LLMs can do cryptanalysis, and find that the answer is increasingly yes. We introduce CryptanalysisBench, 191 tasks across six families of cryptographic primitives (block ciphers, hash functions, etc.) drawn primarily from four NIST standardization competitions. Our benchmark consists of three tiers: (i) primitives with known practical breaks; (ii) primitives with no known practical break, evaluated both at full strength and as scaled-down variants; and (iii) a challenge set of production primitives at the frontier of cryptanalysis. Five frontier models (Claude Opus 4.8, Sonnet 5, Mythos 5, GPT-5.5, and the open-weights GLM-5.2) break 65%86% of Tier 1 schemes, 612 Tier-2 schemes at full strength, and 2461 across all scaled-down variants. Beyond deriving known results, models produce novel cryptanalysis, such as a key-recovery attack that exploits a design flaw in the SpoC AEAD and an error in KINDI’s published CCA-security proof, both to the best of our knowledge not previously known...
Analysis Summary
# Research: CryptanalysisBench: Can LLMs do Cryptanalysis?
## Metadata
- **Authors:** Not explicitly listed in text (Anthropic Research Team referenced)
- **Institution:** Anthropic (and associated researchers)
- **Publication:** arXiv (Preprint) / Technical Analysis via Schneier on Security
- **Date:** July 2026 (Note: Based on context of simulated date in source)
## Abstract
This research investigates the capability of Large Language Models (LLMs) to perform mathematical cryptanalysis—the process of identifying vulnerabilities in cryptographic algorithms. The researchers introduce **CryptanalysisBench**, a comprehensive framework consisting of 191 tasks across diverse cryptographic primitives. The study reveals that frontier AI models are no longer merely reciting known exploits but are capable of discovering novel attacks and identifying flaws in published security proofs, signaling a shift in the role of AI in high-stakes cybersecurity reasoning.
## Research Objective
The research aims to determine if LLMs have evolved from simple pattern matching to sophisticated mathematical reasoning capable of original cryptanalytic discovery. Specifically, it seeks to benchmark AI performance against both historically broken algorithms and modern, production-grade primitives.
## Methodology
### Approach
The researchers developed a three-tiered evaluation framework:
1. **Tier 1 (Known Breaks):** Testing models on historical schemes with established practical vulnerabilities.
2. **Tier 2 (Scaled/Full-Strength):** Evaluating models on schemes with no known practical breaks, including "scaled-down" (simplified) versions to measure incremental reasoning.
3. **Tier 3 (Frontier Challenge):** Testing against modern production primitives that represent the current state-of-the-art in security.
### Dataset/Environment
- **191 total tasks.**
- **Six families of primitives** (including block ciphers, hash functions, and AEAD schemes).
- **Sources:** Primarily drawn from four major NIST (National Institute of Standards and Technology) standardization competitions.
### Tools & Technologies
- **Models Tested:** Claude 4.8 Opus, Claude 5 Sonnet, Mythos 5, Mythos Preview, GPT-5.5, and GLM-5.2 (open-weights).
- **Verification:** Automated verification of practical attacks to ensure mathematical rigor.
## Key Findings
### Primary Results
1. **High Success on Known Vulnerabilities:** Frontier models successfully broke between 65% and 86% of Tier 1 schemes.
2. **Original Discovery:** Models successfully identified **novel attacks** not previously documented in academic literature.
3. **Proof Verification:** Models identified an error in the published CCA-security proof for the KINDI algorithm.
4. **Scaling Performance:** Models broke 24–61 scaled-down variants of Tier 2 schemes and successfully attacked 6–12 full-strength Tier 2 schemes.
### Supporting Evidence
- **Specific Exploits:** Discovery of a key-recovery attack via a design flaw in the SpoC AEAD (Authenticated Encryption with Associated Data).
- **Variant Testing:** Success rates increased significantly as the complexity of primitives was "scaled down," providing a metric for how close models are to breaking full-strength modern encryption.
### Novel Contributions
- **CryptanalysisBench:** The first large-scale, automated benchmark specifically for mathematical cryptanalysis.
- **Novel Attack Vectors:** The documentation of new vulnerabilities in the SpoC and KINDI schemes, as well as new vulnerabilities in Hawk and reduced-round AES found by Mythos Preview.
## Technical Details
The research moves beyond "textbook" cryptanalysis. Instead of simply identifying a "Linear Cryptanalysis" attack by name, the models demonstrated the ability to derive the specific mathematical equations required to recover keys or forge signatures. For the SpoC AEAD exploit, the AI identified a flaw in the internal state initialization or permutation logic that allowed for state recovery—a task requiring deep understanding of the algorithm's specific construction rather than general knowledge.
## Practical Implications
### For Security Practitioners
- **Automated Auditing:** LLMs can now be used as "automated red-teamers" for new cryptographic proposals.
- **Rapid Prototyping:** Designers can use AI to stress-test candidate schemes during the design phase rather than waiting for years of peer review.
### For Defenders
- **Shortened Lifespan of Weak Primitives:** Algorithms with subtle flaws are likely to be compromised much faster than in the pre-AI era.
- **Verification Priority:** There is an urgent need to re-verify existing security proofs, as AI has demonstrated the ability to find "bugs" in human-written proofs.
### For Researchers
- **Shift in Human Effort:** Human cryptanalysts may pivot toward designing more complex, AI-resistant structures or guiding AI in exploring vast search spaces for differential paths.
## Limitations
- **Verification Bottleneck:** While attacks can be automatically verified, the *synthesis* of complex attacks still requires significant compute.
- **Scaling Limits:** While successful on multi-round "scaled-down" variants, the jump to full-strength, full-round modern ciphers (like AES-256) remains a significant barrier for current models.
## Comparison to Prior Work
Previous research focused on the AI's ability to write code for *known* attacks. This paper differs by showing that frontier models can perform **original research**—finding flaws that human experts had overlooked during formal NIST standardization processes.
## Real-world Applications
- **NIST Competitions:** Future cryptographic standards can be evaluated via CryptanalysisBench to ensure "day-zero" robustness.
- **Protocol Hardening:** Real-time analysis of bespoke cryptographic implementations in industry.
## Future Work
- **Expanding the Frontier:** Testing AI on post-quantum cryptographic (PQC) primitives.
- **Human-AI Collaboration:** Exploring how "Human+AI" teams compare to standalone AI in breaking hardened targets.
- **Mitigation Research:** Developing schemes specifically designed to be difficult for AI-based mathematical solvers.
## References
- CryptanalysisBench: Can LLMs do Cryptanalysis? (ArXiv: 2607.18538)
- Anthropic: "Discovering Cryptographic Weaknesses"
- Schneier on Security: hxxps://www.schneier.com/blog/archives/2026/07/measuring-llms-ability-to-perform-cryptanalysis.html