Full Report
A real-world benchmark tests whether powerful AI models can keep an investigation trustworthy when new evidence invalidates their conclusions.
Analysis Summary
# Research: Sol Searching | Can Frontier Models Tackle Autonomous Long-Horizon Malware Analysis?
## Metadata
- **Authors:** Juan Andrés Guerrero-Saade, Gabriel Bernadett-Shapiro
- **Institution:** SentinelOne (SentinelLABS)
- **Publication:** SentinelOne Labs Blog
- **Date:** July 22, 2026
## Abstract
This research evaluates the capabilities of "Frontier-class" AI models in performing complex, long-horizon malware investigations. By recreating a real-world investigation of **fast16**—a sophisticated 2005 sabotage implant—SentinelLABS established a benchmark to test if AI can maintain a trustworthy investigation when new evidence contradicts previous conclusions. The study finds that while most models fail at project-scale recovery, OpenAI’s GPT-5.6 Sol demonstrated the ability to autonomously navigate an eight-stage investigation, marking a milestone in agentic cybersecurity capabilities.
## Research Objective
The research addresses whether Large Language Models (LLMs) can move beyond simple vulnerability discovery and "bounded tasks" to perform **autonomous long-horizon reverse engineering**. Specifically:
1. Can a model handle "analytical pivots" where new data invalidates earlier hypotheses?
2. Can a model manage a multi-layered investigation (user-land, kernel-land, and encrypted frameworks) without losing the logical thread?
## Methodology
### Approach
The researchers developed a multi-stage benchmark based on a previous investigation into **fast16**. Success was measured not just by technical accuracy in a single task, but by "project-scale recovery"—the ability to withdraw contradicted conclusions and update all dependent reporting when new evidence surfaced.
### Dataset/Environment
The test subject was **fast16**, a 2005 Windows toolkit designed to sabotage nuclear-weapons modeling software. It features:
- A surface-level Windows service (`svcmgmt.exe`).
- An encrypted Lua-driven operations framework.
- A kernel driver containing 101 patching rules for software sabotage.
### Tools & Technologies
The benchmark compared several frontier models, specifically:
- OpenAI: **GPT-5.6 Sol** (Public), **GPT-5.*-cyber-preview** (Trusted Access Program), **o1-preview**.
- Anthropic: **Opus 4.x family**, **Mythos Preview** (Glasswing initiative).
- Zhipu AI: **GLM-5.2**.
- Google DeepMind: **Big Sleep** (referenced).
## Key Findings
### Primary Results
1. **Sol Consistency:** GPT-5.6 Sol was the only publicly available model to successfully complete the full eight-stage investigation.
2. **Failure of Local Analysis:** While models like GLM-5.2 and Opus 4.x performed well on isolated technical tasks, they failed to carry findings through the "gradient" of the full investigation.
3. **The "Check-Back" Requirement:** Successful runs were characterized by the model's ability to repair technical artifacts and update reporting when its earlier assumptions were proved wrong.
### Supporting Evidence
- **fast16 Complexity:** The implant punishes shallow analysis; failure to decrypt the Lua framework results in a complete misunderstanding of the tool's purpose (sabotage vs. simple malware). GPT-5.6 successfully pivoted when the Lua framework was discovered.
### Novel Contributions
- **Long-Horizon Benchmarking:** Moving the goalposts from "find a bug" to "investigate a campaign."
- **Focus on Trustworthiness:** Defining success as the model’s ability to maintain a coherent, self-correcting narrative throughout an evolving investigation.
## Technical Details
The investigation required the AI to transition between layers of abstraction.
- **Layer 1:** Reverse engineering a Windows service.
- **Layer 2:** Identifying and decrypting a non-standard Lua-driven engine.
- **Layer 3:** Analyzing kernel-level driver logic to understand a patching engine.
- **Layer 4:** Extrapolating the impact of 101 unique patching rules on high-precision solvers.
## Practical Implications
### For Security Practitioners
- **Supervised Agency:** AI is moving from a "chatbot" to an "investigative agent." Analysts should use these models to perform heavy lifting while the human defines the objective.
- **Expertise Augmentation:** These tools can bridge the gap in thin talent pools for reverse engineering, provided a senior analyst oversees the output.
### For Defenders
- AI can significantly accelerate the "time to understanding" for complex, multi-stage implants that would normally take weeks of manual work by senior REs.
### For Researchers
- There is a need for "defensive benchmarks" that focus on campaign analysis rather than just exploit generation.
## Limitations
- **Semantic Errors:** Even the most successful models (Sol) made semantic mistakes and occasionally accepted weak quality controls.
- **Premature Readiness:** Models often claimed they had finished the investigation before they had fully explored the evidence.
- **Access Barriers:** The highest-performing models are increasingly locked behind "Trusted Access" programs or government clearances.
## Comparison to Prior Work
Previous AI benchmarks focused heavily on **vulnerability discovery** (e.g., Google's Big Sleep or OpenAI's Aardvark). This research shifts the focus to **malware forensics and sabotage analysis**, where the goal is understanding *intent* and *impact* rather than just identifying a crash.
## Real-world Applications
- **Malware Triage:** Automating the deep-dive analysis of complex APT kits.
- **Knowledge Synthesis:** Compiling technical findings into executive reporting that remains updated as the investigation progresses.
## Future Work
- **Closing the Reasoning Gap:** Improving model awareness of "blind spots" during an investigation.
- **Safety and Guardrails:** Balancing the need for "unfiltered" cybersecurity tools with the risk of enabling malicious code generation.
## References
- SentinelLABS research on `fast16` (s1.ai/fast16)
- OpenAI Daybreak Initiative
- Anthropic Glasswing / Mythos Preview
- Black Hat Asia 2026: "Cyber Paleontology in the Age of AI"