Full Report
Meanwhile, others tried to social-engineer the chatbot itself Nation-state goons and cybercrime rings are experimenting with Gemini to develop a "Thinking Robot" malware module that can rewrite its own code to avoid detection, and build an AI agent that tracks enemies' behavior, according to Google Threat Intelligence Group.…
Analysis Summary
# Threat Actor: APT28 (Fancy Bear, Forest Blizzard, FrozenLake)
## Attribution & Identity
Attributed by the US and UK as part of Russia's General Staff Main Intelligence Directorate (GRU) military unit.
## Activity Summary
Observed using a new data-mining malware named **PromptSteal** against Ukraine in June. This malware leverages Large Language Models (LLMs) during live operations to dynamically generate commands for execution.
## Tactics, Techniques & Procedures
- **LLM Querying for Command Generation:** Malware (PromptSteal) queries LLMs via the Hugging Face API to generate operational commands rather than using hard-coded commands.
- **Data Exfiltration/Mining:** Using AI-powered malware to gather system information.
- **Evolving TTPs:** Spotted adding obfuscation and changing command-and-control methods in newer samples.
- **Social Engineering of AI:** While not directly tied to APT28's malware operation described here, the general threat environment indicates attempts to socially engineer chatbots to bypass safety filters (though the specific successful social engineering attempt mentioned was attributed to a China-linked actor).
## Targeting
- **Sectors:** Not explicitly detailed beyond the observed victim location.
- **Geography:** Ukraine.
- **Victims:** Undisclosed victims targeted during the campaign involving PromptSteal.
## Tools & Infrastructure
- **Malware Families Used:** PromptSteal (a new data-mining malware prefixing with "Prompt").
- **Infrastructure:** Queries the API for Hugging Face to generate commands.
## Implications
APT28 represents a sophisticated threat actor showing early adoption of LLMs *in live malicious operations* (first use of malware querying an LLM in a live campaign). This signals an evolution towards highly adaptive and dynamically controlled malware, potentially increasing the complexity of detection and response.
## Mitigations
- Monitoring for outbound API calls or unusual activity directed towards platforms like Hugging Face from internal endpoints.
- Increased scrutiny of malware exhibiting dynamic command generation based on external model interaction.
---
# Threat Actor: Financially Motivated Actors (Unattributed)
## Attribution & Identity
Unattributed group, though filename conventions suggest behaviors commonly associated with financially motivated actors.
## Activity Summary
Associated with an experimental malware dropper tracked as **PromptFlux**, which utilizes an LLM (Gemini) during execution. This malware aims to rewrite its own source code to evade detection and establish persistence.
## Tactics, Techniques & Procedures
- **Self-Modifying Code/Evasion:** The "Thinking Robot" module interacts with Gemini's API to request VBScript obfuscation/evasion techniques to rewrite its source code, avoiding static detection.
- **Automated Persistence:** Rewrites and saves the new version of the code to establish persistence.
- **AI Social Engineering:** One variation includes a "Thinking" function attempting to trick Gemini into hourly code rewrites by instructing the model to act as an "expert VBScript obfuscator."
- **Malware Language:** Written in VBScript.
## Targeting
- **Sectors:** General; indicators suggest financially motivated targets rather than nation-state espionage.
- **Geography:** Unknown.
- **Victims:** Currently experimental; the current form lacks the capability to compromise networks or devices.
## Tools & Infrastructure
- **Malware Families Used:** PromptFlux (VBScript dropper).
- **Infrastructure:** Interacts with Gemini's API.
## Implications
The development of **PromptFlux** demonstrates an attempt to integrate LLMs directly into the malware execution lifecycle for real-time evasion ("just in time AI in malware"). While currently experimental, successful deployment would significantly challenge signature-based defenses.
## Mitigations
- Implement robust endpoint detection and response (EDR) capable of detecting heuristic/behavioral anomalies, specifically VBScript attempting to rewrite its own source code or make external API calls for code generation.
- Deploy defenses against LLM prompt injection/social engineering attempts used to manipulate AI services hosting the malware's logic.
---
# Threat Actor: Nation-State Goons / State-Sponsored Actors (China-Linked)
## Attribution & Identity
A user linked to China, mentioned in the context of attempting to exploit Gemini's safety features.
## Activity Summary
Observed attempting to solicit sensitive hacking information from Gemini by socially engineering the chatbot. They tried to overcome safety refusals by claiming they were participating in a Capture-The-Flag (CTF) security competition, successfully obtaining information that could be misused to exploit compromised systems (specifically asking about identifying bugs).
## Tactics, Techniques & Procedures
- **Social Engineering of AI:** Actively attempting to bypass safety guardrails using pretexting (claiming participation in a CTF) to elicit potentially malicious information or code.
- **Reconnaissance/Vulnerability Research:** Seeking information about exploiting systems.
## Targeting
- **Sectors:** General systems/vulnerabilities.
- **Geography:** Associated with China.
- **Victims:** Not specific victims mentioned, but the targeting is focused on acquiring exploit-relevant information.
## Tools & Infrastructure
- **Tools:** Used Gemini chatbot proactively for reconnaissance aids.
## Implications
This highlights the risk of state-sponsored actors using generative AI for vulnerability research and target preparation by circumventing built-in alignment controls.
## Mitigations
- Implement strict access controls and monitoring on AI services for suspicious use patterns, particularly when prompts mimic security testing or vulnerability discovery.
- Strengthen internal security auditing based on intelligence gathered from socially engineered AI responses.
---
# Threat Actor: APT42 (Cyber Arm of Iran's IRGC)
## Attribution & Identity
The cyber-arm of Iran’s Islamic Revolutionary Guard Corps (IRGC).
## Activity Summary
Observed experimenting with Gemini to build a **"data processing agent."** This agent converts natural language requests into SQL queries to analyze PII, correlating it with asset ownership, location, demographics, and behavior data. They also previously used AI for translation.
## Tactics, Techniques & Procedures
- **LLM for Data Analysis:** Using AI to convert natural language into complex SQL queries for PII aggregation and analysis.
- **Information Synthesis:** Generating insights about individuals' sensitive attributes (assets, travel, behavior).
- **Account Disablement:** Google Threat Intelligence Group (GTIG) disabled the accounts connected to this activity.
## Targeting
- **Sectors:** Likely targets involving collection and analysis of Personally Identifiable Information (PII).
- **Geography:** Not specified, but linked to Iran's intelligence objectives.
- **Victims:** Individuals whose PII was targeted for analysis.
## Tools & Infrastructure
- **Tools:** Used the Gemini API to power their data processing agent.
## Implications
APT42 is leveraging AI for sophisticated, targeted intelligence gathering by operationalizing LLMs to turn raw data into actionable profiles using custom SQL generation.
## Mitigations
- Secure databases against novel SQL injection methods derived from LLM outputs.
- Enhance identity and access management protocols around sensitive data repositories targeted by intelligence agencies.