Full Report
AI agents (utilizing LLMs and RAG) are being used within SOCs and SIEMS to both help identify attacks and assist analysts with working more efficiently; however, I’ve done a little bit of research one sunny British afternoon and found that these agents can be abused by attackers and made to go rogue. They can be made to modify the details of an attack, hide attacks altogether, or create fictitious events to cause a distraction while the real target is attacked instead. Furthermore, if the LLM has access to additional tools or excessive privileges, then an attacker can venture out into other attacks.
Analysis Summary
# Rogue AI Agents In Your SOCs and SIEMs – Indirect Prompt Injection via Log Files
=====================================
## Key Points
- The use of Large Language Models (LLM) and Rapidly Adapting Groups (RAG) chatbots with agents and tools within Security Operations Centers (SOCs) and Security Information and Event Management Systems (SIEMs) can introduce a potential vulnerability for indirect prompt injection via log files.
- An attacker could exploit this by elevating privileges, abusing excessive privileges, or using functions/tools to laterally move within the system.
- Indirect prompt injection is a real-world concern that requires organizations to consider all possible sources and sinks in their systems, especially when AI agents are involved.
## Threat Actors
- LevelBlue Completes Acquisition of Trustwave to Form the World's Largest Pure-Play MSSP
- LevelBlue
- Trustwave
## TTPs
- Indirect Prompt Injection via Log Files
- Exploiting vulnerabilities in LLM and RAG chatbots within SOCs/SIEMs
- Elevating privileges or abusing excessive privileges
- Using functions/tools for lateral movement
## Affected Systems
- Security Operations Centers (SOCs)
- Security Information and Event Management Systems (SIEMs)
## Mitigations
- Implement robust logging and monitoring mechanisms to detect potential security breaches.
- Conduct regular vulnerability assessments and penetration testing to identify potential weaknesses in LLM and RAG chatbots within SOCs/SIEMs.
- Establish clear boundaries and guardrails for AI agents and tools within these systems to prevent indirect prompt injection.
## Conclusion
The use of LLM and RAG chatbots with agents and tools within SOCs/SIEMs introduces a potential vulnerability for indirect prompt injection via log files. Organizations must be aware of this risk and implement necessary mitigations to protect themselves against such threats.