Full Report
Python interface for LLMs infected with malware via polluted CI/CD pipeline Two versions of LiteLLM, an open source interface for accessing multiple large language models, have been removed from the Python Package Index (PyPI) following a supply chain attack that injected them with malicious credential-stealing code.…
Analysis Summary
# Incident Report: Supply Chain Compromise of LiteLLM via Polluted Trivy CI/CD
## Executive Summary
LiteLLM, a popular open-source Python interface for Large Language Models, fell victim to a supply chain attack after its CI/CD pipeline was compromised through a malicious update to the Trivy vulnerability scanner. Attackers exfiltrated PyPI publishing tokens to inject credential-stealing code into LiteLLM versions v1.82.7 and v1.82.8. The incident highlights the risks of relying on unpinned version tags in automated GitHub Actions workflows.
## Incident Details
- **Discovery Date:** March 24, 2026
- **Incident Date:** Late February 2026 (Lifecycle start) to March 24, 2026
- **Affected Organization:** Berri AI (maintainers of LiteLLM) and Aqua Security (Trivy maintainers)
- **Sector:** Software Development / Artificial Intelligence
- **Geography:** Global (Open Source Ecosystem)
## Timeline of Events
### Initial Access
- **Date/Time:** Late February 2026
- **Vector:** Misconfiguration in Trivy’s GitHub Actions environment.
- **Details:** Attackers (identified as TeamPCP) exploited a misconfiguration in the Trivy project to steal a privileged access token.
### Lateral Movement
- **March 19 - 22, 2026:** Attackers used the stolen Trivy credentials to publish malicious versions (v0.69.4 - v0.69.6) and, more critically, modified existing GitHub Action version tags (`trivy-action`).
- **March 23, 2026:** LiteLLM’s CI/CD pipeline executed the compromised `trivy-action`. The malicious script exfiltrated LiteLLM’s `PYPI_PUBLISH` token, which was stored as an environment variable, and sent it to an attacker-controlled endpoint.
### Data Exfiltration/Impact
- **Credential Theft:** Attackers used the stolen PyPI token to bypass 2FA and upload two malicious versions of LiteLLM (v1.82.7 and v1.82.8) containing `litellm_init.pth`.
- **Malware Functionality:** The injected code was designed to steal credentials and environment variables from any system where the infected LiteLLM versions were installed.
### Detection & Response
- **March 24, 2026:** The malicious versions were identified and reported.
- **Counter-Interference:** Attackers launched a spam campaign (AI-generated "Thanks, that helped!" comments) on the GitHub vulnerability report to obscure detection and discussion.
- **Response:** Berri AI revoked all PyPI tokens and moved to take down the infected packages.
## Attack Methodology
- **Initial Access:** Exploitation of misconfigured GitHub Action secrets in the upstream Trivy repository.
- **Persistence:** Injection of malicious code into existing Docker images and GitHub Action version tags to ensure continuous execution in downstream pipelines.
- **Privilege Escalation:** Pivoted from a compromised vulnerability scanner token to a PyPI publishing token.
- **Defense Evasion:** Use of automated bot accounts to spam issue trackers and drown out security warnings; modification of existing version tags rather than creating new, suspicious ones.
- **Credential Access:** Exfiltration of `.env` variables and PyPI publishing tokens from CI/CD memory/logs.
- **Exfiltration:** Unauthorized upload of malicious packages to the Python Package Index (PyPI).
- **Impact:** Supply chain pollution leading to credential theft on end-user machines.
## Impact Assessment
- **Financial:** Unknown; potential costs related to incident response and remediation for downstream users.
- **Data Breach:** Exposure of environment variables and LLM API keys for an unknown number of LiteLLM users.
- **Operational:** Disruption of LiteLLM development and removal of package versions from PyPI.
- **Reputational:** Significant impact on the perceived security of the LiteLLM and Trivy projects.
## Indicators of Compromise
- **File indicators:** `litellm_init.pth` (contained within LiteLLM v1.82.7 and v1.82.8).
- **Version indicators:** Trivy v0.69.4, v0.69.5, v0.69.6; LiteLLM v1.82.7, v1.82.8.
- **Behavioral indicators:** Abnormal outbound traffic from CI/CD runners to unknown endpoints; rapid influx of AI-generated spam on GitHub issues.
## Response Actions
- **Containment:** Removal of infected versions (v1.82.7 and v1.82.8) from PyPI.
- **Eradication:** Deletion of all existing PyPI publishing tokens and rotation of GitHub secrets.
- **Recovery:** Implementation of "Trusted Publishing" via OIDC/JWT tokens to eliminate the need for long-lived PyPI tokens.
## Lessons Learned
- **Tag Mutability:** Relying on mutable version tags (e.g., `@v1`) instead of immutable commit SHAs allows attackers to hijack established workflows without changing the workflow file.
- **Token Scope:** Environment variables containing sensitive tokens should be restricted to specific environments and not accessible by third-party scanning tools unless strictly necessary.
- **Bot Interference:** Attackers are using AI to generate noise on public bug trackers to slow down human triage and response.
## Recommendations
- **Pin Actions to SHAs:** Always pin GitHub Actions to a full length commit SHA (e.g., `uses: aqua/trivy-action@ [commit_hash]`) rather than a version tag.
- **Use Trusted Publishers:** Move from manual PyPI tokens to CID/CD Trusted Publishing (OIDC) to minimize the risk of token theft.
- **Secret Masking:** Ensure CI/CD logs and third-party tools are audited for potential secret leakage.
- **Credential Rotation:** Users who installed LiteLLM v1.82.7 or v1.82.8 should immediately rotate all environment-accessible keys (OpenAI, Anthropic, etc.).