Full Report
Two malicious LiteLLM releases sat on PyPI for about 40 minutes in March carrying credential-stealing code capable of harvesting cloud keys, SSH keys, Kubernetes tokens, database passwords, and other secrets from systems that installed them. Threat intelligence firm CloudSEK now says a dataset it obtained, built from roughly 434,000 files the attackers captured, maps potential exposure to more
Analysis Summary
# Incident Report: Malicious LiteLLM Supply Chain Compromise (TeamPCP)
## Executive Summary
In March 2024, threat actor group TeamPCP (tracked as UNC6780) successfully compromised the LiteLLM project to distribute two malicious package versions (1.82.7 and 1.82.8) via PyPI. The poisoned versions contained credential-stealing code that harvested cloud keys, SSH keys, and CI/CD secrets, potentially impacting over 2,100 organizations including major tech and finance firms. Although the packages were live for only 40 minutes, the exfiltrated long-lived secrets provide attackers with ongoing access to sensitive cloud and development environments.
## Incident Details
- **Discovery Date:** March 24, 2024
- **Incident Date:** March 24, 2024 (Related activity beginning March 19, 2024)
- **Affected Organization:** Users of LiteLLM (AI Gateway); 2,100+ organizations identified in exfiltrated data.
- **Sector:** Technology, Finance, Logistics, Automotive (Broad/Cross-sector)
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** March 19, 2024
- **Vector:** Incomplete credential rotation following a prior breach.
- **Details:** Attackers retained access to the Trivy-action repository and force-pushed malicious commits, eventually pivoting to the LiteLLM project.
### Lateral Movement
- **Details:** Attackers leveraged compromised credentials from the Trivy/Aqua Security ecosystem to gain unauthorized publishing rights for the LiteLLM package on PyPI.
### Data Exfiltration/Impact
- **Details:** Malicious code harvested environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY), SSH keys, Kubernetes tokens, and database passwords from CI/CD runners and developer workstations.
### Detection & Response
- **Discovery:** PyPI automated systems/security researchers identified the malicious behavior within 40 minutes of the 1.82.7/1.82.8 release.
- **Response actions:** PyPI quarantined the affected versions; LiteLLM released version 1.83.0; CloudSEK released a public lookup tool for organizations to check exposure.
## Attack Methodology
- **Initial Access:** Exploitation of existing/unrotated repository credentials.
- **Persistence:** Implementation of `litellm_init.pth` which executes code at Python interpreter startup, regardless of whether the package is imported.
- **Privilege Escalation:** Harvesting of high-privilege CI/CD tokens and Kubernetes secrets.
- **Defense Evasion:** Short exposure window (40 minutes) and use of legitimate-looking package versions.
- **Credential Access:** Scraping environment variables and local secret files (SSH/Cloud keys).
- **Collection:** Gathering secrets from host environments and CI pipelines.
- **Exfiltration:** Data encrypted and sent to attacker-controlled domain: `models.litellm[.]cloud`.
- **Impact:** Ongoing potential for secondary breaches via stolen long-lived credentials.
## Impact Assessment
- **Financial:** High potential cost due to mandatory rotation of thousands of secrets and potential misuse of cloud resources.
- **Data Breach:** Exposure of 434,000+ files and logs containing secrets from 2,100+ organizations (including NVIDIA, Cisco, and X Corp).
- **Operational:** Disruption of CI/CD pipelines as teams must pause to audit and rotate all environmental secrets.
- **Reputational:** Significant trust impact on the AI/Open-source supply chain ecosystem.
## Indicators of Compromise
- **Network indicators:** `models.litellm[.]cloud` (Defanged)
- **File indicators:** `litellm_init.pth` (contained in malicious package versions)
- **Behavioral indicators:** Unexpected outbound traffic from CI/CD runners to unauthorized domains; unusual Python interpreter startup scripts.
## Response Actions
- **Containment:** PyPI quarantined versions 1.82.7 and 1.82.8.
- **Eradication:** Versions removed from PyPI history; replaced by clean version 1.83.0.
- **Recovery:** Organizations advised to rotate all credentials (API keys, SSH, K8s tokens) exposed during the March 24 window.
## Lessons Learned
- **Credential Hygiene:** An "incomplete credential rotation" after a previous breach (Trivy) allowed the attackers to maintain a foothold.
- **Transitive Risk:** Users who did not explicitly install LiteLLM were infected because it was pulled in as an unpinned transitive dependency by other AI orchestration tools.
- **Python Execution:** The use of `.pth` files allows attackers to execute code even if the library is never explicitly called by the application.
## Recommendations
1. **Immediate Rotation:** Rotate all secrets (OpenAI, Anthropic, AWS, GCP, Azure, SSH) that were present in environments where Python packages were updated on March 24.
2. **Move to OIDC:** Transition from long-lived static tokens to short-lived identity-based authentication (OIDC) for CI/CD pipelines.
3. **Dependency Pinning:** Use `requirements.txt` or `Poetry` with strict hashes to prevent automatic updates to malicious transitive dependencies.
4. **Environment Isolation:** Use isolated, ephemeral runners for CI/CD jobs and restrict their outbound network access to only required domains.