Full Report
The TeamPCP hacking group continues its supply-chain rampage, now compromising the massively popular "LiteLLM" Python package on PyPI and claiming to have stolen data from hundreds of thousands of devices during the attack. [...]
Analysis Summary
# Incident Report: LiteLLM PyPI Supply Chain Compromise
## Executive Summary
The popular open-source Python library "LiteLLM" was compromised in a supply chain attack orchestrated by the threat group TeamPCP. Malicious versions (1.82.7 and 1.82.8) were published to PyPI, containing a multi-stage infostealer that harvests cloud credentials, Kubernetes secrets, and SSH keys. The attack is part of a broader campaign targeting the software supply chain, potentially impacting hundreds of thousands of devices.
## Incident Details
- **Discovery Date:** March 24, 2026
- **Incident Date:** March 24, 2026
- **Affected Organization:** Users of the LiteLLM Python package
- **Sector:** Information Technology / Software Development / AI
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** March 24, 2026
- **Vector:** Supply Chain Compromise (PyPI Package Injection)
- **Details:** Attackers gained unauthorized access to the LiteLLM PyPI repository and published two malicious versions (1.82.7 and 1.82.8).
### Lateral Movement
- **Mechanism:** The malware attempts to move across Kubernetes clusters by deploying privileged pods to every node to gain broader access to the infrastructure.
### Data Exfiltration/Impact
- **Details:** The "TeamPCP Cloud Stealer" harvests SSH keys, AWS/GCP/Azure tokens, K8s secrets, crypto wallets, and `.env` files. Data is archived in `tpcp.tar.gz`, encrypted, and sent to attacker-controlled infrastructure. Reports suggest up to 500,000 exfiltrations occurred.
### Detection & Response
- **Discovery:** Detected by security researchers at Endor Labs.
- **Response:** Malicious versions were removed from PyPI. Version 1.82.6 was reinstated as the latest clean release.
## Attack Methodology
- **Initial Access:** Compromised PyPI developer credentials/token.
- **Persistence:** Implementation of a `.pth` file (`litellm_init.pth`) and a systemd user service disguised as "System Telemetry Service."
- **Privilege Escalation:** Deployment of privileged pods in Kubernetes environments.
- **Defense Evasion:** Payloads were Base64-encoded within legitimate library files (`proxy_server.py`) and disguised as telemetry services.
- **Credential Access:** Scraping of environment variables, cloud provider metadata, and local configuration files (SSH, TLS, K8s).
- **Discovery:** System reconnaissance using standard shell commands (`ip addr`, `printenv`, `uname -a`).
- **Lateral Movement:** Kubernetes pod injection and node-wide deployment.
- **Collection:** Bundling stolen data into an encrypted archive (`tpcp.tar.gz`).
- **Exfiltration:** HTTPS exfiltration to `models.litellm[.]cloud`.
- **Impact:** Massive data theft and potential for cascading breaches across victims' cloud environments.
## Impact Assessment
- **Financial:** High (Cost of rotating thousands of enterprise secrets and potential cloud resource abuse).
- **Data Breach:** Critical (Full exposure of cloud environment secrets, CI/CD tokens, and TLS private keys).
- **Operational:** Moderate (Requires manual intervention to purge persistence and rotate credentials).
- **Reputational:** High (Significant impact on the trust of the LiteLLM open-source project).
## Indicators of Compromise
- **Network Indicators:**
- `checkmarx[.]zone` (C2 Polling)
- `models.litellm[.]cloud` (Exfiltration)
- **File Indicators:**
- `litellm_init.pth` (Python environment persistence)
- `~/.config/sysmon/sysmon.py` (Persistence script)
- `tpcp.tar.gz` (Stolen data archive)
- `/tmp/pglog` and `/tmp/.pg_state` (Temporary artifacts)
- **Behavioral Indicators:**
- Unauthorized pods appearing in the `kube-system` namespace.
- Systemd services named "System Telemetry Service" not authorized by IT.
## Response Actions
- **Containment:** PyPI revoked and removed malicious versions 1.82.7 and 1.82.8.
- **Eradication:** Users must delete the malicious `.pth` files and systemd service files.
- **Recovery:** Mandatory rotation of all cloud tokens, SSH keys, and secrets stored in environment variables on affected machines.
## Lessons Learned
- **Dependency Vulnerability:** Popular packages with millions of downloads are high-value targets; the "trust" associated with popular libraries can be exploited.
- **Cascading Breaches:** TeamPCP utilized a "leapfrog" strategy, using secrets from one breach (Aqua Security/Trivy) to facilitate the next (LiteLLM).
- **The .pth Vector:** Attackers are increasingly using Python's automatic path processing for stealthy persistence.
## Recommendations
- **Version Pinning:** Pin dependencies to specific known-good versions and use hash verification (e.g., `requirements.txt` with hashes).
- **Secret Management:** Move away from local `.env` files toward hardware security modules (HSM) or dedicated secret managers (Vault) that do not store tokens in plain text.
- **Runtime Monitoring:** Implement monitoring for unauthorized systemd changes and suspicious pod creation in Kubernetes namespaces.
- **Credential Rotation:** Establish an automated rotation policy for all cloud and CI/CD service tokens.