Full Report
TeamPCP, the threat actor behind the recent compromises of Trivy and KICS, has now compromised a popular Python package named litellm, pushing two malicious versions containing a credential harvester, a Kubernetes lateral movement toolkit, and a persistent backdoor. Multiple security vendors, including Endor Labs and JFrog, revealed that litellm versions 1.82.7 and 1.82.8 were published on
Analysis Summary
# Incident Report: TeamPCP Supply Chain Compromise of LiteLLM
## Executive Summary
TeamPCP compromised the popular Python package `litellm` by injecting malicious code into versions 1.82.7 and 1.82.8, likely leveraging credentials stolen from a previous compromise of the Trivy CI/CD workflow. The malicious payloads included a credential harvester, a Kubernetes lateral movement toolkit, and a persistent systemd backdoor. The attack represents a significant escalation in a multi-ecosystem campaign targeting developer infrastructure and security tools.
## Incident Details
- **Discovery Date:** March 24, 2026
- **Incident Date:** March 24, 2026
- **Affected Organization:** BerriAI (Maintainers of LiteLLM) and downstream users
- **Sector:** Software Development / Open Source AI Infrastructure
- **Geography:** Global
## Timeline of Events
### Initial Access
- **Date/Time:** March 24, 2026
- **Vector:** Targeted CI/CD Pipeline Compromise
- **Details:** Attackers likely used secrets or access tokens gained from the earlier compromise of Trivy/KICS security scanners to pivot into the LiteLLM build environment.
### Lateral Movement
- The malware used Kubernetes service account tokens to enumerate cluster nodes. It then deployed privileged pods to every node to gain host-level access via `chroot`.
### Data Exfiltration/Impact
- Credentials (SSH keys, cloud secrets, K8s secrets, crypto wallets, `.env` files) were collected.
- Data was packaged into an encrypted archive named `tpcp.tar.gz` and exfiltrated to a C2 server.
### Detection & Response
- **Discovery:** Detected by security vendors Endor Labs and JFrog during proactive package monitoring.
- **Response Actions:** The malicious versions (1.82.7 and 1.82.8) were reported and removed from the Python Package Index (PyPI).
## Attack Methodology
- **Initial Access:** Supply chain injection via compromised GitHub Actions/CI/CD runners.
- **Persistence:** Dropping a systemd service (`sysmon.service`) that executes a Python script at `~/.config/sysmon/sysmon.py`.
- **Privilege Escalation:** Deployment of privileged Kubernetes pods to breakout of containers and access the host file system.
- **Defense Evasion:** Use of `.pth` files to trigger execution on Python startup; use of a "kill switch" (checking for YouTube URLs) to abort if sandbox-detected.
- **Credential Access:** Sweeping for SSH keys, cloud provider secrets, and Kubernetes tokens.
- **Discovery:** Enumeration of K8s cluster nodes and environment variables.
- **Lateral Movement:** Automated deployment of pods across all detected cluster nodes.
- **Collection:** Gathering sensitive files into a central archive (`tpcp.tar.gz`).
- **Exfiltration:** HTTPS POST requests to a domain mimicking the legitimate project (`models.litellm[.]cloud`).
- **Impact:** Complete environment takeover and broad credential theft.
## Impact Assessment
- **Financial:** Potential for cryptocurrency theft and unauthorized cloud resource usage.
- **Data Breach:** High-volume theft of sensitive developer and infrastructure credentials.
- **Operational:** Disruption of CI/CD pipelines and requirement for full rotational resets of all secrets.
- **Reputational:** Significant impact on the trust of LiteLLM and the broader open-source ecosystem.
## Indicators of Compromise
- **Network indicators:**
- models.litellm[.]cloud (Exfiltration)
- checkmarx[.]zone (C2 Polling)
- **File indicators:**
- `tpcp.tar.gz` (Encrypted archive)
- `litellm_init.pth` (Malicious launcher)
- `sysmon.service` / `sysmon.py` (Persistence)
- **Behavioral indicators:**
- Python processes spawning detached child processes via `subprocess.Popen`.
- Unusual creation of privileged pods across all nodes in a K8s cluster.
## Response Actions
- **Containment:** Removal of malicious versions from PyPI.
- **Eradication:** Automation of "kill switches" where applicable; security advisories released to the community.
- **Recovery:** Users advised to downgrade or move to a clean version and rotate all credentials stored in systems where the malicious versions were installed.
## Lessons Learned
- **Cross-Tool Contamination:** Security tools (Trivy/KICS) are high-leverage targets; a compromise there can lead to a domino effect across the entire supply chain.
- **Aggressive Vectors:** The use of `.pth` files shows that malware can execute even if the specific compromised library is not explicitly imported by the user.
## Recommendations
- **Secret Rotation:** Immediately rotate all cloud, SSH, and K8s credentials if the affected versions were used.
- **Pinning Dependencies:** Use hash-based pinning for all dependencies in CI/CD environments.
- **Network Segmentation:** Implement strict egress filtering for CI/CD runners to prevent exfiltration to unknown domains.
- **Audit .pth files:** Monitor for unexpected `.pth` files in Python `site-packages` directories.