Full Report
Automation flaw in CI/CD workflow let a bad pull request unleash worm into npm PostHog says the Shai-Hulud 2.0 npm worm compromise was "the largest and most impactful security incident" it's ever experienced after attackers slipped malicious releases into its JavaScript SDKs and tried to auto-loot developer credentials.…
Analysis Summary
# Incident Report: Shai-Hulud 2.0 npm Worm Compromise
## Executive Summary
Attackers exploited an automation flaw within PostHog's Continuous Integration/Continuous Deployment (CI/CD) workflow, allowing a malicious pull request to inject the Shai-Hulud 2.0 worm into several JavaScript SDKs published on npm. This led to the compromise of developer credentials and secrets across multiple organizations, resulting in a widespread supply chain incident described by PostHog as its most impactful security event. Response actions included revoking tokens and deploying "known-good" releases.
## Incident Details
- **Discovery Date:** Not explicitly stated, but inferred shortly after the malicious packages were published/installed by a significant number of developers.
- **Incident Date:** The attack culminated in the publishing of trojanized packages around Friday, November 28, 2025 (based on the date of the article/postmortem release implied).
- **Affected Organization:** PostHog (Primary focus of the postmortem), Zapier, AsyncAPI, ENS Domains, and Postman.
- **Sector:** Technology / Software Development
- **Geography:** Global, impacting public npm repositories and associated developers/organizations.
## Timeline of Events
### Initial Access
- **Date/Time:** Predating the final package release.
- **Vector:** Malicious Pull Request (PR) targeting the repository configuration/workflow.
- **Details:** An attacker submitted a malicious pull request to a targeted repository (implied to be PostHog's). The compromised CI/CD workflow blindly executed code from the attacker's branch.
### Lateral Movement
- **Date/Time:** Concurrent with initial access and subsequent installations.
- **Vector:** Worm propagation via infected npm dependent packages.
- **Details:** The initial breach allowed exfiltration of a bot's Personal Access Token (PAT) with organizational write permissions. This token was used to commit new malicious code. A subsequent modified lint workflow was deployed to harvest all GitHub secrets, including the npm publishing token.
### Data Exfiltration/Impact
- **Date/Time:** As packages were installed and pre-install scripts executed.
- **Vector:** Compromised npm packages running malicious pre-install scripts.
- **Details:** Infected SDKs (`posthog-node`, `posthog-js`, `posthog-react-native`, etc.) contained scripts that ran `TruffleHog` upon installation. These scripts scanned developers' machines for credentials (npm/GitHub tokens, AWS/Azure/GCP cloud secrets, CI/CD secrets, environment variables), which were then exfiltrated to public GitHub repositories. Over 25,000 developers had secrets compromised within three days.
### Detection & Response
- **Date/Time:** Undisclosed, but followed the infection proliferation.
- **Vector:** PostHog security analysis and public reporting (implied).
- **Details:** PostHog revoked all compromised tokens, removed the malicious package versions from npm, and began systematically issuing "known-good" releases.
## Attack Methodology
- **Initial Access:** Injection of malicious code via a specially crafted Pull Request that successfully executed within the CI/CD pipeline due to insufficient privilege separation or validation.
- **Persistence:** The worm maintained persistence by using stolen npm credentials to continuously publish new malicious package versions to the registry, ensuring the threat persisted in dependency trees.
- **Privilege Escalation:** Exploitation of a bot's PAT with broad write permissions across the organization to commit further malicious changes.
- **Defense Evasion:** The use of a pre-install script executed automatically upon package installation bypassed many typical runtime security checks.
- **Credential Access:** Utilization of `TruffleHog` to scan developer machines/build systems for plaintext secrets. Methods targeted npm tokens, GitHub tokens, and cloud provider credentials (AWS, Azure, GCP).
- **Discovery:** Local reconnaissance using credential scanning tools on compromised build environments.
- **Lateral Movement:** The threat acted as a worm, propagating by embedding itself in dependent SDKs and spreading once those dependencies were installed by other developers/organizations.
- **Collection:** Stealing secrets directly from environment variables and configuration files on developer machines/build systems.
- **Exfiltration:** Uploading stolen secrets to attacker-controlled, public GitHub repositories.
- **Impact:** Widespread supply chain compromise affecting multiple high-profile organizations via third-party libraries.
## Impact Assessment
- **Financial:** Inferred significant costs related to incident response, remediation, and potential service downtime for affected clients.
- **Data Breach:** Compromise of numerous developer credentials and secrets, including tokens for:
* npm publishing.
* GitHub repository access.
* Cloud infrastructure (AWS, Azure, GCP).
* Sensitive CI/CD secrets and environment variables.
- **Operational:** Potential disruption to the development and deployment pipelines of PostHog and at least three other major organizations.
- **Reputational:** Described by PostHog as their "largest and most impactful security incident," leading to significant reputational damage within the developer community.
## Indicators of Compromise
Since the direct artifacts are not provided, these are behavioral/structural indicators:
- **Network Indicators (Defanged):** Outbound connections from build environments/developer machines attempting to access newly created public GitHub repositories immediately following package installation.
- **File Indicators:** Presence of installed npm packages containing suspicious pre-install scripts (`package.json` `preinstall` or `install` hooks). File activity indicative of `TruffleHog` scanning sensitive directories (e.g., `~/.aws/`, `~/.npm/`, `~/.ssh/`).
- **Behavioral Indicators:** Unscheduled executions of credential scanning tools on build servers. Authenticated API calls to GitHub or npm using compromised tokens shortly after package installation.
## Response Actions
- **Containment:** Revocation of all known and suspected compromised tokens (PATs and npm tokens). Isolation of affected build systems (implied).
- **Eradication:** Removal of malicious package versions from the npm registry.
- **Recovery:** Issuing and distributing "known-good" versions of the affected SDKs.
## Lessons Learned
- **Workflow Trust:** Blindly executing code from untrusted sources (like a PR author's branch) within high-privilege contexts (CI/CD) is a catastrophic design flaw.
- **Privilege Separation:** Bots and automation tools within CI/CD pipelines must operate under the Principle of Least Privilege; no single asset should possess broad write permissions (like publishing tokens) that can be leveraged by a successful code injection.
- **Dependency Security:** The dependency update mechanism itself (e.g., pre-install scripts) is a critical vulnerability vector when not strictly audited or disabled.
## Recommendations
- Implement mandatory **"Trusted Publisher" models** for all dependency releases (npm, etc.), requiring cryptographic signing or approval from multiple high-trust individuals before publishing.
- **Strictly audit and disable auto-execution of install scripts** (`postinstall`, `preinstall`) in all CI/CD environments unless absolutely necessary, treating them as hostile code.
- **Overhaul workflow change reviews:** Implement stricter, context-aware checks for PRs that modify workflow configurations or introduce new hooks/scripts.
- **Secret Management:** Rotate all secrets (especially publishing tokens) immediately and ensure they are only accessible by the necessary, narrowly scoped execution context.