Full Report
Unit 42 discovered a Vertex AI Python SDK vulnerability that allows remote code execution via bucket squatting. Read the article for more. The post Pickle in the Middle – Hijacking Vertex AI Model Uploads for Cross-Tenant RCE appeared first on Unit 42.
Analysis Summary
# Vulnerability: Vertex AI SDK Bucket Squatting via Pickle Deserialization
## CVE Details
- **CVE ID**: CVE-2024-37168
- **CVSS Score**: 8.8 (High)
- **CWE**: CWE-502 (Deserialization of Untrusted Data) / CWE-427 (Uncontrolled Search Path Element)
## Affected Systems
- **Products**: Vertex AI SDK for Python (`google-cloud-aiplatform`)
- **Versions**: All versions prior to v1.53.0
- **Configurations**: Systems using the `aiplatform.Model.upload()` or `aiplatform.Endpoint.deploy()` methods where a Google Cloud Storage (GCS) bucket is not explicitly specified for staging artifact uploads.
## Vulnerability Description
The flaw stems from a "bucket squatting" condition combined with unsafe Python pickle deserialization. When a user uploads a model using the Vertex AI SDK without specifying a staging bucket, the SDK automatically generates a default bucket name based on the project ID and region (e.g., `temp-vertex-staging-<region>-<project-id>`).
If this bucket does not already exist, an attacker can pre-emptively create (squat) a bucket with that exact name. During a model upload, the SDK scripts package model artifacts into a `.pkl` file. Because the attacker-controlled bucket is used as the staging area, the attacker can inject a malicious pickle payload. When the SDK or the Vertex AI service backend processes this file, it triggers remote code execution (RCE) via unsafe deserialization.
## Exploitation
- **Status**: PoC available (demonstrated by Unit 42); no known exploitation in the wild.
- **Complexity**: Medium (Requires knowledge of the target's Project ID and Region).
- **Attack Vector**: Network (Cross-tenant).
## Impact
- **Confidentiality**: High (Access to service account tokens and sensitive model data).
- **Integrity**: High (Ability to modify models and execute arbitrary code).
- **Availability**: High (Ability to disrupt AI workloads).
## Remediation
### Patches
- **google-cloud-aiplatform v1.53.0**: This version mitigates the issue by verifying bucket ownership and changing the default behavior for artifact handling.
### Workarounds
- **Explicit Bucket Definition**: Always manually define a staging bucket that you own and have restricted permissions on when calling `init()` or `upload()` methods.
- **Bucket Locks**: Ensure the required staging buckets are created and locked within your organization so they cannot be claimed by external actors.
## Detection
- **Indicators of Compromise**:
- Unexpected GCS buckets in the project environment or external buckets being accessed during model upload workflows.
- Presence of `model.pkl` files containing `RCE` payloads (e.g., `os.system` or `subprocess` calls).
- **Detection Methods**:
- Audit Cloud Storage logs for `storage.objects.create` or `storage.objects.get` calls originating from Vertex AI service accounts to buckets not owned by the organization.
- Scan Python environments for vulnerable versions of `google-cloud-aiplatform`.
## References
- **Unit 42 Blog**: hxxps[://]unit42[.]paloaltonetworks[.]com/pickle-in-the-middle-hijacking-vertex-ai-model-uploads/
- **Google Cloud Security Advisory**: hxxps[://]cloud[.]google[.]com/security/bulletins/docs/google-cloud-security-advisories
- **GitHub Repository**: hxxps[://]github[.]com/googleapis/python-aiplatform/releases/tag/v1.53.0