Full Report
We designed a behavioral clustering model to map cloud identity roles from audit logs, enabling continuous threat detection using standard SQL queries. The post Unmasking Cloud Identities: From Behavioral Clustering to Automated Detection appeared first on Unit 42.
Analysis Summary
# Research: Unmasking Cloud Identities: From Behavioral Clustering to Automated Detection
## Metadata
- **Authors:** Unit 42 (Palo Alto Networks)
- **Institution:** Unit 42 Research
- **Publication:** Unit 42 Blog
- **Date:** October 30, 2024
## Abstract
This research introduces a novel methodology for securing cloud environments by focusing on the "behavioral identity" of cloud roles rather than static permissions. By applying unsupervised machine learning (K-Means clustering) to AWS CloudTrail audit logs, the researchers developed a system to group cloud identities based on their actual API usage patterns. This creates a "behavioral map" that allows security teams to detect anomalies—such as a developer role suddenly performing administrative tasks—using standard SQL queries, effectively bridging the gap between complex data science and practical security operations.
## Research Objective
The research addresses the difficulty of monitoring complex cloud environments where traditional Rule-Based Detection (RBD) fails to account for the sheer volume of unique identities and the fluidity of their permissions. The core question is: *How can we automatically categorize cloud identities based on what they actually do, and use that categorization to detect potential compromises or insider threats?*
## Methodology
### Approach
1. **Data Extraction:** Aggregating AWS CloudTrail logs to capture API calls.
2. **Feature Engineering:** Transforming raw logs into a numerical format. Each identity is represented by a vector where each dimension corresponds to the frequency of a specific API call.
3. **Clustering:** Applying the **K-Means algorithm** to group identities with similar API call profiles.
4. **Thresholding:** Calculating the distance of each identity from its cluster centroid to identify outliers.
5. **Detection Translation:** Converting the resulting clusters into SQL-based detection logic for real-time monitoring.
### Dataset/Environment
The research utilized large-scale AWS CloudTrail datasets, focusing on diverse API actions (e.g., `DescribeInstances`, `CreateUser`, `PutObject`) across multiple accounts and service roles.
### Tools & Technologies
- **AWS CloudTrail:** Primary source of behavioral data.
- **Python (Scikit-learn):** Used for initial K-Means modeling and feature scaling.
- **SQL (Trino/Athena):** Used for the operationalization of the detection model.
## Key Findings
### Primary Results
1. **Behavioral Consistency:** Cloud identities naturally form distinct clusters based on their functional roles (e.g., "Network Admins," "Read-Only Auditors," "DevOps Automation").
2. **SQL Operationalization:** Behavioral clustering results can be successfully exported into simple SQL `CASE` statements, allowing for high-performance anomaly detection without requiring a live ML inference engine.
3. **Anomaly Identification:** The model effectively identified "Role-Defying" behavior, where an identity begins acting like a member of a different, potentially more privileged, cluster.
### Supporting Evidence
- The researchers demonstrated that by using the **Elbow Method** and **Silhouette Scores**, they could determine the optimal number of clusters (*k*) to minimize "noise" in detection.
### Novel Contributions
- **The "Behavioral Map":** Moving beyond static Least Privilege checks to dynamic behavioral profiling.
- **ML-to-SQL Pipeline:** A framework that trains complex models offline but executes detections using standard database queries, making it highly scalable for SOC teams.
## Technical Details
The core of the detection lies in the **Centroid Calculation**. After clustering, each cluster has a "center" representing the average behavior of its members.
- **Intra-cluster Anomaly:** An identity that drifts too far from its own centroid (detected via Euclidean distance).
- **Cross-cluster Anomaly:** An identity that suddenly shifts its coordinates to align with a more sensitive cluster (e.g., a "Guest" role moving toward the "IAM Admin" cluster).
## Practical Implications
### For Security Practitioners
- Shift from managing thousands of individual alerts to monitoring a few dozen "behavioral archetypes."
- Identification of "Over-permissioned" roles that possess high privileges but only exhibit low-privilege behavior.
### For Defenders
- **Actionable Insight:** Use the provided SQL structure to create alerts when an identity performs an API call that is statistically foreign to its assigned behavioral cluster.
- **Reduced False Positives:** By grouping similar identities, the "normal" baseline is more robust than per-user baselines.
### For Researchers
- The methodology provides a blueprint for applying unsupervised learning to other audit log types, such as Kubernetes (K8s) audit logs or GitHub activity logs.
## Limitations
- **Cold Start Problem:** New identities with no history are difficult to cluster accurately.
- **Feature Sparsity:** In very large environments with thousands of unique API calls, the feature matrix can become extremely sparse, requiring dimensionality reduction (like PCA).
- **Adversarial Adaptation:** A sophisticated attacker who moves slowly and mimics the existing cluster behavior ("living off the land") may evade detection.
## Comparison to Prior Work
Traditional Cloud Infrastructure Entitlement Management (CIEM) focuses on **what a role *can* do** (permissions). This research focuses on **what a role *does* do** (behavior). Unlike previous ML approaches that require expensive real-time scoring engines, this model emphasizes **portability** by converting ML insights into SQL.
## Real-world Applications
- **Credential Theft Detection:** If an attacker steals a developer's keys and begins running discovery commands, they will move out of the "Developer" cluster.
- **Policy Refinement:** Identifying roles that are in the "Admin" cluster but only behave like "Viewers," suggesting their permissions should be revoked.
## Future Work
- Integrating **Temporal Analysis** (detecting shifts in behavior over time rather than just static snapshots).
- Expanding the model to include **Resource-based features** (not just *what* was done, but *which* specific sensitive bucket was accessed).
## References
- *Unit 42: Unmasking Cloud Identities* (Original Article)
- *AWS CloudTrail Documentation*
- *Scikit-Learn K-Means Clustering Guide*