Full Report
Why adding nodes to your primary site hurts performance, and how to scale with secondary sites instead
Analysis Summary
# Best Practices: Symantec PAM Clustering & Scaling
## Overview
These practices address the optimization of Privileged Access Manager (PAM) architectures. They focus on balancing high availability (HA) with system performance by leveraging synchronous replication for local resilience and asynchronous replication for geographic scaling and disaster recovery.
## Key Recommendations
### Immediate Actions
1. **Cap Primary Site at Three Nodes:** Audit your current PAM deployment; if the primary site exceeds three nodes, plan to decommission or move extra nodes to avoid unnecessary consensus latency.
2. **Maintain Odd Node Counts:** Ensure the primary site always consists of an odd number of members (specifically three) to satisfy MySQL Group Replication quorum requirements efficiently.
3. **Validate Quorum Status:** Confirm that the primary site can tolerate a single node failure without losing the ability to commit transactions (2 out of 3 nodes).
### Short-term Improvements (1-3 months)
1. **Deploy Secondary Sites for Scaling:** Instead of adding nodes to the primary cluster to handle increased user load, deploy a secondary site.
2. **Implement Asynchronous Replication:** Configure secondary sites to receive data streams asynchronously to ensure WAN latency between data centers does not stall primary site transactions.
3. **Designate Site Leaders:** Configure secondary site "leaders" to manage the incoming replication stream from the primary site, reducing cross-WAN traffic.
### Long-term Strategy (3+ months)
1. **Architect for Site-Level Resiliency:** Develop and test a promotion plan where a secondary site is promoted to Primary status in the event of a total regional data center outage.
2. **Zero-Downtime Scaling Model:** Establish a workflow for adding or removing nodes from secondary sites during peak loads without requiring maintenance windows or primary site downtime.
## Implementation Guidance
### For Small Organizations
- Stick to a single **3-node primary cluster**. This provides full HA and handles a high volume of concurrent sessions without the complexity of multiple sites.
### For Medium Organizations
- Deploy a **3-node primary site** and a **2-node secondary site** in a different geographic region or availability zone. This provides DR (Disaster Recovery) capabilities without degrading performance.
### For Large Enterprises
- Utilize a **3-node primary site** for core operations.
- Deploy **multiple secondary sites** distributed globally to place PAM resources closer to users.
- Use the "Site Leader" configuration to minimize global WAN traffic.
## Configuration Examples
**Quorum Calculation for Primary Site:**
- **3 Nodes:** 2 confirmations required (1 external + 1 local). *Optimal.*
- **4 Nodes:** 3 confirmations required (2 external + 1 local). *Sub-optimal: Higher latency, same fault tolerance as 3 nodes.*
- **5 Nodes:** 3 confirmations required. *Sub-optimal: Higher latency.*
**Replication Logic:**
- **Intra-Site:** Synchronous (MySQL Group Replication).
- **Inter-Site:** Asynchronous (Streaming via Site Leader).
## Compliance Alignment
- **NIST SP 800-53:** Aligns with CP (Contingency Planning) and SC (System and Communications Protection) controls by ensuring availability and low-latency security processing.
- **ISO/IEC 27001:** Supports Availability (A) of the CIA triad and Business Continuity Management.
- **CIS Controls:** Aligns with Control 11 (Data Recovery) and Control 6 (Access Control Management).
## Common Pitfalls to Avoid
- **"Cheap Insurance" Fallacy:** Adding a 4th or 5th node to a primary site thinking it increases safety; it actually increases the risk of performance stalls due to CPU spikes or network jitter.
- **Synchronous Replication over WAN:** Never include nodes separated by high latency in the same synchronous primary cluster, as it will bottleneck every privileged credential checkout.
- **Even Number Clusters:** Avoid 2 or 4 node clusters in the primary site, as they increase the quorum threshold without providing additional fault tolerance.
## Resources
- **Symantec TechDocs:** `hXXps://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/4-3-1/deploying/set-up-a-cluster.html`
- **Cluster Sync & Recovery:** `hXXps://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/privileged-access-manager/4-3-1/deploying/set-up-a-cluster/cluster-synchronization-promotion-and-recovery.html`
- **Broadcom Tech Insights:** `hXXps://www.security.com/product-insights`