Full Report
Apache Spark is now the de facto standard for data engineering, data exploration and machine learning. Just likeKubernetes (k8s), is for automating containerized application deployment, scaling, and management. The open source ecosystem is now converging towards utilizing k8s as the compute platform in addition to YARN. Today, we are announcing the general availability of Dataproc on Google Kubernetes Engine (GKE), enabling you to leverage k8s to manage and optimize your compute platforms. You can now create a Dataproc cluster and submit Spark jobs on a self-managed GKE cluster. Dataproc on GKE for Spark (GA)K8s builds on 15 years of running Google's containerized workloads and the critical contributions from the open source community. Inspired by Google’s internal cluster management system, Borg, K8s makes everything associated with deploying and managing your application easier. With the widespread adoption of k8s, many customers are now standardizing on k8s for their compute platform management. We are observing a trend towards building applications as containers, to simplify application management among the many other benefits such as, improved agility, security, portability. Dataproc on GKE, now in GA, allows you to run Spark workloads on a self-managed GKE cluster. Letting you derive the benefits of fully automated, most scalable and cost optimized K8s service in the market.You bring your GKE cluster and create a Dataproc ‘virtual’ cluster on it . You can then submit jobs and monitor them the same as you would for Dataproc on Google Compute Engine (GCE). You use the Dataproc Jobs API to submit jobs on the cluster, you cannot use the open source Spark Submit directly. Jobs on Dataproc are submitted as native containers and you even have the ability to customize the containers to include additional libraries and data for your applications. Concepts for existing Dataproc on GCE usersNode Pool RolesDataproc uses GKE node pools to manage the Dataproc cluster configuration. You have the ability to select the machine type for the node pools. All the nodes in the node pool use the same configuration. Configuring the node pool with the following roles allows you to optimize Dataproc cluster configuration. Default: You must have at least a default role for a node pool. If other roles are not defined, default is used to run the workload. Controller: If defined, Dataproc control plane runs this node pool. This role has very low resource requirements. Spark Driver: if defined, Spark job drivers run in this node pool. This allows you to optimize the cluster configuration to the workload characteristics.Spark Executor: If defined, Spark job executors run in this node pool. This allows you to optimize the job executor environment. Workload IdentityDataproc uses GKE Workload Identity to allow pods within the GKE cluster to act with the authority of a linked Google Service Account. This is very similar to the default Service Account for the Dataproc on GCE.AutoscalingDataproc on GKE utilizes the GKE Cluster autoscaler. Once Dataproc has created the node pools, you can define autoscaling policies for the node pool to optimize your environment.Key BenefitsPreview customers with expertise in GKE were able to easily integrate Dataproc into their environments and are now looking forward to migrating Spark workloads and optimizing their execution environments to improve efficiency and save costs. Our advanced customers are exploring GPUs for improved job performance to meet their stringent SLAs needs. As we go GA, these customers are excited about utilizing the advanced k8s compute management and resource sharing for their production workloads. Running on GKE enables you to take advantage of the advanced capabilities of k8s enabling you optimize costs and performance by running: Completely independent jobs on the same cluster.You can now share a Dataproc cluster among multiple applications with distinct libraries and dependencies. Each Job can run its own container. Allowing independent Jobs with conflicting dependencies to run at the same time on the same Cluster. Earlier, each job with a distinct environment required an exclusive cluster. Relaxing this constraint enables customers to further optimize their execution environment. Multiple clusters on the same node pool.You can share the same infrastructure across multiple Dataproc clusters. You can run multiple Dataproc clusters on the same node pools, thereby allowing you further optimize costs. Some customers are now sharing multiple development environments on the same infrastructure. The same is applicable for testing, validation and certification environments. Multiple Spark versions on the same infrastructureYou can easily migrate from one version of Spark to another with the support for multiple versions on the same node pool. Your cluster management is simplified as you do not need to create two distinct environments and do not have to plan scaling down ‘existing’ cluster and scaling up the ‘upgraded’ cluster. Dataproc on GKE Key FeaturesFollowing are some of the salient features of Dataproc on GKESpark Versions: You can run Spark 2.4 and Spark 3.1 jobs on Dataproc on GKE clusters.Metastore Integration: You can integrate Dataproc on GKE with Dataproc Metastore.Job level access controls: You can now specify granular access controls at job level leveraging the k8s RBAC and workload identity. Uniform Dataproc APIs: you can use the same Dataproc APIs to manage clusters, submit jobs and use the same monitoring capabilities as Dataproc on GCE.Running Spark jobs with the infrastructure management style of your choiceWith the general availability of Dataproc on GKE, organizations can now run Spark jobs on their infrastructure management style of choice: Serverless Spark for no-ops deployment, customers standardizing on k8s for infrastructure management can run Spark on GKE to improve resource utilization and simplify infrastructure management. Customers looking for VM-style infrastructure management can run Spark on GCE.What’s NextWe are actively working on integrating Dataproc on GKE with Vertex AI Workbench for data scientists in the upcoming months. With this integration, data scientists can use notebooks for their interactive workloads and even schedule notebooks executions. We are also looking to extend Enhanced Flex Mode Support to Dataproc on GKE allowing you to maximize the benefit of preemptible VMs.To get started, check out this quickstart link.You can now take your knowledge of k8s compute management and leverage Dataproc on GKE to run Spark workload. Related Article Analyze Looker-modeled data through Google Sheets Connected Sheets for Looker brings modeled, trusted data into Google Sheets, enabling users to work in a way that is comfortable and conv... Read Article
Analysis Summary
# Industry News: Google Cloud Signals Data/Compute Convergence with Dataproc on GKE
## Summary
Google Cloud has announced the General Availability (GA) of Dataproc on Google Kubernetes Engine (GKE), unifying its managed Spark service with its flagship container orchestration platform. This move allows organizations to run Apache Spark workloads on self-managed GKE clusters, facilitating the consolidation of data engineering and application workloads onto a single Kubernetes-based compute fabric.
## Key Details
- **Date:** April 14, 2022
- **Companies Involved:** Google Cloud (Apache Spark community)
- **Category:** Product Launch / Infrastructure Modernization
## The Story
As Kubernetes becomes the standard for container orchestration, the open-source ecosystem is shifting away from YARN toward Kubernetes (k8s) as the primary compute platform for big data. Dataproc on GKE bridges the gap between data science and DevOps by allowing users to create "virtual" Dataproc clusters atop existing GKE infrastructure.
Unlike traditional Dataproc on Compute Engine (GCE), which relies on virtual machines, this integration treats Spark jobs as native containers. Key technical features include the use of GKE Node Pools to define specific roles (Controller, Spark Driver, Spark Executor), integration with GKE Workload Identity for security, and the ability to run multiple Spark versions and independent jobs with conflicting dependencies on the same physical infrastructure.
## Business Impact
### For the Companies Involved
- **Google Cloud:** Strengthens its "Data Cloud" value proposition by offering three distinct Spark deployment paths: Serverless, VM-based (GCE), and Container-based (GKE). This increases stickiness within the GKE ecosystem.
### For Competitors
- **AWS & Azure:** Increases pressure on Amazon EMR and Azure HDInsight to provide equally seamless Kubernetes integration. It positions Google as the leader in "K8s-native" data processing.
- **Databricks:** While Databricks offers its own managed environment, Google’s move appeals to "K8s-first" organizations that want to manage their own infrastructure rather than using a third-party abstraction.
### For Customers
- **Cost Optimization:** Customers can improve resource utilization by sharing the same GKE infrastructure across multiple Dataproc clusters and development environments.
- **Operational Efficiency:** Simplifies the CI/CD pipeline by allowing data applications to be bundled as containers with all necessary libraries, reducing "dependency hell."
### For the Market
- **Infrastructure Convergence:** Signals the end of the "siloed" data cluster. The market is moving toward a unified compute layer where microservices and heavy data processing jobs compete for the same resource pool.
## Technical Implications
- **Containerization of Spark:** Jobs are submitted via the Dataproc Jobs API as native containers, allowing for highly customized execution environments.
- **Granular Scaling:** Leverages GKE’s Cluster Autoscaler to dynamically adjust resources based on job demand.
- **Dependency Isolation:** Enables different jobs with conflicting library requirements to run concurrently on the same cluster, a significant improvement over traditional YARN-based clusters.
## Strategic Analysis
- **Market Positioning:** Google is positioning itself as the most flexible cloud provider for Spark, catering to both "No-Ops" users (Serverless) and "Full-Control" users (GKE).
- **Competitive Advantage:** Leveraging "Borg" heritage to offer superior K8s integration. The upcoming integration with Vertex AI Workbench will further bridge the gap between data engineering and AI/ML.
- **Challenges:** Users cannot use open-source `spark-submit` directly; they must use the Dataproc Jobs API, which may introduce minor friction for teams transitioning from pure open-source environments.
## Industry Reactions
- **Analyst Opinions:** Analysts view this as a necessary step to satisfy enterprise platform teams who have standardized on Kubernetes and view separate Hadoop/YARN clusters as legacy technical debt.
- **Market Response:** Early preview customers have highlighted the ability to meet stringent SLAs by utilizing GPUs within GKE for Spark jobs.
## Future Outlook
- **AI/ML Integration:** Watch for the upcoming Vertex AI Workbench integration, which will allow data scientists to trigger Spark-on-GKE jobs directly from notebooks.
- **Preemptible Optimization:** The extension of "Enhanced Flex Mode" to GKE will likely drive further cost reductions by better utilizing spot instances for fault-tolerant Spark workloads.
## For Security Professionals
- **Identity & Access Management (IAM):** The shift to **GKE Workload Identity** is a significant security upgrade. It allows pods to act with specific Google Service Account authority, adhering to the principle of least privilege more effectively than cluster-wide VM identities.
- **Granular Control:** Security teams can now leverage **Kubernetes RBAC** (Role-Based Access Control) in tandem with Dataproc APIs to provide job-level access controls.
- **Vulnerability Management:** Because Spark jobs now run as containers, security practitioners can apply the same container scanning and image registry security policies to data workloads that they apply to standard web applications.