Amazon EMR on EKS now supports Spark Operator and spark-submit
We are excited to announce that Amazon EMR on EKS now supports Spark Operator and spark-submit as new job submission models for Apache Spark, in addition to the existing StartJobRun API. With today’s launch, you now have the flexibility to submit your Apache Spark jobs via your preferred submission model on Amazon EMR on EKS without needing to change your application.
We are excited to announce that Amazon EMR on EKS now supports Spark Operator and spark-submit as new job submission models for Apache Spark, in addition to the existing StartJobRun API. With today’s launch, you now have the flexibility to submit your Apache Spark jobs via your preferred submission model on Amazon EMR on EKS without needing to change your application.
Prior to today’s launch, you could only submit Apache Spark jobs via the StartJobRun API, including using the AWS CLI and AWS Controllers for Kubernetes (ACK). Customers with existing Apache Spark applications running Spark Operator or spark-submit would have to make changes to their applications in order to use Amazon EMR on EKS. With this feature, you can now run your applications on EMR on EKS without changing them, benefit from the EMR Spark runtime performance and features, and save time by using the spark-submit and Spark Operator you are already familiar with.
To learn more about this feature, please visit the development guides for spark operator and spark-submit, as well as the AWS Big Data Blog post: Introducing Amazon EMR on EKS job submission with Spark Operator and spark-submit. Spark Operator and spark-submit are supported on Amazon EMR on EKS 6.10 and above, and available in the regions where Amazon EMR on EKS is currently available.