Amazon EMR now supports Apache Spark 4.0.2 in general availability

Amazon EMR now supports Apache Spark 4.0.2 across all three deployment models. With Spark 4.0.2, you can build and maintain data pipelines more easily with ANSI SQL and VARIANT data types, enforce fine-grained access control (FGAC) at the row level or column level, strengthen compliance and governance frameworks with Apache Iceberg v3 table format, and deploy new real-time applications faster with enhanced streaming capabilities.

With Spark 4.0.2, you can build data pipelines, making data engineering accessible to a broader range of users through standard ANSI SQL support, eliminating the need to learn Spark-specific syntax. Spark 4.0.2 natively supports JSON and semi-structured data through VARIANT data types, providing flexibility for handling diverse data formats. You can enforce fine-grained access control (FGAC) on both read and write operations for AWS Lake Formation registered tables in your Apache Spark jobs. Building on these security capabilities, Apache Iceberg v3 table format provides stronger transaction guarantees and tracks data lineage, creating the audit trails required for regulatory compliance. Enhanced streaming controls simplify management of complex stateful operations and improve monitoring, enabling you to deploy real-time applications for fraud detection, personalization, and other time-sensitive use cases faster.

Apache Spark 4.0.2 is available in all regions where EMR is available. If you are upgrading your existing EMR application, you can use Apache Spark upgrade agent to accelerate your upgrades. To learn more about Apache Spark 4.0.2 on Amazon EMR, visit the Amazon EMR release notes, or get started by creating an EMR application with Spark 4.0.2 from the AWS Management Console.