Announcing generative AI upgrades for Apache Spark in AWS Glue (preview)

AWS Glue announces generative AI upgrades for Apache Spark, a new generative AI capability that enables data practitioners to quickly upgrade and modernize their existing Spark jobs. Powered by Amazon Bedrock, this feature automates the analysis and updating of Spark scripts and configurations, reducing the time and effort required for Spark upgrades from weeks to minutes.

AWS Glue is a serverless, scalable data integration service that makes it easier to discover, prepare, and combine data for analytics, machine learning, and application development. With Spark Upgrades, you can initiate automated upgrades with a single click in the AWS Glue console to modernize your Spark jobs from an older version to AWS Glue version 4.0. This feature analyzes your Python-based Spark jobs and generates upgrade plans detailing code changes and configuration modifications. It leverages generative AI to iteratively improve and validate the upgraded code by executing test runs as Glue jobs. Once validation is successful, you receive a detailed summary of all changes for review, enabling confident deployment of your upgraded Spark jobs. This automated approach reduces the complexity of Spark upgrades while maintaining the reliability of your data pipelines.

The generative AI upgrades for Apache Spark preview is available for AWS Glue in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Asia Pacific (Sydney). To learn more, visit the AWS Glue website, read the Launch blog, or read the documentation.