Announcing generative AI troubleshooting for Apache Spark in AWS Glue (Preview)

AWS Glue announces generative AI troubleshooting for Apache Spark, a new capability that helps data engineers and scientists quickly identify and resolve issues in their Spark jobs. Spark Troubleshooting uses machine learning and generative AI technologies to provide automated root cause analysis for Spark job issues, along with actionable recommendations to fix identified issues.

AWS Glue is a serverless, scalable data integration service that makes it easier to discover, prepare, and combine data for analytics, machine learning, and application development. With Spark troubleshooting, you can initiate automated analysis of failed jobs with a single click in the AWS Glue console. This feature provides root cause analysis and remediation steps for hard-to-diagnose Spark issues like memory errors, data skew problems, and resource not found exceptions. This helps you reduce downtime in critical data pipelines. Powered by Amazon Bedrock, Spark troubleshooting reduces debugging time from days to minutes.

The generative AI troubleshooting for Apache Spark preview is available for jobs running on AWS Glue 4.0, and in the following AWS Regions: US East (N. Virginia), US West (Oregon), Europe (Ireland), US East (Ohio), and more. To learn more, visit the AWS Glue website, read the Launch blog, or read the documentation.