Amazon EMR Serverless introduces per job worker allocation and usage metrics with Amazon CloudWatch
Amazon EMR Serverless is a serverless option that makes it simple for data analysts and engineers to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers.
Amazon EMR Serverless is a serverless option that makes it simple for data analysts and engineers to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers.
We are excited to announce the launch of job worker metrics in Amazon CloudWatch for Amazon EMR Serverless. You can now monitor tracking vCPUs, memory, ephemeral storage, and disk I/O allocation and usage metrics at an aggregate worker level for your Apache Spark and Hive jobs. These new metrics provide granular insights into job performance, throughput, and resource utilization. This enables you to identify root causes for common errors and bottlenecks faster, analyze aggregate worker performance, and fine-tune your jobs for improved efficiency. For example, underutilization of vCPUs or memory can reveal resource wastage, allowing you to optimize worker sizes to achieve potential cost savings. Similarly, tracking spikes in ephemeral storage usage can help identify and mitigate disk bottlenecks by allocating more storage per worker. To get started, deploy the dashboard provided in the emr-serverless-samples Git repository to your account.
For more information on these metrics, visit the Job worker-level monitoring page in EMR Serverless User Guide.