AWS Lambda announces low latency processing for Kafka events

AWS Lambda now supports low latency (sub 100ms) event processing for Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka event sources in Provisioned mode for Kafka ESM. Customers can now set their MaximumBatchingWindowInSeconds parameter to 0 in Kafka ESM configurations, enabling real-time processing of Kafka events. This enhancement significantly reduces end-to-end processing latency for time-sensitive business applications.

Kafka customers increasingly build mission-critical applications that require consistent end-to-end latency of less than 100ms to meet stringent business requirements across industries. Examples include financial services firms processing market data feeds and executing algorithmic trades, e-commerce platforms providing real-time personalized recommendations, and gaming companies managing live player interactions. With today’s launch, Lambda natively supports low latency event processing with efficient optimization of polling and invoking Kafka events, allowing customers’ to build mission-critical or latency-sensitive Kafka applications on Lambda. With MaximumBatchingWindowInSeconds set to 0, Kafka ESM invokes the function with Kafka events immediately after the previous invocation completes. This configuration makes end-to-end latency solely dependent on function duration, thus potentially providing average 50ms end-to-end latencies for critical real-time applications.

This feature is generally available in all AWS Commercial Regions where AWS Lambda Kafka ESM is available, except Israel (Tel Aviv), Asia Pacific (Malaysia), and Canada West (Calgary).

To enable your Kafka ESM for low latency processing, set MaximumBatchingWindowInSeconds set to 0 and enable Provisioned mode for your new or existing Kafka ESM in the ESM API, AWS Console, AWS CLI, AWS SDK, AWS CloudFormation, and AWS SAM. To learn more, read Lambda ESM documentation and AWS Lambda pricing.