AWS Lambda announces native support for Avro and Protobuf formatted Kafka events

AWS Lambda now provides native support for Avro and Protobuf formatted Kafka events with Apache Kafka’s event-source-mapping (ESM), and integrates with AWS Glue Schema registry (GSR), Confluent Cloud Schema registry (CCSR), and self-managed Confluent Schema registry (SCSR) for schema management. This enables you to validate your schema, filter events, and process events using open-source Kafka consumer interfaces. Additionally, customers can use Powertools for AWS Lambda to process their Kafka events without writing custom deserialization code, making it easier to build their Kafka applications with AWS Lambda.

Kafka customers use Avro and Protobuf formats for efficient data storage, fast serialization and deserialization, schema evolution support, and interoperability between different programming languages. They utilize schema registry to manage, evolve, and validate schemas before data enters processing pipelines. Previously, customers were required to write custom code within their Lambda function, in order to validate, de-serialize, and filter events when using these data formats. With today's launch, Lambda natively supports Avro and Protobuf as well as integration with GSR, CCSR and SCSR, enabling customers to process their Kafka events using these data formats, without writing custom code. Additionally, customers can optimize costs through event filtering to prevent unnecessary function invocations.

This feature is generally available in all AWS Commercial Regions where AWS Lambda Kafka ESM is available, except Israel (Tel Aviv), Asia Pacific (Malaysia), and Canada West (Calgary).

To get started, provide your schema registry configuration for your new or existing Kafka ESM in the ESM API, AWS Console, AWS CLI, AWS SDK, AWS CloudFormation, and AWS SAM. Optionally, you can setup filtering rules to discard irrelevant Avro or Protobuf formatted events before function invocations. To build your function with Kafka’s open-source ConsumerRecords interface, add Powertools for AWS Lambda as a dependency within your Lambda function. To learn more, read Lambda ESM documentation and AWS Lambda pricing.