Amazon MSK Replicator now supports enhanced consumer offset synchronization for bidirectional replication

Amazon MSK Replicator now provides enhanced consumer offset synchronization for bidirectional replication, enabling applications to resume processing from the correct position when moving across Kafka clusters. This capability enables you to move producer and consumer applications between clusters independently, in any order, without the risk of data loss.

MSK Replicator is a feature of Amazon MSK that automates data replication between Kafka clusters, eliminating the need to manage custom replication infrastructure or configure open-source tools. Previously, while replicating bidirectionally with MSK Replicator, consumer group offsets were synchronized only when producers and consumers were active on the same cluster, requiring careful sequencing of application migrations between clusters and increasing the risk of duplicate message processing during rollbacks. With this launch, MSK Replicator synchronizes consumer group offsets across source and target clusters regardless of where producers are running, enabling applications to move between clusters without coordination constraints or data duplication risks.

You can enable enhanced consumer offset synchronization when creating a Replicator using the Amazon MSK console, AWS CLI, or AWS CloudFormation. This capability is supported in all AWS Regions where MSK Replicator is available.

To learn more, visit the MSK Replicator documentation, product page, pricing page, and this AWS blog post.