Amazon SageMaker contributes a custom transport to OpenLineage community and offers additional lineage capabilities

AWS announces that Amazon SageMaker has contributed a custom transport ‘AmazonDataZoneTransport’ to the OpenLineage community and enhanced automated lineage capabilities. These lineage enhancements include improvements to automation from sources such as AWS Glue, Amazon Redshift, and automated lineage capture from tools, enabling data scientists and engineers to work more efficiently with their data and models.

The new 'custom transport' contribution to the OpenLineage community allows builders to download the transport along with OpenLineage plugins to augment and automate lineage events captured from OpenLineage-enabled systems. With this, customers can automate lineage capture and send these lineage events to the SageMaker Unified Studio domain, enhancing data governance and traceability within their data workflows. Amazon SageMaker has also introduced enhanced automated lineage capabilities from various sources. These improvements include better support for lineage events from AWS Glue, Amazon Redshift, and automated lineage capture from tools such as vETL processes and notebooks. Additionally, SageMaker has improved its SQL lineage support, particularly for Amazon Redshift, with new features including support for stored procedures and materialized views. These enhancements enable automatic lineage capture of complex data operations, providing a more comprehensive view of data transformations and dependencies.

This feature is available all AWS Regions where Amazon SageMaker is available.

To learn more about the custom transport contribution and enhanced lineage capabilities, visit the Amazon SageMaker. page. For detailed information on how to get started with lineage using these new features, refer to the user documentation.