New updates automatically accelerate Amazon S3 data transfer for ML training
Starting today, the AWS Command Line Interface (AWS CLI) and Python SDK automatically use the AWS Common Runtime (CRT) to accelerate data transfer between Amazon S3 and Amazon EC2 Trn1, P4d, and P5 instances. The AWS CRT implements Amazon S3's performance best practices for request parallelization, automatic retry, DNS load balancing, and more to deliver high data transfer rates between Amazon EC2 and Amazon S3. As a result, machine learning training jobs now download training data from Amazon S3 up to 3x faster and upload model checkpoints to Amazon S3 up to 5x faster, which accelerates total training times.
Starting today, the AWS Command Line Interface (AWS CLI) and Python SDK automatically use the AWS Common Runtime (CRT) to accelerate data transfer between Amazon S3 and Amazon EC2 Trn1, P4d, and P5 instances. The AWS CRT implements Amazon S3's performance best practices for request parallelization, automatic retry, DNS load balancing, and more to deliver high data transfer rates between Amazon EC2 and Amazon S3. As a result, machine learning training jobs now download training data from Amazon S3 up to 3x faster and upload model checkpoints to Amazon S3 up to 5x faster, which accelerates total training times.
This change is automatically included in the latest AWS Deep Learning AMIs (DLAMI) when launching Amazon EC2 Trn1, P4d, and P5 instances, which are ideal for generative AI models, including large language and diffusion models. Now, applications that use the AWS CLI and Python SDK to access Amazon S3 automatically get the performance benefits of the AWS CRT. The AWS CRT optimizes for the high network bandwidth available on these instances, so you can get the most out of your compute resources without manually tuning storage performance. To learn more, visit the Python SDK documentation and the CLI documentation.