For anything larger than a few hundred megabytes, upload to S3 from the AWS CLI, not from the S3 web console. The console pushes the file over a single HTTP connection, and a single TCP stream cannot saturate the link once there is real latency to the bucket's region. The CLI splits the file into parts and uploads them in parallel (multipart upload), which typically lands 3-4x faster. A real case: a 20.3 GiB tar file went from 1.5 MB/s in the browser (~4 hours) to 5.9 MiB/s from the CLI (~1 hour) on the same connection.
Upload the file
The basic form is one command. Multipart and parallelism are automatic; nothing needs to be enabled.
aws s3 cp msmarco_v2_passage.tar s3://msmarco-baris/Tune the parallelism
The defaults are conservative. Raising the number of concurrent part uploads and the part size gives the transfer a better chance of filling the link. These are one-time settings written to the AWS config, so they apply to later uploads too.
aws configure set default.s3.max_concurrent_requests 20 # parallel part uploads (default 10)
aws configure set default.s3.multipart_chunksize 64MB # size of each part (default 8MB)Keep a long upload alive
A multi-hour upload should not depend on the terminal staying open. Run it inside tmux or screen, or detach it with nohup so an SSH drop or a closed terminal does not kill the transfer.
nohup aws s3 cp msmarco_v2_passage.tar s3://msmarco-baris/ > upload.log 2>&1 &
tail -f upload.logNotes
- Region mismatch is fine: the CLI resolves the bucket's region itself. Add
--region <bucket-region>only if you actually hit a region error. - There is no auto-resume. If the upload is interrupted, rerun the same
cpcommand. - The browser console has the same fragility with none of the speed: navigating away from the tab kills the upload.