hesela.devMultipart upload
Multipart upload is an object-transfer process that sends numbered parts independently and assembles them into one object when the upload is completed.
Amazon S3 initiates an upload, accepts parts under an upload ID, and assembles the submitted parts in part-number order at completion. The part ETags submitted in the completion manifest are distinct from the destination object's current ETag used by an If-Match precondition. These details describe S3; another object-storage service may implement different limits or semantics.
Relations
- related
- object-storage, entity-tag, conditional-write
Advantages
- Parts can transfer in parallel, and a failed part transfer can be retried without retransmitting parts that have already succeeded. [1]
- S3 CompleteMultipartUpload can apply If-None-Match or If-Match to the destination object when publishing the assembled upload. [2]
Limitations
- A multipart S3 object's ETag is not the MD5 digest of the complete object; integrity checks must use the documented checksum algorithm and full-object or composite scope. [3, 4]
- An in-progress S3 multipart upload does not reserve its destination key, so another writer can publish before its conditional completion. [5]
- A 409 ConditionalRequestConflict at S3 completion requires a new multipart upload and re-uploading every part; merely repeating completion with the old upload ID is insufficient. [2]
- Unfinished S3 uploads retain part storage until completed or aborted, so abandoned uploads need cleanup; the initial HTTP 200 at completion can also contain an embedded error that a raw API client must parse. [1, 2]
Citations
- Amazon S3: Uploading and copying objects using multipart upload
- Amazon S3 API Reference: CompleteMultipartUpload
- Amazon S3 API Reference: Object, ETag semantics
- Amazon S3: Checking object integrity for data uploads
- Amazon S3: How to prevent object overwrites with conditional writes
- Object ETags: checksums and conditional writes are different