/
Resumable Uploads

Resumable Uploads

Owned by Alexis Rombaut
Last updated: Jan 23, 2025
2 min read

This feature is introduced in version 25.1

Introduction

The MediaHaven REST API already allows for uploading files but this manner is not robust for uploading large files or when the network connection is unstable. Resumable uploads will fix this problem case, by offering a multipart chunked upload.

Design

API

Business Logic

Record status

  • Introduction of a new MediaHaven Record Status named Uploading that indicates for https://mediahaven.atlassian.net/wiki/spaces/CS/pages/4064772158 that its file content will be uploaded in a resumable manner

  • Once the resumable upload is completed the MediaHaven Record Status changes to New

Error handling

  • When the multipart upload to S3 encounters an error (i.e. incorrect checksum of a chunk), the record will be Rejectedwith an appropriate error reason

Storage

The S3 storage is treated as one of the https://mediahaven.atlassian.net/wiki/spaces/CS/pages/20643843 in the system to re-use many existing features

  • A new cluster group “resumable_uploads” with the role TRANSIENT is created, highly similar to the existing cluster group “ingest”, which contains 1 shared storage pool for the S3 object store

  • The uploading record will be linked with this shared storage pool and the session ID will also be stored as the distribution ID

  • The file is stored on the S3 object store using the standardized naming convention <Record ID>/<Record ID>.<Original Extension>

  • The garbage collection will clean up this object store for this record

    • When the ingest workflow has transferred the record to the definitive storage

    • When the record is permanently deleted (manually or automatically after being inactive for 2 weeks or longer)

    • When the upload is still in progress it can use the session ID to delete it

Activation

See https://mediahaven.atlassian.net/wiki/spaces/CS/pages/4606820353/Resumable+upload#Activation