Resumable Uploads

Introduction

The MediaHaven REST API already allows for uploading files but this manner is not robust for uploading large files or when the network connection is unstable. Resumble uploads will fix this problem case, by offering a robust resumable uploads.

Design

Business Logic

• Introduction of a new MediaHaven Record Status named Uploading that indicates for /wiki/spaces/CS/pages/4064772158 that is file content will be uploaded in a resumable manner

• The upload API accepts the record ID as an input parameter. To obtain the corresponding session ID of the S3 multipart upload

  • Store the session ID as a new metadata field RecordInformation.ResumableUpload.SessionId with the type HiddenField

  • The workflow to be started when the resumable upload is completed will be stored in RecordInformation.ResumableUpload.Workflow

  • Stores the result in a memory cache, to prevent excessively contacting the REST API

• Once the resumable upload is completed the MediaHaven Record Status changes to New

Storage

The S3 storage is treated as one of /wiki/spaces/CS/pages/20643843 in the system to re-use many existing features

• A new cluster group “resumable_uploads” with the role TRANSIENT is created, highly similar to the existing cluster group “ingest”, which contains 1 shared storage pool for the S3 object store

• The uploading record will be linked with this shared storage pool

• The file is stored on the S3 object store using the standardized naming convention <Record ID>/<Record ID>.<Original Extension>

• The garbage collection will clean up this object store for this record when

  • The ingest workflow has transferred the record to the definitive storage

  • The record is permanently deleted (manually or automatically after being inactive for 2 weeks or longer)

• The used capacity will tracked by the standard “Storage-free space” workflow