Pools

Introduction

Files archived by MediaHaven / DigiHaven are written to storage using a particular number of copies.

For example

1 copy: This case is not advised as it goes against our archival mandate
2 copies: The most common case where we write a main and backup copy on disk
3 copies: The case at Meemoo where we write to A, B and C tapes

Concepts

The storage concepts in the engine have a tree structure with progressive information, each concept is a child concept of the previous.

Super clusters : groups pools which are mirrored across all organisations (future concept)
Cluster groups: groups pools across all organisations having the same role (one of the mirrored copies)
Clusters: pool from a cluster group for a specific organisation
Storage Pools: pool corresponding with a protocol

Example

Super cluster for all LTO-8 tapes named “tape_8” with contains all A, B and C tapes
Cluster group for all LTO-8 archive tapes named “tape_archive_8”
Cluster for the archive LTO-8 tapes of the organisation vrt named “vrt_tape_archive_8”
Storage Pool for the tape A00154L8 belonging to the cluster “vrt_tape_archive_8”

Tree

In this diagram, the 3rd copy, namely VAULT/C tapes are omitted.

Properties

Concept	Property	Examples	Description

Concept	Property	Examples	Description
Cluster group	cluster group ID		Unique label for the cluster group
Cluster group	type	MOB, BROWSE, CACHE	MOB: Original file BROWSE: Preview file CACHE: Cached file used by the worker daemon
Cluster group	role		ARCHIVE: 1st copy BACKUP: 2nd copy VAULT: 3rd copy DISTRIBUTION: linked with an external system such as YouTube TRANSIENT: non definitive storage such a GPFS buffer EXTERNAL: linked with another MediaHaven installation (syncer)
Cluster group	tape_like	`A%L8` `IM%L6`	Pattern to which the tape barcodes correspond
Cluster group	Free space: calculate		If true, calculate the free space for all pools in this cluster groups
Cluster group	Free space: storage path	`mh-dev-backup.mediahaven.com:/mnt/STORAGE/POOL2`	Server specified by the URI to contact to determine the free space parameters of the volume mounted on the path specified by the URI
Cluster group	Free space: add queued space		Whether to add the queued space to the free space, must be enabled when the ingest and archive storage share the same physical volume
Cluster	cluster ID	`vrt_tape_archive_8`	Unique label for the cluster
Cluster	organisation ID	`100`	restricts the cluster group to particular organisation multiple clusters for the same cluster group and organisation is allowed
Storage Pool	storage pool ID	`100154`	Unique ID (integer) for the storage pool
Storage Pool	ID in cluster	1	When allocating storage it selects mirrored copies across pools of multiple cluster groups having the same ID in cluster
Storage Pool	Total Space		Total storage in bytes
Storage Pool	Free Space		Remaining storage in bytes
Storage Pool	Keep Free Space		Storage space in base to keep free as guideline Setting keep free space to total space, makes the pool illegible for writing
Storage Pool	Queued space		Free space likely to be consumed in the future, i.e. when scheduling jobs to write to tape
Storage Pool	Is Online	`true` `false`	When false this pool is currently unavailable, for example when a tape is removed from the tape library
Storage Pool	Protocol, Credentials, Server, Port, Path	ftp://user:***@server:port/path	Forms together an URI
Storage Pool	Class	`HOT` `COOL`	Object storage can write files in different modes

Queued Space

The queued space tracks future free space to be consumed by pending ingest jobs/workflows or exports from the scheduler
When POSTing a record via the REST API
1. The file is transferred to the TOTRANSCODER pool
2. The queued space of the TOTRANSCODER pool is increased by the file size of the new record
3. When the corresponding ingest workflow of record transfers the file from TOTRANSCODER to the final storage, the queued space of the TOTRANSCODER pool is decreased by the file size of the new record