Page Comparison

The Ingest Monitoring offers an overview of the items for your organisation in MediaHaven. Each item in MediaHaven has a list identifiers and a particular status and is linked with a series of events. Firstly we describe these concepts of status, events and identifiers. Secondly we describe how these concepts are used in the design of the ingest into MediaHaven and the representation in three views in the MediaHaven Ingest Monitoring.

ArchiveStatus

Value

Remark

Status

colour	Blue
title	on_ingest_tape

The item has been detected by the tape ingest server but has not yet been transferred to the ingest server.

Status

colour	Blue
title	in_progress

The item has been detected on the ingest server and is being processed.

Status

colour	Red
title	failed

The item is not successfully processed. The events for this item contain one or more

Status

colour	Red
title	NOK

events and these contain the reason for the failure.

Status

colour	Green
title	on_disk

The item is successfully archived on two or more disks. For files where the ultimate location is tape, this status will not appear.

Status

colour	Green
title	on_tape

The item is successfully archived on N tapes. Depending on the installation N can be 1 (archive), 2 (backup) or 3 (vault).

Status

colour	Green
title	completed

Only used for files of the type:

Ensemble (Collection, Set, Newspaper): all files in the ensemble have either ArchiveStatus
Status
colour Green
title on_disk
,
Status
colour Green
title on_tape
or
Status
colour Green
title completed
.
Metadata Only: the ArchiveStatus is immediately
Status
colour Green
title completed
upon creation

Events

MediaHaven logs the complete history of an item as a list of successive Premis events. The outcome of an event is either good

Status

colour	Green
title	ok

or bad

Status

colour	Red
title	nok

.

Here we offer an overview of event types.

Event

Description

Status

colourGrey

title	CREATE

The item has been created in MediaHaven. From this point on the three identifiers (see below) are defined.

Status

colourGrey

title	SIP_DETECTED

The item has been detected by either the tape ingest server

Status

title	tapeserver

or by the standard ingest service

Status

title	pretranscoder

Status

colourGrey

title	TAPE_ZIP

(tape ingest only) The item has been successfully packed into a ZIP archive.

Status

colourGrey

title	UPDATE

The metadata of the item has been updated in the database.

Status

colour

Grey

title	publish

The file has been manually published by the user or has been automatically published.

Status

colour

Grey

title	pretranscoded

Event generated by the standerd ingest service

Status

title	pretranscoder

to typically log validation errors (e.g. MD5 checksum, archive validation) with outcome

Status

colour	Red
title	nok

.

Status

colourGrey

title	transcode

The service

Status

title	transcoder

has started the transcoding and storage allocation of the item.

Status

colourGrey

title	transcoded

The service

Status

title	transcoder

has finished the transcoding and storage allocation of the item. For installations where the ultimate location is not tape, the item will have the status

Status

colour	Green
title	on_disk

from this point onward.

Status

colour

Grey

title	archiveD_on_tape_ARCHIVE

(For installations that write to 1 or more tapes) The item has been successfully written to an

Status

title	archive

tape. The comment of the event contains the barcode of the tape. If the item has been successfully written to all required tapes, the item will have the status

Status

colour	Green
title	on_tape

after the batch of tapes writes is complete.

Status

colour

Grey

title	archived_on_tape_BACKUP

(For installations that write to 2 or more tapes) The item has been successfully written to a

Status

title	backup

tape. The comment of the event contains the barcode of the tape. If the item has been successfully written to all required tapes, the item will have the status

Status

colour	Green
title	on_tape

after the batch of tapes writes is complete.

Status

colourGrey

title	archived_on_tape_VAULT

(For installations that write to 3 tapes) The item has been successfully written to a

Status

title	vault

tape. The comment of the event contains the barcode of the tape. If the item has been successfully written to all required tapes, the item will have the status

Status

colour	Green
title	on_tape

after the batch of tapes writes is complete.

Status

colourGrey

title	export

An export of the item has been requested by a user. The event can have outcome

Status

colour	Red
title	nok

if permission was denied.

Status

title	exported

The export of the item has been completed with outcome

Status

colour	Green
title	ok

or

Status

colour	Red
title	nok

.

Examples of the sequence of events

An item submitted using the standard online ingest and stored on disk:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Green
title UPDATE

Status
colour Green
title transcode

Status
colour Green
title transcoded
→
Status
colour Green
title on_disk
An item submitted using the standard online ingest with an MD5 checksum failure:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Red
title pretranscoded
→
Status
colour Red
title failed
An item submitted using the standard online ingest and stored on an archive and backup tape:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Green
title UPDATE

Status
colour Green
title transcode

Status
colour Green
title transcoded

Status
colour Green
title archived_on_tape_ARCHIVE

Status
colour Green
title archived_on_tape_BACKUP
→
Status
colour Green
title on_tape
An item submitted using the standard online ingest and but which could not be written to tape due to an internal error:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Green
title UPDATE

Status
colour Green
title transcode

Status
colour Green
title transcoded

Status
colour Red
title archived_on_tape_ARCHIVE

Status
colour Green
title archived_on_tape_BACKUP
→
Status
colour Red
title failed
An item read from an ingest tape and stored on an archive and backup tape:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Green
title TAPE_ZIP

Status
colour Green
title SIP_DETECTED

Status
colour Green
title UPDATE

Status
colour Green
title transcode

Status
colour Green
title transcoded

Status
colour Green
title archived_on_tape_ARCHIVE

Status
colour Green
title archived_on_tape_BACKUP
→
Status
colour Green
title on_tape
An item read from an ingest tape but with an MD5 checksum failure on the ingest tape:
Status
colour Green
title SIP_DETECTED

Status
colour Green
title CREATE

Status
colour Red
title TAPE_ZIP
→
Status
colour Red
title failed

Identification
Anchor
Identification
Identification

In order to unique identify items in the system MediaHaven uses three identifiers:

Umid (aka MediaObjectId): This is a 64 character hex string uniquely representing the item in MediaHaven.
FragmentId: This is a 96 character hex string representing a fragment of an item (e.g. a clip from a video, a page from a document). An item has always at least one fragment, termed the main fragment.
ExternalId: (Advanced installations only) The field contains the identifier of the item provided by the customer. The ingest flow is tightly coupled with this field: it will reject or accept a item based on whether the ExternalId already exists in the system. In the design section below we go into detail about this.

Views

The Ingest Monitoring offers three views:

SIPs: Offers an overview of items coupled with an ExternalId, showing only the most recent submission of each ExternalId.
items: Offers a complete history of items submitted to MediaHaven, including failed or deleted submissions.
Events: Offers the complete history of events in MediaHaven. When restricted to events of a single item, it shows the complete history of events that occurred to that item.

Design

The ingest views are designed around the following philosophy:

...

The screenshot below shows in the view

Status

title	SIps

the item with ExternalId

Status

title	ara_510_2242_00

with status

Status

colour	Green
title	on_tape

that was successfully archived on 07 Oct 2016.

Image Removed

...

Files

The screenshot below shows in the view

Status

title	files

the complete history of submissions of the item

Status

title	ara_510_2242_00

. The most recent submission has status

Status

colour	Green
title	on_tape

, while the previous submissions have status

Status

colour	Red
title	failed

and have been deleted (showed by a shaded or striped colour). In the example the item

Status

title	ara_510_2242_00

has been submitted 8 times. Notice that each submission has its own FragmentId while they share the same ExternalId. Due to design of the ingest, there can be exactly only one non-deleted item for the same ExternalId.

Image Removed

...

Events (succes)

If we look into the view

Status

title	events

of the successful

Status

colour	Green
title	on_tape

submission we see the following events. Notice that the item has been successfully written to the

Status

title	archive

tape

Status

title	a00001L7

and

Status

title	backup

tape

Status

title	B00001L7

. Afterwards it has been exported to an

Status

title	export

tape

Status

title	E00000L6

.

Image Removed

...

Events (failure)

If we look into the view

Status

title	events

of a

Status

colour	Red
title	failed

submission we see for example the following events.

Image Removed

...

We notice that the event

Status

title	pretranscodeD

with outcome

Status

colour	Red
title	nok

. The comment contains the following error:

Warning
METS: referred file does not exist: OriginalData/510_2242_000_00049_000/510_2242_000_00049_000_0_0001.tif

The submitted item was an archive and the METS file linked with it, described a file which was not found inside the archive.

Versions Compared

Old Version 9

New Version Current

Key

ArchiveStatus