Job Clean-up Policy

From version 25.2 onwards, the following harmonised job clean-up policy applies.

Introduction

InformationHaven uses a variety of jobs to organise the ingest and export of data and changes in metadata. Finished jobs are kept for diagnostic purposes. However, these jobs cannot be maintained indefinitely in the database because they would consume too much database volume over time.

In contrast, https://mediahaven.atlassian.net/wiki/spaces/CS/pages/4400775176 are kept forever. Events are also kept indefinitely, except for delete-related event types associated with permanently deleted records, including:

RECORDS.DELETE
RECORDS.DELETE.UNDELETE
RECORDS.DELETE.PERMANENTLY_DELETED

Policy

	Policy	Note	Default

	Policy	Note	Default
1	Cleanup completed jobs older than N days		30
2	Cleanup waiting jobs older than N days	This rarely occurs	14
3	Cleanup jobs with other statuses older than N days	Typically, these are failed jobs	365
4	Cleanup completed jobs when there are more than X completed jobs (up to the limit of X jobs).		10000

Concepts

The following concepts are cleaned up

Jobs from the https://mediahaven.atlassian.net/wiki/spaces/CS/pages/3949264897, including Batches
Jobs from the Transcoder
Jobs from the Export
- Export jobs from the scheduler are not deleted by policy #4
Jobs from the https://mediahaven.atlassian.net/wiki/spaces/CS/pages/3520724993
- The dead-letter jobs clean-up follows the above policy
- The historic jobs are cleaned up after 1 day
Jobs from the tape server (advanced installations)
Events for permanently deleted records, except “delete” related events.

Job Clean-up Policy

Introduction

Policy

Cleanup completed jobs older than N days

Cleanup waiting jobs older than N days

Cleanup jobs with other statuses older than N days

Cleanup completed jobs when there are more than X completed jobs (up to the limit of X jobs).

Concepts