Worker daemon fairness

Owned by Jasper Spruytte
Last updated: Jan 27, 2023 by Alexis Rombaut
2 min read

Priority

In order to improve the user experience of our applications, each job that is sent to the worker daemon is assigned a priority.

To determine the priority we have the following criteria:

Decision criteria

Decision

Decision criteria

Decision

Job duration

Introduced in 22.3

  • Job are catalogued as either SHORT or LONG

  • SHORT or LONG jobs are processed by separate worker daemons

  • This criterium ensures SHORT jobs are not affected by very LONG running jobs

Chained jobs

IMPROVED in 23.2

  • If the first step of the workflow is already completed, then jobs from this workflow are prioritized

  • This criterium ensures already active workflows are completed first

  • 23.2 Chained is only applied after the first LONG job

User priority

Each workflow is configured with a priority based on how quickly a user expects it to be processed.

The following levels exist, from highest to lowest:

  • HIGH: a user interaction triggered this process (UI change is triggered by this process) ex. series_evaluation, media_evaluation

  • NORMAL: propagation

  • LOW: ingest and processes triggered by external systems

  • BACKGROUND: migrations and timers

Organization fairness

Introduced in 22.2

Organizations are prioritized in a round-robin based fashion.

For example, if a job was processed for organization A, the priority for jobs belonging to org B are increased.

The upside for this is that organizations with higher loads don’t suppress jobs of other organisations

Zone

This concept is currently only used for DAV installations.

Worker daemons can be configured to listen for jobs assigned to a specific ‘zone’. This means that a worker daemon will only process jobs assigned to its zone. Multiple worker daemons can listen to the same zone.

One of the uses of zones to make sure files are only accessed from the correct machine. For example, we want to make sure a file has been checked for viruses before they are ever processed on the dav-<env>-archive machines.

Jobs are assigned to a specific zone in their workflow. The following table shows which zones are used by which workflows.

Zone

Workflows

Zone

Workflows

sip_ingest

  • sip_ingest

ingest

  • document_ingest

    • Subprocess of process ‘dossier’

default

  • document_archive

    • Subprocess of process ‘dossier’

  • Workflows without any zone specified.

Â