Worker daemon fairness

Priority

In order to improve the user experience of our applications, each job that is sent to the worker daemon is assigned a priority.

To determine the priority we have the following criteria:

Decision criteria

Decision

Chained jobs

If another job belonging to the same workfow is already in the queue it’s priority is increased.

User priority

Each workflow is configured with a priority based on how quickly a user expects it to be processed.

The following levels exist, from highest to lowest:

HIGH: a user interaction triggered this process (UI change is triggered by this process) ex. series_evaluation, media_evaluation
NORMAL: propagation
LOW: ingest and processes triggered by external systems
BACKGROUND: migrations and timers

Organization fairness INTRODUCED IN 222

Organizations are prioritized in a round-robin based fashion.

For example, if a job was processed for organization A, the priority for jobs belonging to org B are increased.

The upside for this is that organizations with higher loads don’t suppress jobs of other organisations

Zone

This concept is currently only used for DAV installations.

Worker daemons can be configured to listen for jobs assigned to a specific ‘zone’. This means that a worker daemon will only process jobs assigned to its zone. Multiple worker daemons can listen to the same zone.

One of the uses of zones to make sure files are only accessed from the correct machine. For example, we want to make sure a file has been checked for viruses before they are ever processed on the dav-<env>-archive machines.

Jobs are assigned to a specific zone in their workflow. The following table shows which zones are used by which workflows.

Zone	Workflows
sip_ingest	sip_ingest
ingest	document_ingest Subprocess of process ‘dossier’
default	document_archive Subprocess of process ‘dossier’ Workflows without any zone specified.