Configuring channels and the job runner

What is the job runner?

The job runner is the main process managing the dispatch of delayed jobs to available Odoo workers

How does it work?

  • It starts as a thread in the Odoo main process or as a new worker
  • It receives postgres NOTIFY messages each time jobs are added or updated in the queue_job table.
  • It maintains an in-memory priority queue of jobs that is populated from the queue_job tables in all databases.
  • It does not run jobs itself, but asks Odoo to run them through an anonymous /queue_job/runjob HTTP request. [1]

How to use it?

  • Optionally adjust your configuration through environment variables:
    • ODOO_QUEUE_JOB_CHANNELS=root:4 (or any other channels configuration), default root:1.
    • ODOO_QUEUE_JOB_SCHEME=https, default http.
    • ODOO_QUEUE_JOB_HOST=load-balancer, default http_interface or localhost if unset.
    • ODOO_QUEUE_JOB_PORT=443, default http_port or 8069 if unset.
    • ODOO_QUEUE_JOB_HTTP_AUTH_USER=jobrunner, default empty.
    • ODOO_QUEUE_JOB_HTTP_AUTH_PASSWORD=s3cr3t, default empty.
    • ODOO_QUEUE_JOB_JOBRUNNER_DB_HOST=master-db, default db_host or False if unset.
    • ODOO_QUEUE_JOB_JOBRUNNER_DB_PORT=5432, default db_port or False if unset.
  • Alternatively, configure the channels through the Odoo configuration file, like:
[queue_job]
channels = root:4
scheme = https
host = load-balancer
port = 443
http_auth_user = jobrunner
http_auth_password = s3cr3t
jobrunner_db_host = master-db
jobrunner_db_port = 5432
  • Or, if using anybox.recipe.odoo, add this to your buildout configuration:
[odoo]
recipe = anybox.recipe.odoo
(...)
queue_job.channels = root:4
queue_job.scheme = https
queue_job.host = load-balancer
queue_job.port = 443
queue_job.http_auth_user = jobrunner
queue_job.http_auth_password = s3cr3t
  • Start Odoo with --load=web,web_kanban,queue_job and --workers greater than 1 [2], or set the server_wide_modules option in The Odoo configuration file:
[options]
(...)
workers = 4
server_wide_modules = web,web_kanban,queue_job
(...)
  • Or, if using anybox.recipe.odoo:
[odoo]
recipe = anybox.recipe.odoo
(...)
options.workers = 4
options.server_wide_modules = web,web_kanban,queue_job
  • Confirm the runner is starting correctly by checking the odoo log file:
...INFO...queue_job.jobrunner.runner: starting
...INFO...queue_job.jobrunner.runner: initializing database connections
...INFO...queue_job.jobrunner.runner: queue job runner ready for db <dbname>
...INFO...queue_job.jobrunner.runner: database connections ready
  • Create jobs (eg using base_import_async) and observe they start immediately and in parallel.
  • Tip: to enable debug logging for the queue job, use --log-handler=odoo.addons.queue_job:DEBUG

Caveat

  • After creating a new database or installing queue_job on an existing database, Odoo must be restarted for the runner to detect it.
  • When Odoo shuts down normally, it waits for running jobs to finish. However, when the Odoo server crashes or is otherwise force-stopped, running jobs are interrupted while the runner has no chance to know they have been aborted. In such situations, jobs may remain in started or enqueued state after the Odoo server is halted. Since the runner has no way to know if they are actually running or not, and does not know for sure if it is safe to restart the jobs, it does not attempt to restart them automatically. Such stale jobs therefore fill the running queue and prevent other jobs to start. You must therefore requeue them manually, either from the Jobs view, or by running the following SQL statement before starting Odoo:
update queue_job set state='pending' where state in ('started', 'enqueued')

Footnotes

[1]From a security standpoint, it is safe to have an anonymous HTTP request because this request only accepts to run jobs that are enqueued.
[2]It works with the threaded Odoo server too, although this way of running Odoo is obviously not for production purposes.

What is a channel?

class odoo.addons.queue_job.jobrunner.channels.Channel(name, parent, capacity=None, sequential=False, throttle=0)[source]

A channel for jobs, with a maximum capacity.

When jobs are created by queue_job modules, they may be associated to a job channel. Jobs with no channel are inserted into the root channel.

Job channels are joined in a hierarchy down to the root channel. When a job channel has available capacity, jobs are dequeued, marked as running in the channel and are inserted into the queue of the parent channel where they wait for available capacity and so on.

Job channels can be visualized as water channels with a given flow limit (= capacity). Channels are joined together in a downstream channel and the flow limit of the downstream channel limits upstream channels.:

---------------------+
                     |
                     |
 Ch. A C:4,Q:12,R:4  +-----------------------

---------------------+  Ch. root C:5,Q:0,R:4
                     |
---------------------+
 Ch. B C:1,Q:0,R:0
---------------------+-----------------------

The above diagram illustrates two channels joining in the root channel. The root channel has a capacity of 5, and 4 running jobs coming from Channel A. Channel A has a capacity of 4, all in use (passed down to the root channel), and 12 jobs enqueued. Channel B has a capacity of 1, none in use. This means that whenever a new job comes in channel B, there will be available room for it to run in the root channel.

Note that from the point of view of a channel, ‘running’ means enqueued in the downstream channel. Only jobs marked running in the root channel are actually sent to Odoo for execution.

Should a downstream channel have less capacity than its upstream channels, jobs going downstream will be enqueued in the downstream channel, and compete normally according to their properties (priority, etc).

Using this technique, it is possible to enforce sequence in a channel with a capacity of 1. It is also possible to dedicate a channel with a limited capacity for application-autocreated subchannels without risking to overflow the system.

How to configure Channels?

The ODOO_QUEUE_JOB_CHANNELS environment variable must be set before starting Odoo in order to enable the job runner and configure the capacity of the channels.

The general syntax is channel(.subchannel)*(:capacity(:key(=value)?)*)?,....

Intermediate subchannels which are not configured explicitly are autocreated with an unlimited capacity (except the root channel which if not configured gets a default capacity of 1).

A delay in seconds between jobs can be set at the channel level with the throttle key.

Example ODOO_QUEUE_JOB_CHANNELS:

  • root:4: allow up to 4 concurrent jobs in the root channel.
  • root:4,root.sub:2: allow up to 4 concurrent jobs in the root channel and up to 2 concurrent jobs in the channel named root.sub.
  • sub:2: the same.
  • root:4:throttle=2: wait at least 2 seconds before starting the next job