Using Redis Sentinel as your Django Celery broker

In this post we’ll look at using Redis sentinel as your broker when using celery workers and celery beat with django.

Redis Sentinel is a high availability (HA) solution for open source Redis server. In a multi-node setup, Sentinel provides monitoring of nodes and automatic fail over to keep Redis operational without human intervention.

Celery supports using Sentinel as transport broker, but getting the Django settings correct for use with Celery (workers and beat) can take some finesse.

The example we’ll be using is based on a three-node setup with Redis 6.0.5 running on a CentOS 7 systems where the redis and redis-sentinel process are running on all three nodes. Each node uses the default 6379 for redis and the default 26379 port for sentinel. Each node also has the same password set in the redis.conf file, and no password is configured in the redis-sentinel.conf file.

For the django portion we need a number of python packages, all available via pypi. Mileage may vary with different versions and setup. Here’s what is included in our python requirements:

django==2.2.17
django-celery==3.3.1
django-celery-beat==2.0.0
django-redis==4.10.0
django-sentinel==0.1.0
kombu==4.6.7
redis==3.5.3

Celery configuration in the django settings file

All the configuration we need for celery, for example when using worker and/or beat, is set in the django settings file. Note that by default we use the BROKER_URL variable, not CELERY_BROKER_URL. The same is true with the broker transport options; we use BROKER_TRANSPORT_OPTIONS, not CELERY_BROKER_TRANSPORT_OPTIONS. You can actually add the CELERY_ prefix in your django settings by adding the following:

app.config_from_object('django.conf:settings', namespace='CELERY')

See the official celery documentation here for more details regarding celery/django settings: https://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html#latentcall-django-admonition

There are a few other things to note. The BROKER_TRANSPORT_OPTIONS and CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS is where you set the name of the primary being monitored. In the redis-sentinel.conf this the value set in the ‘sentinel monitor’ line. It should look something like this:

sentinel monitor my_primary 127.0.0.1 6379 2

The next thing of note is the BROKER_URL and CELERY_RESULT_BACKEND values. Pay close attention to this syntax. These values need to include all sentinel nodes, the redis password, the sentinel port, and the redis database being used. The password used is the redis password (as set in the redis.conf) NOT the redis-sentinel password (as would be set in redis-sentinel.conf). Also note the colon set before each password. Here’s an example:

sentinel://:my_redis_password@host1:26379/2;sentinel://:my_redis_password@host2:26379/2;sentinel://:my_redis_password@host3:26379/2

Here’s what the complete celery section of our django settings.py file will look like:

# celery settings
BROKER_URL = 'sentinel://:my_redis_password@host1:26379/2;sentinel://:my_redis_password@host2:26379/2;sentinel://:my_redis_password@host3:26379/2'
BROKER_TRANSPORT_OPTIONS = {'master_name': 'my_primary'}
CELERY_RESULT_BACKEND = 'sentinel://:my_redis_password@host1:26379/2;sentinel://:my_redis_password@host2:26379/2;sentinel://:my_redis_password@host3:26379/2'
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {'master_name': 'my_primary'}
CELERYBEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_RESULT_EXPIRES = 60 * 60
BROKER_CONNECTION_TIMEOUT = 30
CELERY_EVENT_QUEUE_EXPIRES = 60
CELERYD_POOL_RESTARTS = True
BROKER_HEARTBEAT = None
BROKER_POOL_LIMIT = 0

A celery worker can be started with the following:

celery worker -A my_django_project

An instance of celery beat can be started with the following:

celery beat -A my_django_project

Hopefully this helps get you going with django, celery, and redis sentinel.

Links and References

https://redis.io/topics/sentinel
https://redis.io/topics/sentinel-clients
https://docs.celeryproject.org/en/stable/getting-started/brokers/redis.html
https://docs.celeryproject.org/en/latest/userguide/configuration.html
https://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html#latentcall-django-admonition