I wrote a post about how to run celery as daemon and it has many steps to do, many files to copy to your server, configure permissions, and configure upstart, but after sometime I found another and easier way to daemonize Celery.
In this method we will use supervisor to daemonize Celery, and this can be achieved by 3 simple steps, so let's do it.
sudo apt-get install supervisor on Ubuntu server. For other OSs please visit the official documentation
This is the most important step, and in this step we will create supervisor configuration file to daemonize Celery.
Supervisor Configuration File
Celery project provide the basic configuration file in their Github repository.
Supervisor configuration files live in
Celery official configuration file
; ================================== ; celery worker supervisor example ; ================================== [program:celery] ; Set full path to celery program if using virtualenv command=celery worker -A proj --loglevel=INFO directory=/path/to/project user=nobody numprocs=1 stdout_logfile=/var/log/celery/worker.log stderr_logfile=/var/log/celery/worker.log autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. ; Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it ; send SIGKILL to its whole process group instead, ; taking care of its children as well. killasgroup=true ; Set Celery priority higher than default (999) ; so, if rabbitmq is supervised, it will start first. priority=1000
Sample Configurtaion File
This file is a sample of supervisor configuration file used to deamonize Celery and Celery Beat.
[program:django-celery] command=/usr/local/bin/celery worker --app=django_projects --loglevel=INFO --autoscale=10,3 directory=/home/user/django_project user=celery numprocs=1 stdout_logfile=/home/user/django_project/log/celery.log stderr_logfile=/home/user/django_project/log/celery.log autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. ; Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it ; send SIGKILL to its whole process group instead, ; taking care of its children as well. killasgroup=true ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=998 environment=DJANGO_SETTINGS_MODULE='django_project.settings.production' [program:django-celerybeat] ; Set full path to celery program if using virtualenv command=/usr/local/bin/celery beat --app=django_project --loglevel=INFO --schedule=celerybeat-schedule.db ; remove the -A myapp argument if you are not using an app instance directory=/home/user/django_project user=celery numprocs=1 stdout_logfile=/home/user/django_project/log/beat.log stderr_logfile=/home/user/django_project/log/beat.log autostart=true autorestart=true startsecs=10 ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=999 environment=DJANGO_SETTINGS_MODULE='django_project.settings.production'
Let's exmain what in this configuration file:
[program:django-celery]this is the service name, and we will use this to stop, start, restart, or check its status.
commandthis is the command needed to run celery, like the command we use to run celery during the development, you can point the command to use system-wide Celery or virtualenv Celery, just point it to Celery path.
directorythe directory path where Your Celery app lives.
stdout_logfilethe path to log file.
stderr_logfilethe path to error log file, and can be the same path as the log file.
userusername who will run the process.
environmentany environment variable will be use by Celery, for example Celery uses
DJANGO_SETTINGS_MODULEenvironment variable to run with Django app.
--autoscale=10,3this will start Celery with 3 workers and auto scale itself till 10 workers.
Last step is to tell supervisor to read our new configuration file, and update the processes it is running.
$ sudo supervisorctl reread django-celery: avaliable django-celerybeat: avaliable $ sudo supervisorctl update django-celery: added process group django-celerybeat: added process group
🎊🎉 Congratulations now Celery and Celery Beat daemonized on your server.
Managing Celery proccess
Now we have Celery deamonized on our server, let's see how to manage it from Terminal.
$ sudo supervisorctl start django-celery django-celery: started $ sudo supervisorctl start django-celerybeat django-celerybeat: started
$ sudo supervisorctl stop django-celery django-celery: stopped $ sudo supervisorctl stop django-celerybeat django-celerybeat: stopped
$ sudo supervisorctl restart django-celery django-celery: stopped django-celery: started $ sudo supervisorctl restart django-celerybeat django-celerybeat: stopped django-celerybeat: started
$ sudo supervisorctl status django-celery django-celery RUNNING pid 16020, uptime 0:01:00 $ sudo supervisorctl status django-celerybeat django-celerybeat RUNNING pid 16030, uptime 0:01:10