Supervisor使用笔记及celery生产部署
1 supervisor作用
Linux的后台进程运行有好几种方法，例如nohup，screen等，但是，如果是一个服务程序，要可靠地在后台运行，我们就需要把它做成daemon，最好还能监控进程状态，在意外结束时能自动重启。
supervisor就是用Python开发的一套通用的进程管理程序，能将一个普通的命令行进程变为后台daemon，并监控进程状态，异常退出时能自动重启。
Supervisor用于linux系统中进行进程守护，supervisor3.x版本仅支持在python2环境下运行，但对其管理的子进程运行环境没有要求，在即将发布的4.x版本开始支持python3。
2 supervisor安装
使用easy_install安装(需要默认python环境为python2)
Git克隆安装
cd /usr/bin
sudo git clone https://github.com/Supervisor/supervisor.git
cd supervisor
sudo python2 setup.py install
3 supervisor配置
生成需要的文件夹
由配置模板生成配置文件并赋权
sudo chmod -R 777 /etc/supervisor/ # 文件夹赋权
sudo echo_supervisord_conf > /etc/supervisord/supervisord.conf # 生成默认配置模板
sudo chmod -R 777 /etc/supervisord/supervisord.conf # 对默认配置文件赋权
sudo gedit /etc/supervisord/supervisord.conf # 编辑默认配置文件
需要特殊配置的地方：
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
[supervisord]
logfile=/var/log/supervisord/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=info ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=1024 ; min. avail startup file descriptors; default 1024
minprocs=200 ; min. avail process descriptors;default 200
childlogdir=/tmp ; 'AUTO' child log dir, default $TEMP
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
# 此处需要配置 用于保证supervisor停止进程后其对应的UNIX进程被同步关闭
# 不然会出现celery已被关闭但是任务队列仍然可用的情况
stopasgroup=true ; send stop signal to the UNIX process group (default false)
# 此处需要配置 保证/etc/supervisord/conf.d/文件夹下所有以.conf类型结尾的文件都被当做配置文件被加载
[include]
files = /etc/supervisord/conf.d/*.conf
4 cleley配置
示例使用的项目根目录为/django/JRFamily/,使用的virtualenv路径为/django/JRFamily/environment/，日志存储路径为/django/JRFamily/log/。
生成日志文件，supervisor不会自动生成：
新增/etc/supervisord/conf.d/JRFamily.conf文件：
[program:jrfamily_celery_beat]
command=/django/JRFamily/environment/bin/python manage.py celery beat -l info
directory=/django/JRFamily
stdout_logfile=/django/JRFamily/log/celery_beat.log
stderr_logfile=/django/JRFamily/log/celery_beat.log
autostart=true
autorestart=true
user=dreamgo
startsecs=10
[program:jrfamily_celery_worker]
numprocs=2
process_name=worker%(process_num)s
command=/django/JRFamily/environment/bin/python manage.py celery worker -E -l info
directory=/django/JRFamily
stdout_logfile=/django/JRFamily/log/celery_worker.log
stderr_logfile=/django/JRFamily/log/celery_worker.log
autostart=true
autorestart=true
user=dreamgo
startsecs=10
[program:jrfamily_celery_beat]
command=/django/JRFamily/environment/bin/python manage.py celerycam
directory=/django/JRFamily
stdout_logfile=/django/JRFamily/log/celery_beat.log
stderr_logfile=/django/JRFamily/log/celery_beat.log
autostart=true
autorestart=true
user=dreamgo
startsecs=10
autostart为是否自动启动，user为当前系统用户名。
上述脚本依次执行的命令为：
python manage.py celery beat -l info # 开启task发送端
python manage.py celery worker -E -l info # -E是为了celerycam能够捕捉到task状态
python manage.py celerycam # 捕捉task执行状态 可在admin->Djcelery->Task查看
5 RabbitMQ配置
在同一台服务器上部署多个Django站点时，若使用同一个RabbitMQ连接，将导致多个Django站点共用同一个任务队列，使得其中某一个站点只能接受到n/m的任务(n为本站点开启的celery wroker数量,m为总站点开启的celery worker数量)。使用RabbitMQ中的vhost功能来隔绝各个站点之间的RabbitMQ连接，从而确保本站点发出的任务(task)只有本站点开启的worker能够接受到并进行处理。
RabbitMQ中的vhost概念相当于Apache中的VirtualHost。
请按照下列命令顺序添加用户并将用户绑定到对应的vhost上：
sudo rabbitmqctl add_user {username} {password} # 添加用户
sudo rabbitmqctl add_vhost {vhost_name} # 添加vhost
sudo rabbitmqctl set_permissions -p {vhost_name} {username} ".*" ".*" ".*" # 为用户设置方位vhost的权限
sudo rabbitmqctl set_user_tags {username} {tag ...}
多个站点可以共用一个用户账号，但是必须使用不同的vhost。 更多rabbitmqctl使用方法请参见rabbitmqctl使用手册。
在Django中的settings文件设置：
多站点问题解决时查看的相关资料：
Run Multiple Django Apps With Celery On One Server With Rabbitmq VHosts
Using celeryd as a daemon with multiple django apps?
Running multiple Django Celery websites on same server
Run Multiple Django Apps With Celery On One Server With Rabbitmq VHosts
6 supervisor使用
开启、断开supervisor.sock连接
状态查看、控制
supervisorctl -c /etc/supervisord/supervisord.conf
> status # 查看程序状态
> stop [process_alias] # 关闭 usercenter 程序
> start [process_alias] # 启动 usercenter 程序
> restart [process_alias] # 重启 usercenter 程序
> reread ＃ 读取有更新（增加）的配置文件，不会启动新添加的程序
> update ＃ 重启配置文件修改过的程序
sudo supervisorctl -c /etc/supervisord/supervisord.conf stop all
# 关闭全部进程 unlink前必须使用该命令关闭supervisord 不然celery依旧存在于系统中 导致内存泄露以及有规律的任务丢失！！！
sudo supervisorctl -c /etc/supervisord/supervisord.conf start all # 启动所有配置
sudo supervisorctl -c /etc/supervisord/supervisord.conf reread # 读取有更新（增加）的配置文件，不会启动新添加的程序
sudo supervisorctl -c /etc/supervisord/supervisord.conf update # 重启配置文件修改过的程序
Linux进程查看、清理
ps auxf|grep celery # 树形显示 推荐
ps -efH|grep celery # 查看celery相关进程 必须是大写的H以格式化显示格式
ps aux|grep celery | awk '{system("kill -9 " $2)}' # 杀死celery相关进程
7 supervisor开机自启动
ubuntu开启自启动脚本原始链接
修改后脚本/etc/init.d/supervisord：
#! /bin/sh
#
# Downloaded from:
# http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/supervisor/trusty/view/head:/debian/supervisor.init
#
# skeleton example file to build /etc/init.d/ scripts.
# This file should be used to construct scripts for /etc/init.d.
#
# Written by Miquel van Smoorenburg <miquels@cistron.nl>.
# Modified for Debian
# by Ian Murdock <imurdock@gnu.ai.mit.edu>.
# Further changes by Javier Fernandez-Sanguino <jfs@debian.org>
# Modified by sbilly <superli.1980@gmail.com> Added supervisorctl to status
#
# Version: @(#)skeleton 1.9 26-Feb-2001 miquels@cistron.nl
#
### BEGIN INIT INFO
# Provides: supervisor
# Required-Start: $remote_fs $network $named
# Required-Stop: $remote_fs $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start/stop supervisor
# Description: Start/stop supervisor daemon and its configured
# subprocesses.
### END INIT INFO
. /lib/lsb/init-functions
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/supervisord
SUPERVISORCTL=/usr/local/bin/supervisorctl
NAME=supervisord
DESC=supervisor
test -x $DAEMON || exit 0
LOGDIR=/var/log/supervisord
PIDFILE=/tmp/supervisord.pid
DODTIME=5 # Time to wait for the server to die, in seconds
# If this value is set too low you might not
# let some servers to die gracefully and
# 'restart' will not work
# Include supervisor defaults if available
if [ -f /etc/default/supervisor ] ; then
. /etc/default/supervisor
fi
DAEMON_OPTS="-c /etc/supervisord/supervisord.conf $DAEMON_OPTS"
set -e
running_pid()
{
# Check if a given process pid's cmdline matches a given name
pid=$1
name=$2
[ -z "$pid" ] && return 1
[ ! -d /proc/$pid ] && return 1
(cat /proc/$pid/cmdline | tr "\000" "\n"|grep -q $name) || return 1
return 0
}
running()
{
# Check if the process is running looking at /proc
# (works for all users)
# No pidfile, probably no daemon present
[ ! -f "$PIDFILE" ] && return 1
# Obtain the pid and check it against the binary name
pid=`cat $PIDFILE`
running_pid $pid $DAEMON || return 1
return 0
}
force_stop() {
# Forcefully kill the process
[ ! -f "$PIDFILE" ] && return
if running ; then
kill -15 $pid
# Is it really dead?
[ -n "$DODTIME" ] && sleep "$DODTIME"s
if running ; then
kill -9 $pid
[ -n "$DODTIME" ] && sleep "$DODTIME"s
if running ; then
echo "Cannot kill $NAME (pid=$pid)!"
exit 1
fi
fi
fi
rm -f $PIDFILE
return 0
}
case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon --start --quiet --pidfile $PIDFILE \
--startas $DAEMON -- $DAEMON_OPTS
test -f $PIDFILE || sleep 1
if running ; then
echo "$NAME."
else
echo " ERROR."
fi
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
echo "$NAME."
;;
force-stop)
echo -n "Forcefully stopping $DESC: "
force_stop
if ! running ; then
echo "$NAME."
else
echo " ERROR."
fi
;;
#reload)
#
# If the daemon can reload its config files on the fly
# for example by sending it SIGHUP, do it here.
#
# If the daemon responds to changes in its config file
# directly anyway, make this a do-nothing entry.
#
# echo "Reloading $DESC configuration files."
# start-stop-daemon --stop --signal 1 --quiet --pidfile \
# /var/run/$NAME.pid --exec $DAEMON
#;;
force-reload)
#
# If the "reload" option is implemented, move the "force-reload"
# option to the "reload" entry above. If not, "force-reload" is
# just the same as "restart" except that it does nothing if the
# daemon isn't already running.
# check wether $DAEMON is running. If so, restart
start-stop-daemon --stop --test --quiet --pidfile $PIDFILE \
--startas $DAEMON \
&& $0 restart \
|| exit 0
;;
restart)
echo -n "Restarting $DESC: "
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
[ -n "$DODTIME" ] && sleep $DODTIME
start-stop-daemon --start --quiet --pidfile $PIDFILE \
--startas $DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
status)
echo -n "$NAME is "
if running ; then
echo "running"
else
echo " not running."
exit 1
fi
$SUPERVISORCTL $DAEMON_OPTS status
;;
*)
N=/etc/init.d/$NAME
# echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $N {start|stop|restart|force-reload|status|force-stop}" >&2
exit 1
;;
esac
exit 0
自启动脚本修改注意事项：
- 需要修改DAEMON、SUPERVISORCTL两个路径，均位于/usr/local/bin目录下，原始目录为/usr/bin。
- 务必确保PIDFILE的位置和/etc/supervisord/supervisord.conf中的pidfile对应相同的文件。
- 确保LOGDIR设置的目录和/etc/supervisord/supervisord.conf中的logfile所在的目录相同。
设置脚本为Ubuntu中的服务
sudo chmod -R 777 /etc/init.d/supervisord
sudo update-rc.d supervisord defaults
sudo service --status-all
移除服务