0%

解决分布式系统Celery beat会产生多份相同任务的问题

主要使用工具single-beat, 来实现多个servers上只有一个celery beat在运行

安装

redis

自己搭建, 很简单的哦

single-beat

so easy 😄

1
pip install single-beat

Usage

创建简单的测试脚本(/tmp/start.sh)

1
2
3
4
5
6
7
#!/bin/bash

while true;
do
echo `date`;
sleep 2;
done
1
chmod +x /tmp/start.sh

Server 1

1
SINGLE_BEAT_REDIS_SERVER='redis://redis-ip:6379' single-beat /tmp/start.sh

Server 2

1
SINGLE_BEAT_REDIS_SERVER='redis://redis-ip:6379' single-beat /tmp/start.sh

supervisord 配置启动

1
2
3
4
5
6
7
8
9
10
11
12
[program:single-beat-test]
environment=PYTHONUNBUFFERED=1,SINGLE_BEAT_REDIS_SERVER='redis://127.0.0.1:6379',SINGLE_BEAT_IDENTIFIER='test'
directory=/tmp
command=single-beat /tmp/start.sh
autostart=true
autorestart=unexpected
exitcodes=0,2
user=ymmops
redirect_stderr=true
stdout_logfile=/tmp/single-beat-test.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

grinning all the time

😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄 😄