Drupal 整合 Gearman实战

By Richard Yu (理查)

@Chegg China

Senior Developer

声明

• 是分享，不是授课
• 所说的不一定全对，个人见解
• Gearman不一定最好，方案众多
• Gearman很深，不要轻易下结论
• Gearman也很简单，不妨试试

Gearman 工作原理

Gearman 工作原理

• Client: 客户端，发出任务处理请求
• Job Server: Job服务器，接收来自客户端的任务处理请求,然后以负载均衡的方式分发给注册的worker.
• Worker: 工人，处理Job服务器分发过来的任务处理请求

Gearman 工作原理

<?php
// Reverse Client Code
$client = new GearmanClient();
$client->addServer();
print $client->doBackground("reverse", "Hello World!");

Client

Worker

<?php
// Reverse Worker Code
$worker = new GearmanWorker();
$worker->addServer();
$worker->addFunction("reverse", function ($job) {
  return strrev($job->workload());
});
while ($worker->work());

并发任务

$client->addTask('lookup_user', 'joe@joe.com', 'lookup_user');
$client->addTask('baconate', 'joe@joe.com', 'baconate');
$client->addTask('get_latest_posts_by', 'joe@joe.com', 'get_latest_posts_by');

$client->runTasks();

Gearman 特点

• 开源
• 多语言驱动
• 灵活，没有任何模式限制
• 快，基于C/C++
• 可以很容易嵌入现有系统
• 无单点故障
• 不限制消息大小
• 适合大型网站架构

Gearman 使用场景

• 格式转化，视频转码
• 缩略图生成
• 缓存更新，静态化
• 发邮件
• 分布式日志查询
• 搜索引擎
• Map/Reduce

使用 Gearman 的注意事项

• 多Worker进程实现负载均衡
• Job持久化
• Worker进程守护
• 逻辑更改，需要重启Worker

Worker守护进程

• nohup + &
• screen
• Gearman monitor
• Shell daemon
• PHP daemon
• Supervisord

Gearman最简安装

yum install epel-release
yum install gearman

pecl install gearman

配置

vim /etc/sysconfig/gearmand

OPTIONS=" \
-q mysql \
--mysql-host=localhost \
--mysql-user=root \
--mysql-password=root \
--mysql-db=gearman \
--mysql-table=gearman_queue \
--mysql-port=3306 \

"

gearmand -d
/etc/init.d/gearman start

监控

gearadmin --workers
gearadmin --status
gearadmin --show-jobs

(echo status; sleep 0.1) | nc 127.0.0.1 4730

启动

测试

$ gearman -w -f wc -- wc -l

$ gearman -f wc < /etc/passwd
26

$ wc -l < /etc/passwd

Supervisord

Supervisord是用Python实现的一款非常实用的进程管理工具。supervisord会帮你把管理的应用程序转成daemon程序，而且可以方便的通过命令开启、关闭、重启等操作，而且它管理的进程一旦崩溃会自动重启，这样就可以保证程序执行中断后的情况下有自我修复的功能。

Supervisord 简介

安装

## Mac
$ sudo easy_install pip
$ sudo pip install supervisor

## CentOS
$ sudo yum install supervisor

$ echo_supervisord_conf > /etc/supervisord.conf

logfile=/tmp/supervisord.log

[inet_http_server] 
port=127.0.0.1:9001 
username=user 
password=123

[program:zinch_gearman_background_process]
command = /Applications/MAMP/bin/php/php5.3.28/bin/drush -r /Users/zhicheng/Sites/zinch/html gearman-worker background_process
process_name=%(program_name)s_%(process_num)02d
autostart = true
autorestart = true
numprocs = 3
stdout_logfile=/tmp/background_process.log
user = zhicheng
redirect_stderr = true

配置

tail -f /tmp/supervisord.log
2015-05-25 13:49:34,389 INFO RPC interface 'supervisor' initialized
2015-05-25 13:49:34,390 INFO RPC interface 'supervisor' initialized
2015-05-25 13:49:34,390 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-05-25 13:49:34,391 INFO supervisord started with pid 89964
2015-05-25 13:49:34,631 INFO spawned: 'zinch_gearman_user_register_02' with pid 1531
2015-05-25 13:49:34,634 INFO spawned: 'zinch_gearman_user_register_01' with pid 1532
2015-05-25 13:49:34,638 INFO spawned: 'zinch_gearman_user_register_00' with pid 1533
2015-05-25 13:49:35,639 INFO success: zinch_gearman_user_register_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-05-25 13:49:35,640 INFO success: zinch_gearman_user_register_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-05-25 13:49:35,640 INFO success: zinch_gearman_user_register_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Supervisord日志

启动

supervisord -c /etc/supervisord.conf

$ supervisorctl -h

Documented commands (type help <topic>):
========================================
EOF    exit  maintail  quit    restart   start   stop
clear  help  open      reload  shutdown  status  tail

控制

Worker 日志

tail -f /tmp/background_process.log
Gearman version: 1.1.12
function background_process added!
Gearman version: 1.1.12
function background_process added!
Gearman version: 1.1.12
function background_process added!

Supervisord Web 控制台

Gearman的可靠性思考

持久化能解决可靠性问题么？？？

解决方案

• 自己维护 job 列表和 job 的状态
• 即使失败了还可以重新启动
• 与Worker的持久化并存，互不干涉

CREATE TABLE `background_process_log` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `callback` varchar(255) DEFAULT NULL,
  `params` longblob,
  `cid` varchar(255) DEFAULT NULL,
  `uid` int(10) unsigned DEFAULT NULL,
  `timestamp` int(10) unsigned DEFAULT NULL,
  `processed` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8

数据结构

<?php
function drupal_background_process($callback_params) {
  
  // Record to background_process_log
  // If enabled, Using GearmanClient add job to job server
  // If not enabled, Using original way to process the job

  return true;
}

Add job to job server

<?php

$background_params = array(
    'callback' => 'registration_new_user_profile_makeup',
    'params'   => array($user, $values, request_uri())
      );

drupal_background_process($background_params);

drush gearman-worker background_process

Drush 启动 Worker

Worker 定义

function drush_process_zinch_gearman_worker($workerKey = NULL) {
  $workers = _gearman_workers_info();

  $worker= new GearmanWorker();

  try {

    $worker->addServer(GEARMAN_SERVER_HOST, GEARMAN_SERVER_PORT);
  } catch (Exception $e) {
    drush_die($e->getMessage());
  }

  if (isset($workers[$workerKey])) {
    $worker->addFunction($workerKey, $workers[$workerKey]);
    drush_print(strtr('function !function added!', array('!function' => $workerKey)));
    while ($worker->work());
  }
}

drush gearman-client background_process 1

客户端测试

Thank you!~

Questions?

参考资料

http://blog.csdn.net/jiao_fuyou/article/details/16330195