Gearman Distribute Computing Framework

Introduction:
Today many web services will need to to do some complex work which can cost some computations. As a result, we need some kind of job dispatcher to help us to dispatch jobs. Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

Here are some features about Gearman:

• load balance
• Client API: C, PHP, Python, Perl
• no broadcast feature
• auto dispatch task
• persistent queue (sqlite, mysql)
• client
  • blocking task
  • non-blocking (concurrent multi-task), callback function when complete or fail!
• worker
  • blocking task handle

Under the following picture, you can see how Gearman works. Your application will use Gearman Client API to send task to Gearman Job Server. The Job Server will find which Worker had low workload, and send the task to the Worker through Gearman Worker API. All communication is through Gearman API, you don’t need to consider any TCP connection. And Client Node, Job Server, and Worker Node can be in different machine.

Job Server fail over:
The Worker and Client need to connect to all Job Server (it’s ok to set 2 or 3 Job Servers). Then if one Job Server fail over, there is still one Job Server remain. The Client can still work.

Cons:
No monitor tool, the cluster monitor tool is still under development.
Reference:
http://gearman.org/
http://blog.flier.lu/2010/04/gearman-performance-tunning/