Calling Python Celery Tasks from a Different Machine Using send_task

Prerequisites

To follow along, you will need:

  • Python installed on both the client and worker machines.
  • Celery and a message broker (RabbitMQ) installed. Redis will be used as the result backend.
  • Basic knowledge of Python and familiarity with Celery.

Step 1: Setup the Worker

First, let’s set up the Celery worker. On the worker machine, create a file named tasks.py:

from celery import Celery

app = Celery("tasks", broker='amqp://username:password@localhost',
             backend='redis://localhost:6379/0')

@app.task(name='celery_project.tasks.add')
def add(x, y):
    return x + y

Here, we define a simple task named add that takes two arguments and returns their sum. Adjust the broker and backend URLs to point to your actual RabbitMQ and Redis services.

Step 2: Start the Celery Worker

Run the following command on the worker machine to start the Celery worker:

.venv\Scripts\python.exe -m celery -A tasks worker --loglevel=info -E --pool=solo

This command starts a Celery worker that listens for tasks to execute.

Step 3: Setup the Client

On the client machine, you don’t need the full task definitions—only the Celery app configuration and the task signatures. Create a file named main.py:

from celery import Celery

app = Celery("tasks", broker='amqp://username:password@localhost',
             backend='redis://localhost:6379/0')

result = app.send_task('celery_project.tasks.add', args=[4, 4])
print(result.get())

Here, send_task is used to dispatch the task. It requires the name of the task (which must match the name given in the worker’s task decorator) and the arguments for the task.

Step 4: Calling the Task from the Client

Run the main.py script on the client machine:

python main.py

This script sends the add task to the worker machine via the message broker, and then fetches the result using result.get().

Or Use Minimal Task Definitions approach

On the client side, you only need a minimal definition of the tasks to send them. You can redefine the tasks in a simple module that just includes the task names, without their implementations:

client_tasks.py:

from celery import Celery

app = Celery('client_tasks', broker='pyamqp://guest@your_broker_ip//')

@app.task(name='your_module_name.tasks.add')
def add(x, y):
    pass  # Implementation is not needed on the client

Then on the client:

from client_tasks import add
result = add.delay(4, 4)
print(result.get(timeout=10))