DEV Community

Cover image for Celery Tasks Running Twice? How to Prevent Duplicate Execution"
sizan mahmud0
sizan mahmud0

Posted on

Celery Tasks Running Twice? How to Prevent Duplicate Execution"

Celery doesn't promise exactly-once execution. In many setups, it's at-least-once, so your tasks need to survive running twice. Here's a practical breakdown.

Why a task runs twice

  • acks_late=True + worker crash: the broker redelivers the task, possibly after partial work.
  • Redis visibility timeout: unacknowledged messages become visible again (1 hour by default). Long tasks or long countdown/eta values can be redelivered while still running.
  • Retries: a task that fails after a side effect repeats that side effect.
  • Double submissions: double-clicks, duplicate webhooks, client retries.
  • Celery Beat: overlapping runs, or more than one Beat process.
  • Queuing inside a transaction: the worker can start before the data is committed.

1. Make tasks idempotent

# Not idempotent
@shared_task
def give_bonus(user_id):
    Wallet.objects.filter(user_id=user_id).update(balance=F("balance") + 10)

# Idempotent
@shared_task
def recalculate_total(order_id):
    order = Order.objects.get(pk=order_id)
    order.total = calculate_total(order)
    order.save(update_fields=["total"])
Enter fullscreen mode Exit fullscreen mode

Compute from source data, prefer update_or_create(), and check state before acting.

2. Use a unique constraint as a safety net

Check-then-act fails when two copies run at the same moment. Let the database decide:

from django.db import IntegrityError, transaction

@shared_task
def create_payout(order_id):
    order = Order.objects.get(pk=order_id)
    try:
        with transaction.atomic():
            Payout.objects.create(order=order, amount=order.total)
    except IntegrityError:
        return  # duplicate run
Enter fullscreen mode Exit fullscreen mode

For concurrent updates to one row, use select_for_update() inside transaction.atomic().

3. Use a lock for non-overlapping work

@shared_task(bind=True)
def generate_invoice(self, order_id):
    lock = redis_client.lock(f"lock:invoice:{order_id}", timeout=300)
    if not lock.acquire(blocking=False):
        return "skipped: already running"
    try:
        build_invoice(order_id)
    finally:
        try:
            lock.release()
        except redis.exceptions.LockError:
            pass
Enter fullscreen mode Exit fullscreen mode

Rules of thumb:

  • Always set a timeout, or a crashed worker blocks the task forever
  • Make the timeout longer than the task's normal runtime
  • A lock is not idempotency: a duplicate arriving after the lock is released still runs
  • cache.add() works as a simple lock only on shared backends like Redis, not local-memory cache

4. Idempotency keys for webhooks

class ProcessedEvent(models.Model):
    event_id = models.CharField(max_length=200, unique=True)
    processed_at = models.DateTimeField(auto_now_add=True)

@shared_task
def handle_payment_webhook(event_id, payload):
    try:
        with transaction.atomic():
            ProcessedEvent.objects.create(event_id=event_id)
            apply_payment(payload)
    except IntegrityError:
        return  # already handled
Enter fullscreen mode Exit fullscreen mode

The key and the business change share one transaction, so a failed run leaves no key behind and a retry can still succeed.

5. Queue after commit

def place_order(request):
    with transaction.atomic():
        order = Order.objects.create(...)
        transaction.on_commit(lambda: process_order.delay(order.id))
Enter fullscreen mode Exit fullscreen mode

Pass IDs, not model objects.

6. Check your config

CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 43200}  # 12 hours
Enter fullscreen mode Exit fullscreen mode

Set visibility_timeout higher than your longest task and longest countdown. For Beat, run exactly one instance and lock long periodic tasks.

Which fix when?

Situation Approach
Any retryable task Idempotent design
Create a record once Unique constraint + IntegrityError
Concurrent updates select_for_update()
Expensive, must not overlap Redis lock with timeout
Webhooks, API retries Idempotency key in the database
Periodic tasks Lock + single Beat instance

Full guide

The complete post has a pre-ship checklist and an FAQ:

👉 https://blog.reiteachaigean.com/blog/celery-prevent-duplicate-task-execution

More on Celery and large datasets:

How do you handle duplicate tasks in production?

Top comments (0)