Skip to content

Support Google PubSub Broker (added in Celery 5.5) #1435

@PamelaM

Description

@PamelaM

celery-flower.log
celery.log

Is your feature request related to a problem? Please describe.
We are implementing Celery to provide our product with task queues and cron-like facility. Our Product currently runs within a Kubernetes cluster on GCP, so our preferred broker would be "Google PubSub", which was added as part of Celery 5.5.

However... I could not get Flower to access Celery events (i.e. everything that's actually, you know, useful :) ) within our local development's docker compose configuration.

Bad Behavior encountered:

  • Bringing up the Flower web page did show the actual Celery Worker (I renamed it from celery@celery to worker@celery just to make sure Flower saw the change and it did)
  • But clicking on that worker's link just returned "Unknown worker worker@celery". The compose logs showed a "GET" request returning a 404 error.
  • And the "Broker" tab showed something along the lines of "pubsub not supported"

Modified the local dev compose to use AMQP/RabbitMQ, and voilà! Everything works!

Details and logs are below or attached.

Describe the solution you'd like
I'd like Flower to support GCP, at least for Celery events, etc. It would be great if you could support a GCP PubSub broker UX, too, but we'd be mainly using our own Google Console/Dashboards to monitor that.


PubSub Emulator service config in compose-dev.yml :

gc-pubsub:
      image: google/cloud-sdk:emulators
      hostname: gcpubsub
      ports:
        - "8085:8085"
      command: [
        "gcloud", "beta", "emulators", "pubsub", "start",
        "--project=measurement-enablement-dev",
        "--host-port=0.0.0.0:8085"
      ]
      stop_grace_period: 2s # -- SIGKILL after 2s

Celery Environment Variables :

CELERY_BROKER_URL=gcpubsub://projects/measurement-enablement-dev
PUBSUB_EMULATOR_HOST=gcpubsub:8085
PUBSUB_PROJECT_ID=measurement-enablement-dev

Flower service config in compose.yml :

  celery-flower:
    build:
      context: .
      dockerfile: Dockerfile
      target: server_local
    user: luser
    command: [
        "celery",
        "--config=celeryconfig",
        "-A", "celery_app",
        "flower",
        "--port=5555",
        "–inspect-timeout=10000",
       "--debug",
        "--loglevel=INFO"
    ]

I ran both celery worker and celery flower services with --loglevel=DEBUG, and I will do so again and attach the logs. I noticed that the subscription names didn't quite match between the two services.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions