-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem? Please describe.
We are implementing Celery to provide our product with task queues and cron-like facility. Our Product currently runs within a Kubernetes cluster on GCP, so our preferred broker would be "Google PubSub", which was added as part of Celery 5.5.
However... I could not get Flower to access Celery events (i.e. everything that's actually, you know, useful :) ) within our local development's docker compose configuration.
Bad Behavior encountered:
- Bringing up the Flower web page did show the actual Celery Worker (I renamed it from
celery@celery
toworker@celery
just to make sure Flower saw the change and it did) - But clicking on that worker's link just returned "Unknown worker worker@celery". The compose logs showed a "GET" request returning a 404 error.
- And the "Broker" tab showed something along the lines of "pubsub not supported"
Modified the local dev compose to use AMQP/RabbitMQ, and voilà! Everything works!
Details and logs are below or attached.
Describe the solution you'd like
I'd like Flower to support GCP, at least for Celery events, etc. It would be great if you could support a GCP PubSub broker UX, too, but we'd be mainly using our own Google Console/Dashboards to monitor that.
PubSub Emulator service config in compose-dev.yml
:
gc-pubsub:
image: google/cloud-sdk:emulators
hostname: gcpubsub
ports:
- "8085:8085"
command: [
"gcloud", "beta", "emulators", "pubsub", "start",
"--project=measurement-enablement-dev",
"--host-port=0.0.0.0:8085"
]
stop_grace_period: 2s # -- SIGKILL after 2s
Celery Environment Variables :
CELERY_BROKER_URL=gcpubsub://projects/measurement-enablement-dev
PUBSUB_EMULATOR_HOST=gcpubsub:8085
PUBSUB_PROJECT_ID=measurement-enablement-dev
Flower service config in compose.yml
:
celery-flower:
build:
context: .
dockerfile: Dockerfile
target: server_local
user: luser
command: [
"celery",
"--config=celeryconfig",
"-A", "celery_app",
"flower",
"--port=5555",
"–inspect-timeout=10000",
"--debug",
"--loglevel=INFO"
]
I ran both celery worker
and celery flower
services with --loglevel=DEBUG
, and I will do so again and attach the logs. I noticed that the subscription names didn't quite match between the two services.