-
-
Notifications
You must be signed in to change notification settings - Fork 89
Description
Hi!
I faced the following problem.
I have defined middleware with the shutdown method. The task is launched using the command taskiq worker mypath.tkq:broker --workers --fs-discover --tasks-pattern **/scheduled_tasks.py --workers 1 --shutdown-timeout 40 --wait-tasks-timeout 5 --hardkill-count 1.
The code example.
class ShutdownMiddleware(TaskiqMiddleware):
async def shutdown(self):
logger.info("shutdown has been called")
class TaskExecutionLoggerMiddleware(TaskiqMiddleware):
async def pre_execute(self, message: "TaskiqMessage") -> TaskiqMessage:
logger.info("some text")
return message
async def post_save(
self,
message: "TaskiqMessage",
result: "TaskiqResult[Any]",
) -> None:
logger.info("some text")
broker = ListQueueBroker(
url="some_url",
).with_result_backend(
RedisAsyncResultBackend("some_url"),
)
broker.add_middlewares(
TaskExecutionLoggerMiddleware(),
ShutdownMiddleware(),
)
taskiq_fastapi.init(broker, "my_path:get_app")
redis_source = RedisScheduleSource("some_url")
label_schedule_source = LabelScheduleSource(broker)
scheduler = TaskiqScheduler(
broker=broker,
sources=[label_schedule_source, redis_source],
)
In that case everything works properly on my local computer (Mac OS).
If I remove a hardkill-count parameter the middleware doesn't call when I interrupt a process on my local PC (I use ctrl + c to stop a process). The program just stops.
On the server, the application runs in a pod inside kubernetes using helm-chart. The launch command is the same.
In case when we use a hardkill-count parameter the app doesn't call a shutdown method. Here is an example of logs when the pod restarts.
2025-03-14 18:28:17.511 | [2025-03-14 15:28:17,510][taskiq.process-manager][INFO ][MainProcess] Started process worker-0 with pid 7 |
-- | -- | --
| | 2025-03-14 18:28:17.484 | [2025-03-14 15:28:17,484][taskiq.worker][INFO ][MainProcess] Starting 1 worker processes. |
| | 2025-03-14 18:28:17.484 | [2025-03-14 15:28:17,484][taskiq.worker][INFO ][MainProcess] Pid of a main process: 1 |
| | 2025-03-14 18:08:42.867 | [2025-03-14 15:08:42,867][taskiq.receiver.receiver][INFO ][worker-0] The runner is stopped. |
| | 2025-03-14 18:08:42.867 | [2025-03-14 15:08:42,867][taskiq.receiver.receiver][INFO ][worker-0] Stoping prefetching messages... |
| | 2025-03-14 18:08:41.919 | [2025-03-14 15:08:41,919][taskiq.process-manager][WARNING][MainProcess] Workers are scheduled for shutdown.
As you may see there aren't logs inside of shutdown method.
In that case when I start the pod on the server without a hardkill-count parameter middleware doesn't call as well.
Generally, the problem is the shutdown doesn't work on the server although it works locally.