Skip to content

In some cases shutdown middleware is not called. #424

@SergeyCh92

Description

@SergeyCh92

Hi!
I faced the following problem.
I have defined middleware with the shutdown method. The task is launched using the command taskiq worker mypath.tkq:broker --workers --fs-discover --tasks-pattern **/scheduled_tasks.py --workers 1 --shutdown-timeout 40 --wait-tasks-timeout 5 --hardkill-count 1.
The code example.

class ShutdownMiddleware(TaskiqMiddleware):
    async def shutdown(self):
        logger.info("shutdown has been called")


class TaskExecutionLoggerMiddleware(TaskiqMiddleware):
    async def pre_execute(self, message: "TaskiqMessage") -> TaskiqMessage:
        logger.info("some text")
        return message

    async def post_save(
        self,
        message: "TaskiqMessage",
        result: "TaskiqResult[Any]",
    ) -> None:
        logger.info("some text")


broker = ListQueueBroker(
    url="some_url",
).with_result_backend(
    RedisAsyncResultBackend("some_url"),
)

broker.add_middlewares(
    TaskExecutionLoggerMiddleware(),
    ShutdownMiddleware(),
)

taskiq_fastapi.init(broker, "my_path:get_app")

redis_source = RedisScheduleSource("some_url")
label_schedule_source = LabelScheduleSource(broker)
scheduler = TaskiqScheduler(
    broker=broker,
    sources=[label_schedule_source, redis_source],
)

In that case everything works properly on my local computer (Mac OS).
If I remove a hardkill-count parameter the middleware doesn't call when I interrupt a process on my local PC (I use ctrl + c to stop a process). The program just stops.

On the server, the application runs in a pod inside kubernetes using helm-chart. The launch command is the same.
In case when we use a hardkill-count parameter the app doesn't call a shutdown method. Here is an example of logs when the pod restarts.

2025-03-14 18:28:17.511 | [2025-03-14 15:28:17,510][taskiq.process-manager][INFO   ][MainProcess] Started process worker-0 with pid 7 |  
-- | -- | --
  |   | 2025-03-14 18:28:17.484 | [2025-03-14 15:28:17,484][taskiq.worker][INFO   ][MainProcess] Starting 1 worker processes. |  
  |   | 2025-03-14 18:28:17.484 | [2025-03-14 15:28:17,484][taskiq.worker][INFO   ][MainProcess] Pid of a main process: 1 |  
  |   | 2025-03-14 18:08:42.867 | [2025-03-14 15:08:42,867][taskiq.receiver.receiver][INFO   ][worker-0] The runner is stopped. |  
  |   | 2025-03-14 18:08:42.867 | [2025-03-14 15:08:42,867][taskiq.receiver.receiver][INFO   ][worker-0] Stoping prefetching messages... |  
  |   | 2025-03-14 18:08:41.919 | [2025-03-14 15:08:41,919][taskiq.process-manager][WARNING][MainProcess] Workers are scheduled for shutdown.

As you may see there aren't logs inside of shutdown method.
In that case when I start the pod on the server without a hardkill-count parameter middleware doesn't call as well.
Generally, the problem is the shutdown doesn't work on the server although it works locally.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions