`ObjectDisposedException` persists

[lukebakken]

Describe the bug

Reported here: #1749 (comment)

Object name: 'System.Threading.SemaphoreSlim'.', classId=0, methodId=0, exception=System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Threading.SemaphoreSlim'.
   at System.Threading.SemaphoreSlim.WaitAsync(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.MaybeHandlePublisherConfirmationTcsOnChannelShutdownAsync(ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.Channel.OnChannelShutdownAsync(ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.Channel.OnSessionShutdownAsync(Object sender, ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.Channel.FinishCloseAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.HandleChannelCloseOkAsync(IncomingCommand cmd, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.HandleCommandAsync(IncomingCommand cmd, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.ProcessFrameAsync(InboundFrame frame, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.ReceiveLoopAsync(CancellationToken mainLoopCancellationToken)
   at RabbitMQ.Client.Framing.Connection.MainLoop()
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Threading.SemaphoreSlim'.
   at System.Threading.SemaphoreSlim.WaitAsync(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.MaybeHandlePublisherConfirmationTcsOnChannelShutdownAsync(ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.Channel.OnChannelShutdownAsync(ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.Channel.OnSessionShutdownAsync(Object sender, ShutdownEventArgs reason)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.AsyncEventingWrapper`1.InternalInvoke(Delegate[] handlers, Object sender, TEvent event)
   at RabbitMQ.Client.Impl.Channel.FinishCloseAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.HandleChannelCloseOkAsync(IncomingCommand cmd, CancellationToken cancellationToken)
   at RabbitMQ.Client.Impl.Channel.HandleCommandAsync(IncomingCommand cmd, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.ProcessFrameAsync(InboundFrame frame, CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.ReceiveLoopAsync(CancellationToken mainLoopCancellationToken)
   at RabbitMQ.Client.Framing.Connection.MainLoop()
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.Impl.Channel.QueueBindAsync(String queue, String exchange, String routingKey, IDictionary`2 arguments, Boolean noWait, CancellationToken cancellationToken)

Reproduction steps

@ZajacPiotr98 reports: "It happens when I create and remove a lot of queues as I said. Firstly I have operation cancelled exception from the QueueDeclare and then problem occurs."

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at RabbitMQ.Client.Impl.Channel.QueueDeclareAsync(String queue, Boolean durable, Boolean exclusive, Boolean autoDelete, IDictionary`2 arguments, Boolean passive, Boolean noWait, CancellationToken cancellationToken)

[lukebakken]

@ZajacPiotr98 does your code try to concurrently declare a queue on the same channel as the channel is being closed?

Again, it would really, really, really be helpful if you could share the code that triggers this issue. It must be unique to your use of this library or I think I would be hearing more about it. Thanks!

[ZajacPiotr98]

Unfortunately, I can't share the code. Channels are never used concurrently from multiple threads and also there's always check for flag IsOpen before channel is used. I think it's the same or similar case as here #1749 (comment), but in this case it's initiated by client, not server

[lukebakken]

@ZajacPiotr98 can you determine where / how the TaskCanceledException originates?

[lukebakken]

@ZajacPiotr98 another question, in your code do you always call CloseAsync() on a channel before DisposeAsync or before the channel's using block exits? If not, please modify your code to always await CloseAsync() on a channel before disposing it. That may resolve the issue, and if it does, it would give me a hint as to the correct fix. Thanks.

[lukebakken]

I'm assuming your code is still similar to this comment: #1749 (comment)

        try
        {
            return await channel.QueueDeclarePassiveAsync(queueName);
        }
        catch (Exception)
        {
            await channel.DisposeAsync();
            throw;
        }

So the TaskCanceledException comes from a QueueDeclareAsync (or QueueDeclarePassiveAsync) and then your code immediately disposes the channel.

Is this correct?

[ZajacPiotr98]

I think TaskCancelledException is from the Continuation as error occurs when requests take much time (around 10 seconds). The code is similar - there is no CloseAsync invokations before disposal. I'll add this and check if it solves the problem

[ZajacPiotr98]

So I added CloseAsync before disposal and got this error System.InvalidOperationException: Received unexpected command of type QueueDeclareOk! at RabbitMQ.Client.Impl.Channel.CloseAsync(ShutdownEventArgs args, Boolean abort, CancellationToken cancellationToken). So it seems that when there was only Dispose invoked, the exception was aborted and caused disposal of semaphores, while it was still waiting for ChannelCloseOk command. So cancellation of commands is not working properly I think.

I could remove closing and disposing channel when TaskCanceledException is thrown from the Channel commands methods. However I don't see a way to make sure the channel is ready to use from other thread

[danielmarbach]

Isn't this something that can happen when the order of unregistering the async event handlers that we manage internally is not entirely correct, leading to the semaphore getting disposed from outside while the event handler triggers trying to acquire the semaphore?