Threads dont exit upon disconnection using 1.2.0 paho client

[prmathur-microsoft]

Please fill out the form below before submitting, thank you!

• [x ] Bug exists Release Version 1.2.0 ( Master Branch)
• Bug exists in Snapshot Version 1.2.1-SNAPSHOT (Develop Branch)

If this is a bug regarding the Android Service, please raise the bug here instead: https://github.com/eclipse/paho.mqtt.android/issues/new

After updating to latest paho client i.e 1.2.0 we notice that threads don't exit upon disconnect. Eg we still

1. Mqtt Con
2. Mqtt Rec
3. Mqtt call

Still remain active even after calling disconnect.

[jpwsutton]

@prmathur-microsoft Could you enable logging (https://wiki.eclipse.org/Paho/Log_and_Debug_in_the_Java_client) and provide the log from the point at which disconnect is called please?

[dougiman]

I have attached the log, thank you.

paho0.log

[prmathur-microsoft]

@jpwsutton - Can you please look at logs above and provide us a fix ?

[prmathur-microsoft]

@jpwsutton - Can you please look at logs that we provided ?

[aromanelli]

I too am seeing this problem, and would appreciate @jpwsutton (or others) review of this issue. Thank you in advance for your time and efforts, they are appreciated.

[jpwsutton]

Thanks for the log, this actually looks a bit similar to issue #330 where the client enters a deadlock state during a disconnection from an unstable connection. In this case it looks like the client had automatically reconnected before disconnecting. This is quite helpful to know as it should help us track down the core issue for this. Once we've found a resolution for #330, I'll commend back here so you can see if it's fixed your problem.

How easy is this for you to recreate reliably? Getting as much info about this problem would be very useful.

[aromanelli]

@jpwsutton Wasn't sure if you were addressing me or @prmathur-microsoft with your recreation question, but in my case, I can recreate it 100% of the time with a simple/small three class app/IJ project. In my case I am running the server (embedded) and the client on the same PC, so I don't believe I have any 'unstable connection' issues. Having said that, I'm newish to MQTT, so I may be doing a self-inflicted wound, but if you think my small app would help, I would be glad to zip it up and email it to you. Please advise.

Additional: Yesterday, when I first posted, I was getting the same quantity of active threads, MQTTs and a couple of pools, as @prmathur-microsoft , but today, I am only getting a single "pool-2-thread-1" active thread, which is strange as I did not change any of my code from yesterday to today. Ran a stack trace via YourKit Java Profiler, and got the following on the non-terminating active "pool-2-thread-1" thread ...

1. java.util.concurrent.ThreadPoolExecutor.execute(Runnable) ThreadPoolExecutor.java:1367
2. io.moquette.spi.impl.BrokerInterceptor.notifyClientConnected(MqttConnectMessage) BrokerInterceptor.java:97
3. io.moquette.spi.impl.ProtocolProcessor.processConnect(Channel, MqttConnectMessage) ProtocolProcessor.java:283
4. io.moquette.server.netty.NettyMQTTHandler.channelRead(ChannelHandlerContext, Object) NettyMQTTHandler.java:46
5. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Object) AbstractChannelHandlerContext.java:340
6. io.moquette.server.netty.metrics.MQTTMessageLogger.channelRead(ChannelHandlerContext, Object) MQTTMessageLogger.java:44
7. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Object) AbstractChannelHandlerContext.java:340
8. io.moquette.server.netty.metrics.MessageMetricsHandler.channelRead(ChannelHandlerContext, Object) MessageMetricsHandler.java:48
9. io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Object) AbstractChannelHandlerContext.java:340
10. io.moquette.server.netty.metrics.BytesMetricsHandler.channelRead(ChannelHandlerContext, Object) BytesMetricsHandler.java:49
11. java.lang.Thread.run() Thread.java:748

And the current runtime state of the "pool-2-thread-1" thread ...

Stacks at 2017-11-08 03:35:08 PM. Uptime is 26m 0s 900ms.
pool-2-thread-1 State: WAITING CPU usage on sample: 0ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() AbstractQueuedSynchronizer.java:2039
java.util.concurrent.LinkedBlockingQueue.take() LinkedBlockingQueue.java:442
java.util.concurrent.ThreadPoolExecutor.getTask() ThreadPoolExecutor.java:1074
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1134
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

I can also supply a YourKit Java Profiler snapshot file (3Kb), that generated the above information, if you wish.

[prmathur-microsoft]

Is there any update on this issue ?

[prmathur-microsoft]

@jpwsutton - Any update on this issue ?

[frank-montyne]

I'm having the same problem. Any update? This is a real critical stability issue. Thanks

[timtay-microsoft]

A workaround that I just found is to close your mqttAsyncClient after disconnecting it.

IMqttToken disconnectToken = mqttAsyncClient.disconnect();
disconnectToken.waitForCompletion();

//add this line
mqttAsyncClient.close();

Inside this close call, the Executor Service that is hosting those errant threads is shutdown, so those threads die. It also makes your mqttAsyncClient unusable according to its documentation, so make sure not to reuse an MqttAsyncClient after closing it.

It still seems to me like a bug that the threads survive the disconnect, but if you are in a rush, this workaround worked great for me.

@jpwsutton do you have any comments on using close in this way?

[aromanelli]

@timtay-microsoft Thanks for the suggested workaround. I had already had a close(true) (force the close) call in my code that reproduces the bug. I tried just calling close() instead, but I still see the same issue, the app never closes, and a thread is left out there running. So the workaround didn't work in my case. Do appreciate the effort/knowledge sharing, thank you for that.