Akka Persistence Throwing Unobserved Task Exception

[stedmanm]

Version Information
Version of Akka.NET? 1.5.28
Which Akka.NET Modules? Akka.Persistence

Describe the bug
Bug appears to be in method CallThrough of class AtomicState. If WaitAsync(Timeout timeout) completes before the task implemented by SnapshotStore/Journal, and the task in question throws an exception, then this exception will become unobserved.

To Reproduce

1. Snapshot loading takes longer than call timeout specified by Akka framework, and Snapshot call throws an exception.

Expected behavior
Would expect Akka framework to observe exception thrown by SnapshotStore/Journal.

Actual behavior
Causes unobserved task exception and crashes our service.

Screenshots

Environment
Windows Service and .NET 7

[Aaronontheweb]

@stedmanm thanks for reporting this! We'll look into it

[Arkatufus]

A fix to this issue will have a very high knock on effect.

To fix this effectively, we'd need to implement cancellation token to all journal/snapshot database related methods to effectively signal a cancellation to the underlying task.

That means implementing methods supporting cancellation tokens to ALL public async API methods that is also backward compatible, this will be a binary breaking change for all persistence plugins.

[Arkatufus]

This is an actual bug, here is a simple reproduction to illustrate this: https://share.linqpad.net/pbjjq82i.linq

An .AsyncWait() call does throw a TimeoutException when the execution time exceeds the timeout argument, but it does not stop the running task, making it a detached task running in the background.