NullReferenceException in MySqlConnector.MySqlDataReader.ActivateResultSet

[feiazifeiazi]

An optimization introduced in v2.3.0 started recycling MySqlDataReader instances: #1277.

If client code held onto the MySqlDataReader and disposed it after the associated MySqlConnection had been returned to the connection pool, this could cause that connection to be disposed while it was being used by another thread, leading to unpredictable NullReferenceExceptions.

While many access patterns didn't trigger this bug (e.g., no problems have been reported by users of Pomelo.EntityFrameworkCore.MySql), using CommandBehavior.CloseConnection or Dapper's QueryAsync<T> made it more likely to happen.

Original bug report below.

---

Software versions
MySqlConnector version: 2.3.5
Server type (MySQL, MariaDB, Aurora, etc.) and version: 8.0.25
.NET version: net7.0
Dapper version: 2.1.28

Describe the bug
NullReferenceException

Exception

[Error Message]=Object reference not set to an instance of an object.
[Exception Type]=System.NullReferenceException
[Source]=MySqlConnector
[TargetSite]=Void ActivateResultSet(System.Threading.CancellationToken)
[Stack Trace]=   at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 133
   at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 505
   at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 78
   at MySqlConnector.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 317
   at MySqlConnector.MySqlCommand.ExecuteNonQuery() in /_/src/MySqlConnector/MySqlCommand.cs:line 108
   at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, CommandDefinition& command, Action`2 paramReader) in /_/Dapper/SqlMapper.cs:line 2975
   at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, CommandDefinition& command) in /_/Dapper/SqlMapper.cs:line 656
   at Wwtfly.DBMonitor.DAL_Dapper.DBLinkConfigDAL.UpdateExecCount()

Code sample

-

Expected behavior

Additional context
This method is an update SQL . Its execution frequency is approximately 50 times per second.

The error randomly occurs 20 times per day.

Link to the exception line

[bgrainger]

Can you provide a self-contained example that reproduces the problem?

A NullReferenceException in MySqlConnector is most typically indicative of broken internal invariants, usually caused by multithreaded misuse. See https://mysqlconnector.net/troubleshooting/connection-reuse/ and note that connections must not be accessed by multiple threads at the same time.

[bgrainger]

Related: #1427 (comment)

[feiazifeiazi]

Can you provide a self-contained example that reproduces the problem?

A NullReferenceException in MySqlConnector is most typically indicative of broken internal invariants, usually caused by multithreaded misuse. See https://mysqlconnector.net/troubleshooting/connection-reuse/ and note that connections must not be accessed by multiple threads at the same time.

thank you very much. I will study the code. (This is old code, converted from Oracle MySQL driver.)

[jayrowe]

I'm seeing the same thing.

Nothing database-related changed other than updating the versions of MySqlConnector and Dapper earlier this week.

I looked for threading issues. The places where database connections are created and used are pretty minimal (abstracted into some common code). Every connection create has a using statement and there aren't any places where the connection is created and then returned to the caller.

I'm going to try rolling back to an older version to see what happens.

More Context:

• Upgraded MySqlConnector from 2.1.11 to 2.3.5
• Upgrade Dapper from 2.0.123 to 2.1.35

Sometimes there is a null reference during Dispose:

   at MySqlConnector.MySqlDataReader.DisposeAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 612
   at System.Data.Common.DbDataReader.Dispose(Boolean disposing)
   at MySqlConnector.MySqlDataReader.Dispose(Boolean disposing) in /_/src/MySqlConnector/MySqlDataReader.cs:line 443
   at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 472

Sometimes there is an null reference during ActivateResultSet:

   at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 117
   at MySqlConnector.MySqlDataReader.InitAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, IDictionary`2 cachedProcedures, IMySqlCommand command, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 483
   at MySqlConnector.Core.CommandExecutor.ExecuteReaderAsync(CommandListPosition commandListPosition, ICommandPayloadCreator payloadCreator, CommandBehavior behavior, Activity activity, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/Core/CommandExecutor.cs:line 56
   at MySqlConnector.MySqlCommand.ExecuteReaderAsync(CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 357
   at MySqlConnector.MySqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlCommand.cs:line 350

There is a packet out-of-order error, too:

One or more errors occurred. (Packet received out-of-order. Expected 1; got 2.)

I don't have a usable stack trace for the last one, since it is wrapped up in an AggregateException

[bgrainger]

Sometimes there is a null reference during Dispose:

   at MySqlConnector.MySqlDataReader.DisposeAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 612
   at System.Data.Common.DbDataReader.Dispose(Boolean disposing)
   at MySqlConnector.MySqlDataReader.Dispose(Boolean disposing) in /_/src/MySqlConnector/MySqlDataReader.cs:line 443
   at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 472

This call stack doesn't "look right". I wonder if there's an earlier exception that's being thrown, but this NullReferenceException is coming from a second exception being thrown while finally blocks are being run.

[jayrowe]

@bgrainger

I agree, but if you look at that stack trace, you can see that it's in Dapper. Dapper is opening and closing the connections and disposing the reader for me. I'm really not managing connections other than creating them and calling Dispose or DisposeAsync. I'm dealing with the readers even less frequently.

I rolled back to 2.1.11 yesterday midday, and it hasn't happened since. It was happening quite consistently since I upgraded. The chart of error counts related to this is pretty damning so far. You can see when I upgraded and when I downgraded; usage has been consistent throughout the time period.

[bgrainger]

I work on systems with MySqlConnector 2.3.5 deployed to production and opening about 1,000,000 database connections per hour. We have not seen widespread NullReferenceExceptions like you're reporting.

Are you able to provide a small repro that demonstrates the problem?

Or can you provide real (not abbreviated or sanitised) examples of what your calling code (with Dapper) looks like?

[jayrowe]

I tried inducing the failure before I rolled back but was unable to do it on demand.

It's pretty bog standard. Things are abstracted for instrumentation purposes so the failing methods generally are one of these:

    protected async Task<List<TResult>> QueryAsync<TResult>(string sql, object? parameters = null, CommandFlags flags = CommandFlags.Buffered, CancellationToken cancellationToken = default)
    {
        flags |= CommandFlags.Buffered;

        using var connection = _connectionFactory.Create();

        return (List<TResult>)await connection.QueryAsync<TResult>(
            new CommandDefinition(
                sql,
                parameters,
                flags: flags,
                cancellationToken: cancellationToken));
    }

    protected async Task ExecuteAsync(string sql, object? parameters = null, CancellationToken cancellationToken = default)
    {
        using var connection = _connectionFactory.Create();

        await connection.ExecuteAsync(
            new CommandDefinition(sql, parameters, cancellationToken: cancellationToken));
    }

    protected async Task<TResult> ExecuteScalarAsync<TResult>(string sql, object? parameters = null, CancellationToken cancellationToken = default)
    {
        using var connection = _connectionFactory.Create();

        return await connection.ExecuteScalarAsync<TResult>(
            new CommandDefinition(sql, parameters, cancellationToken: cancellationToken));
    }

_connectionFactory.Create() just creates the new instance with the proper connection string as there are separate databases and users with different permissions.

I looked through the cases not using one of those methods and most follow the exact same pattern - create connection with a using statement, call Dapper. The ones that don't follow the pattern are where there is a low number of iterations calling a stored procedure to do something complicated, so the code calls OpenAsync on the connection once and leaves it open.

It's happening across different connections to different databases with different connection strings, and rolling back made the problem disappear for the last 24+ hours after having been consistent for five or so days.

I'm on .NET 8 and I'm connecting to an AWS Aurora MySQL cluster if either of those is a complicating factor.

[bgrainger]

Thanks for the update. That code looks pretty "bog standard", as you said. CommandFlags.Buffered is a little unusual (I don't think most Dapper users use it), but shouldn't cause any problems.

Does _connectionFactory.Create() also open the connection, or do you (almost always) let Dapper open and close the connection?

[jayrowe]

It just creates the connection and returns it; it does not open it. With very few exceptions I let Dapper open and close the connection.

I'm swamped right now but I'll try to circle back when I get a chance. I can probably deploy a version to my staging environment with an instrumented build to try to catch it in action. It happens in that environment but the frequency of data updates is much lower so it isn't as consistent as it was in my production environment.

[bgrainger]

I see you're passing in cancellationToken: cancellationToken. Is it possible that these exceptions are correlated with cancelled queries? Do you know how frequently cancellation is happening?

[jayrowe]

I would expect it to be rare; the case would be a cancelled http request in a user-facing path. Most of our content is served up from OpenSearch, not from the database.

The paths throwing exceptions are in background message processing for data updates and those don't pass in any CancellationToken.

[lazorfuzz]

+1 we are seeing the exact same issue, with Dapper 2.0.90 and MySqlConnector 2.3.7, on .NET 6. Pretty standard usage, no crazy black magic with threads as far as I can tell.

[bgrainger]

Is anyone who is experiencing this problem able to provide a self-contained example that reproduces the exception?

[bgrainger]

Upgraded MySqlConnector from 2.1.11 to 2.3.5

One major change in 2.3.5 was #1277 which recycles MySqlDataReader instances. Having two threads use the same instance at once would be a significant bug (and misuse of the library).

at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 133

The only thing that could be null on https://github.com/mysql-net/MySqlConnector/blob/2.3.5/src/MySqlConnector/MySqlDataReader.cs#L133 is m_resultSet.ReadResultSetHeaderException but we're inside an if block that checks if it's not null.

at MySqlConnector.MySqlDataReader.ActivateResultSet(CancellationToken cancellationToken) in /_/src/MySqlConnector/MySqlDataReader.cs:line 117

This line https://github.com/mysql-net/MySqlConnector/blob/2.3.5/src/MySqlConnector/MySqlDataReader.cs#L117 also can't throw, but the lines before and after also read m_resultSet.ReadResultSetHeaderException.

This value is set to null in ResultSet.Reset; if that were somehow being called on another thread, we could see this exception.