spanner: transaction fails with `Session not found` error

[110y]

Client

Spanner: v0.37.4 (But as far as I know, it also occurs when we use latest version)

Describe Your Environment

Alpine Docker on GKE

Expected Behavior

If possible, this client library should retries a transaction when it fails with Session not found error.

Actual Behavior

We sometimes get the Session not found errors.
This client library retries transactions only when the Abort error occurred, so when taken session is not active, it just returns the NotFound error to callers without retrying.

• google-cloud-go/spanner/client.go

  Lines 394 to 398 in e4bd323

  	sh, err = c.idleSessions.takeWriteSession(ctx)
  	if err != nil {
  	// If session retrieval fails, just fail the transaction.
  	return err
  	}

• google-cloud-go/spanner/client.go

  Lines 439 to 454 in e4bd323

  	// Only retry on ABORTED.
  	if isAbortErr(funcErr) {
  	// Aborted, do exponential backoff and continue.
  	b, ok := extractRetryDelay(funcErr)
  	if !ok {
  	b = backoff.DefaultBackoff.Delay(retryCount)
  	}
  	trace.TracePrintf(ctx, nil, "Backing off after ABORTED for %s, then retrying", b)
  	select {
  	case <-ctx.Done():
  	return errContextCanceled(ctx, funcErr)
  	case <-time.After(b):
  	}
  	retryCount++
  	continue
  	}

Since this library creates the pool of the sessions, I think it should retry failed transactions caused by Session not found by taking another session. But are there any problems to take another session from the pool?

[olavloite]

The client library creates and maintains a session pool, and takes measures to keep these sessions alive by doing a GetSession ping call for sessions that have not been used for a while. Still it is possible that Cloud Spanner (or another user) deletes a session server side, causing a Session not found error when using the client library.

This could be mitigated by retrying failed transactions and other server calls that operate on a session by first taking a new session and then retrying. This is however not complete straightforward, as there are roughly 4 different categories of calls that we would need to consider in order of increasing complexity:

1. Single-use calls that just do one query or one update. These calls could simply be retried with a new session. This also includes the methods that do more than one gRPC call under water to achieve their goals, but that are presented to the user as a single call, such as executePartitionedUpdate.
2. Read/write transactions that fail with a Session not found. These are already executed in a retry-loop, and the Session not found condition could be added to the conditions that should cause the transaction to be retried, but with the additional requirement that the transaction needs to take a new session.
3. The BeginTransaction call of a MultiUseReadOnlyTransaction. Read-only transactions are normally not executed within a retry loop, but the initial BeginTransaction call could be retried on a new session.
4. Any query within a MultiUseReadOnlyTransaction that fails with a Session not found error would need a new read-only transaction to be created on a different session with the same read timestamp as the session that failed, and the query must be retried on the new transaction. The client library must execute any subsequent query for the transaction on the new read-only transaction.

[olavloite]

@110y Do you have any specific use case that reproduces this problem more frequently than others (or even always)?

[110y]

@olavloite

Do you have any specific use case that reproduces this problem more frequently than others (or even always)?

Unfortunately, I do not understand why this problem occurs...

One of the my case is:

• Just executing ReadWriteTransaction.
• There is at least 1 request to the spanner per second.
• Creates 60 sessions at initial time.
• Did not delete sessions in my code.
• Errors caused by Session not found are rarely happen (recently, there are 2 errors per week).

The document says:

There are two ways to delete a session:

• A client can delete a session.
• The Cloud Spanner database service can delete a session when the session is idle for more than 1 hour.

and, as far as my understanding is correct, if we are continuously sending requests to the spanner then there are no idle sessions for more than 1 hour since this library manages sessions in FIFO manner and used session will be pushed back to the session pool. That's why I'm wondering this problem and guessing the spanner deletes sessions even though idle time is not over 1 hour.

Do you have any thoughts or suggestions?

[olavloite]

@110y To my understanding, it is possible (although not very common) that Cloud Spanner deletes sessions that have been idle for less than 1 hour, which could cause this problem. The reason I asked whether you had a specific use case that would always (or often) cause this problem, was to check whether you were running into some unknown bug in the session pool. Considering your error rate of 1-2 errors per week at 1QPS, I don't think that this is a specific bug in the Go session pool.

The Java client library for Cloud Spanner added a protection against this problem a couple of months ago along the lines that I mentioned above. I'll have a look to see if it is feasible to add this protection for the Go client library as well.

[110y]

@olavloite

As trial, I send a CL that makes ReadWriteTransaction retry on Session not found error (case 2 you mentioned above).

[110y]

@olavloite

Could you please take a look this CL which fix the problem for ReadWriteTransaction?

https://code-review.googlesource.com/c/gocloud/+/45910

[olavloite]

@110y
Sorry for the delay on this. We had an integration test that started failing after one of the related changes for this was merged (not this one, the one for SingleUse transactions). It looks now like those fails were unrelated to these changes, but I wanted to make sure before proceeding with this. But I'll take another look into this change asap.

[110y]

@olavloite

I've updated my CL based on your comments.
Could you please take another look?

[kazegusuri]

@olavloite We still suffered from this error frequently. It happens hundreds of times everyday. What's the status of this issue?