spanner: Intermittent error "Transaction was started in a different session"

[lc-caigwatkin]

Client

Spanner

Environment

Cloud Run

Go Environment

go version go1.19.5, go1.20.1 to go1.20.4

Code

Seems to not be isolated to any particular piece of our code.

Here is an example of cloud.google.com/go/spanner related stack trace when we do see the error:

cloud.google.com/go/spanner.(*ReadWriteTransaction).runInTransaction
	/root/project/vendor/cloud.google.com/go/spanner/transaction.go:1505
cloud.google.com/go/spanner.(*Client).rwTransaction.func2
	/root/project/vendor/cloud.google.com/go/spanner/client.go:557
cloud.google.com/go/spanner.runWithRetryOnAbortedOrFailedInlineBeginOrSessionNotFound.func1
	/root/project/vendor/cloud.google.com/go/spanner/retry.go:92
cloud.google.com/go/spanner.runWithRetryOnAbortedOrFailedInlineBeginOrSessionNotFound
	/root/project/vendor/cloud.google.com/go/spanner/retry.go:132
cloud.google.com/go/spanner.(*Client).rwTransaction
	/root/project/vendor/cloud.google.com/go/spanner/client.go:522
cloud.google.com/go/spanner.(*Client).ReadWriteTransaction
	/root/project/vendor/cloud.google.com/go/spanner/client.go:490

Expected behavior

Transaction executes without error OR documentation includes information to understand and resolve this error.

Actual behavior

The following error occasionally pops up in our production environment across multiple databases/microservices in which query logic and database schema barely change.

spanner: code = "InvalidArgument", desc = "Transaction was started in a different session."

Screenshots

In our production environment, last 30 days, showing our error or warning logs when saw that error:

Additional context

We started seeing this error "Transaction was started in a different session" pop up occasionally in our production environment after upgrading to v1.44.0 in February 2023 (the first time this happened was 2023-02-12).
This appears to happen after the transaction automatically retries.
It appears to occur in our Spanner databases with higher contention, although our database usage has been higher in the past without this particular error occurring.

We cannot find any information about this new error to help us understand if this is something we can fix.
We have looked in GCP documentation as well as the client library's docs and code.
We are used to seeing and dealing with "Transaction was aborted" errors or similar, however, this new one seems different and we are still seeing those other errors.

Hoping you can help us understand what this error means.

[rahul2393]

Hello @lc-caigwatkin what is the Go client library version to replicate the issue?

[lc-caigwatkin]

Hello @lc-caigwatkin what is the Go client library version to replicate the issue?

Hi @rahul2393, we started seeing this error "Transaction was started in a different session" pop up occasionally in our production environment after upgrading to v1.44.0 in February 2023.

[rahul2393]

Thanks, can you also please share the operations which you are performing inside ReadWriteTransaction

[lc-caigwatkin]

Thanks for having this looked into and fixed! Did you still need any extra information? Otherwise we can keep an eye out to see that it doesn't come up again after the new version is released.