Spanner: TransactionRunner.run() sometimes doesn't retry on ABORTED

[fyhuang-rideos]

Hello, we've noticed that sometimes running executeUpdate throws an AbortedException (directly from inside executeUpdate). This happens, for example, when running queries like DELETE FROM Table WHERE true. When this happens inside TransactionRunner.run(), the transaction is not automatically retried, but the AbortedException is immediately rethrown to the caller. Note that the AbortedException appears to be thrown from executeUpdate, not transaction.commit().

The message in the exception says something like:

com.google.cloud.spanner.AbortedException: ABORTED: com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Transaction aborted. Database schema probably changed during transaction, retry may succeed.

We've found that if we wrap this in a retry loop, it does in fact succeed as suggested. We're wondering whether this is a case that should be handled by TransactionRunner's retry logic, or whether there's something we should be doing differently in our code. Thanks!

Environment details

2. OS type and version: OSX High Sierra.
3. Java version: 1.8.0
4. google-cloud-java version(s): google-cloud-spanner v0.70.0

Code example

// we are primarily seeing this while using RemoteSpannerHelper
final RemoteSpannerHelper remoteSpannerHelper = RemoteSpannerHelper.create();
final Database db = remoteSpannerHelper.createTestDatabase(DDL);
final DatabaseClient dbClient = remoteSpannerHelper.getDatabaseClient(db);

while (true) {
  dbClient.readWriteTransaction().run(tx -> {
    // this sometimes throws AbortedException
    tx.executeUpdate(Statement.of("DELETE FROM Table WHERE true"));
  });
}

Stack trace

com.google.cloud.spanner.AbortedException: ABORTED: com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Transaction aborted. Database schema probably changed during transaction, retry may succeed.
retry_delay {
  nanos: 10997310
}

	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:117)
	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
	at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.get(GapicSpannerRpc.java:526)
	at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.executeQuery(GapicSpannerRpc.java:456)
	at com.google.cloud.spanner.SpannerImpl$TransactionContextImpl$2.call(SpannerImpl.java:1643)
	at com.google.cloud.spanner.SpannerImpl$TransactionContextImpl$2.call(SpannerImpl.java:1640)
	at com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:246)
	at com.google.cloud.spanner.SpannerImpl$TransactionContextImpl.executeUpdate(SpannerImpl.java:1639)
	(one line from our closure, that calls executeUpdate)
	at com.google.cloud.spanner.SpannerImpl$TransactionRunnerImpl.runInternal(SpannerImpl.java:1312)
	at com.google.cloud.spanner.SpannerImpl$TransactionRunnerImpl.run(SpannerImpl.java:1280)
	at com.google.cloud.spanner.SessionPool$PooledSession$1.run(SessionPool.java:410)

[olavloite]

This issue was fixed in PR #4181 that was released with version 0.73.0. If you upgrade to version 0.73.0 or newer of the client library, you should no longer experience this issue. I was able to reproduce the problem in 0.70.0 and to confirm that the current version of the client library does not have this problem.
Please re-open if this does not solve your issue.