Catalog get table API return null sometimes

[humengyu2012]

Bug Report

1. Describe the bug

When I use method catalog.getTable(db,table), I may get a null object of TiTableInfo.
I use client-java in mapreduce, and there are more than 2000 calls of the method catalog.getTable(db,table), each mapper will call one or more times.
I found that a very small proportion(1%) of mappers will get null TiTableInfo object.

Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.lang.IllegalStateException: Table `tag_core`.`content_tag_object_relation` no longer exists in TiDB
        at io.tidb.bigdata.hive.TiDBSerde.initialize(TiDBSerde.java:62)
        at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
        at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531)
        at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:177)
        at org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:186)
        at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:386)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:106)
        ... 22 more
Caused by: java.lang.IllegalStateException: Table `tag_core`.`content_tag_object_relation` no longer exists in TiDB
        at io.tidb.bigdata.tidb.ClientSession.lambda$getTableMust$0(ClientSession.java:173)
        at java.util.Optional.orElseThrow(Optional.java:290)
        at io.tidb.bigdata.tidb.ClientSession.getTableMust(ClientSession.java:171)
        at io.tidb.bigdata.tidb.ClientSession.getTableColumnsMust(ClientSession.java:227)
        at io.tidb.bigdata.hive.TiDBSerde.initialize(TiDBSerde.java:60)
        ... 28 more

The implements of ClientSession.lambda$getTableMust$0(ClientSession.java:173) is:

2. Minimal reproduce step (Required)

I can't reproduce it because it happens by accident.

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What are your Java Client and TiKV versions? (Required)

• Client Java: 3.2.0
• TiKV: 5.0.2

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[pingyu]

Steps to reproduce:

1. Setup a TiDB cluster of v5.0.2
2. Create a database sbtest
3. Prepare and run sysbench workload
4. `getDatabase("sbtest") repeatedly.

while (true) {
    TiConfiguration conf = TiConfiguration.createDefault("172.16.5.32:44379");
    try (TiSession session = TiSession.create(conf)) {
      Catalog catalog = session.getCatalog();
      TiDBInfo db = catalog.getDatabase("sbtest");
      Objects.requireNonNull(db, "db is null");
    }
}

Note that:

1. getTable return null is caused by getDatabase is null.
2. This issue also exists in 3.3.0.
3. Can also be reproduced on TiKV v6.4.0.

Cause

getDatabase is implemented by KvScan. When KvScan meets locks, the current codes just ignore the error. So getDatabase can an empty list of key-value pairs.

See doScan.

Additional logs in doScan when getDatabase return null (resp is the value of ScanResponse):

INFO    2022-11-20 22:58:27,016 115223  org.tikv.common.region.RegionStoreClient        [thread-0]      handleScanResponse, resp:pairs {
  error {
    locked {
      primary_lock: "t\200\000\000\000\000\000\0009_r\200\000\000\000\000\000\003\256"
      lock_version: 437506882096070751
      key: "t\200\000\000\000\000\000\000\252_r\200\000\000\000\000\000\003o"
      lock_ttl: 20083
      txn_size: 1
      lock_for_update_ts: 437506882096070751
      use_async_commit: true
      min_commit_ts: 437506882122285136
    }
  }
}
error {
  locked {
    primary_lock: "t\200\000\000\000\000\000\0009_r\200\000\000\000\000\000\003\256"
    lock_version: 437506882096070751
    key: "t\200\000\000\000\000\000\000\252_r\200\000\000\000\000\000\003o"
    lock_ttl: 20083
    txn_size: 1
    lock_for_update_ts: 437506882096070751
    use_async_commit: true
    min_commit_ts: 437506882122285136
  }
}

INFO    2022-11-20 22:58:27,017 115224  org.tikv.common.region.RegionStoreClient        [thread-0]      handleScanResponse, locks:1