[BUG] Region Exception occurred EpochNotMatch

[yongpan0709]

Describe the bug
When concurrently writing data into tikv, QPS of raw_put has 3 kinds of response:

1. QPS reflected on grafana keeps going as the pressure from client.
2. QPS starts at the expect speed such as 3000 or 6000, suddenly drops to 10 ~ 100.
3. QPS is stable at 10 ~ 100, but the pressure on client is 1000 or higher.

And the client gets the same warn info:
org.tikv.common.exception.RegionException: Region Exception occurred EpochNotMatch current epoch of region 17487791 is conf_ver: 695 version: 7749, but you sent conf_ver: 695 version: 7748.

What did you do
Different concurrency and varied frequency of calling put func.

What do you expect

1. stable QPS
2. QPS would recover to the expected after tens of milliseconds when exceptions happened.

What happens instead
2021-11-02 12:01:14,821 [tmp-write-3] WARN org.tikv.raw.RawKVClient - Retry for put error
org.tikv.common.exception.RegionException: Region Exception occurred EpochNotMatch current epoch of region 17487791 is conf_ver: 695 version: 7749, but you sent conf_ver: 695 version: 7748
at org.tikv.common.region.RegionStoreClient.rawPutHelper(RegionStoreClient.java:932)
at org.tikv.common.region.RegionStoreClient.rawPut(RegionStoreClient.java:916)
at org.tikv.raw.RawKVClient.put(RawKVClient.java:137)
at org.tikv.raw.RawKVClient.put(RawKVClient.java:115)
at org.pingcap.ccbload.Workload$TmpWrite.accept(Workload.java:175)
at org.pingcap.ccbload.Workload$TmpWrite.accept(Workload.java:155)
at org.pingcap.ccbload.Workload.lambda$build$1(Workload.java:314)

Java Client and TiDB/TiKV version info
3.1.3-SNAPSHOT

[github-actions]

This issue is stale because it has been open 30 days with no activity.