RATIS-1882. Support read-after-write consistency

[SzyWilliam]

What changes were proposed in this pull request?

Support read-after-write consistency.
Thanks @szetszwo very much for the patch. Only made minor changes and added unit tests to the patch.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-1882

How was this patch tested?

New unit tests.

[szetszwo]

+1 the change looks good. Just a very minor comment inlined.

Also, spent some time to add another unit test; see
https://issues.apache.org/jira/secure/attachment/13062665/TestReadAfterWrite.patch
Could you include it?

[szetszwo]

Just found that these few lines can be simplified as below.

if (ref != null) {
  return ref.get();
}

[SzyWilliam]

Done

[szetszwo]

@SzyWilliam , Should sendReadAfterWrite use UnorderedAsync.send(..)? Indeed, all the linearizable reads should use UnorderedAsync.

[SzyWilliam]

That's true! Thanks for pointing out, I'll change all the linearizable reads to use UnorderedAsync

[SzyWilliam]

@szetszwo Thanks very much for the patch! It's good to know CodeInjectionForTesting can be used for blocking. I'll try using Injection to replace the sleep time in the linearizable read index tests.

[szetszwo]

Let's use OrderedAsync due to the test failure. Could you also include the following?

diff --git a/ratis-common/src/main/java/org/apache/ratis/protocol/RaftClientRequest.java b/ratis-common/src/main/java/org/apache/ratis/protocol/RaftClientRequest.java
index 4b0c5166..220694ce 100644
--- a/ratis-common/src/main/java/org/apache/ratis/protocol/RaftClientRequest.java
+++ b/ratis-common/src/main/java/org/apache/ratis/protocol/RaftClientRequest.java
@@ -227,7 +227,10 @@ public class RaftClientRequest extends RaftClientMessage {
         case MESSAGESTREAM:
           return toString(getMessageStream());
         case READ:
-          return "RO";
+          final ReadRequestTypeProto read = getRead();
+          return read.getReadAfterWriteConsistent()? "RaW"
+              : read.getPreferNonLinearizable()? "RO(pNL)"
+              : "RO";
         case STALEREAD:
           return "StaleRead(" + getStaleRead().getMinIndex() + ")";
         case WATCH:
diff --git a/ratis-examples/src/test/java/org/apache/ratis/server/impl/TestReadAfterWrite.java b/ratis-examples/src/test/java/org/apache/ratis/server/impl/TestReadAfterWrite.java
index f8f2405c..f515628c 100644
--- a/ratis-examples/src/test/java/org/apache/ratis/server/impl/TestReadAfterWrite.java
+++ b/ratis-examples/src/test/java/org/apache/ratis/server/impl/TestReadAfterWrite.java
@@ -128,8 +128,10 @@ public class TestReadAfterWrite
     LOG.info("readAfterWrite.get");
     try {
       // readAfterWrite should time out
-      readAfterWrite.get(100, TimeUnit.MILLISECONDS);
-      Assert.fail();
+      final RaftClientReply reply = readAfterWrite.get(100, TimeUnit.MILLISECONDS);
+      final DoubleValue result = (DoubleValue) Expression.Utils.bytes2Expression(
+          reply.getMessage().getContent().toByteArray(), 0);
+      Assert.fail("result=" + result + ", reply=" + reply);
     } catch (TimeoutException e) {
       LOG.info("Good", e);
     }

[szetszwo]

Sorry, it actually not working since UnorderedAsync use a different channel. It can arrive the server earlier than the previous write request and get a small readIndex. TestReadAfterWrite will fail.

• Using unordered async

2023-09-04 11:16:59,888 DEBUG server.RaftServer$Division (LeaderStateImpl.java:getReadIndex(1099)) - readIndex=1, readAfterWriteConsistentIndex=1

• Using ordered async

2023-09-04 11:11:36,917 DEBUG server.RaftServer$Division (LeaderStateImpl.java:getReadIndex(1099)) - readIndex=3, readAfterWriteConsistentIndex=3

[SzyWilliam]

You are right. Just investigated on OrderedAsync and UnorderedAsync and now I know why we should choose OrderedAsync. Thanks very much for your time spent and the detailed explanation!

[tisonkun]

Cool! I'm checking this out.

[tisonkun]

May I ask what is the target release version of this feature?

[szetszwo]

We should include this to both 3.0.0 and the next branch 2 releases.