HDDS-10593. Make client priority read from IN_SERVICE status Datanode

[xichen01]

What changes were proposed in this pull request?

Make client priority read from IN_SERVICE status Datanode.

When Ozone client read Block form Datanode, the SCM will return all the Datanodes where the replicas are located, including the nodes in the IN_MAINTENANCE, etc state, and the order may be random, so the client may try to access the IN_MAINTENANCE state node first.
Sometimes, we set a Datanode to IN_MAINTENANCE, this may be due to some faults of the machine or wanting to restart a Datanode, in this scenario, that the Client does not read data form these IN_MAINTENANCE datanode is a good strategy.

PS:
Since OM has a cache of Datanode state (expiration time 6min), the client can't get the latest state of Datanode, so we need to enhance the cache evicting mechanism.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10593

How was this patch tested?

Unit Test.

[ivandika3]

Thanks for the patch. LGTM +1.

[sumitagrawl]

@xichen01 Thanks for working over this, LGTM

[xichen01]

Thanks @ivandika3 and @sumitagrawl for the review.

[sodonnel]

Change seems sensible to me. I tried to think if there was a nicer way to do this, and thought that the following was easier to follow:

    Iterator<DatanodeDetails> iter = datanodeList.iterator();
    List<DatanodeDetails> notInService = null;
    while(iter.hasNext()) {
      DatanodeDetails dn = iter.next();
      if (dn.getPersistedOpState() != HddsProtos.NodeOperationalState.IN_SERVICE) {
        iter.remove();
        if (notInService == null) {
          notInService = new ArrayList<>();
        }
        notInService.add(dn);
      }
    }
    if (notInService != null) {
      datanodeList.addAll(notInService);
    }

It could be shorter, but I tried to avoid allocating a list for notInService unless it was really needed, as it won't be needed most of the time.

Only change this if you feel it is better - I am not insisting it should be changed.

[xichen01]

Thanks for your suggestion. This Implementation need to remove and readd the element, this may not bring in more benefits compare with the sort.

[adoroszlai]

Thanks @xichen01 for the patch, @ivandika3, @sodonnel, @sumitagrawl for the review.