Skip to content

Conversation

@SpriCoder
Copy link
Contributor

@SpriCoder SpriCoder changed the title upgrade dropwizard-metric from 3.2.5 to 4.2.9(latest) and remain metr… RATIS-1391 and RATIS-1565: upgrade dropwizard-metric from 3.2.5 to 4.2.9(latest) and remain metr… Apr 15, 2022
@SpriCoder
Copy link
Contributor Author

@szetszwo PTAL!

@szetszwo
Copy link
Contributor

@SpriCoder , TestRaftLogMetrics failed in a few re-runs. Could you take a look? See https://github.com/apache/ratis/runs/6035048607?check_suite_focus=true#step:5:511

@SpriCoder
Copy link
Contributor Author

@szetszwo I have already find the reason, that is the implementation of DefaultObjectNameFactory in dropwizard are different between 3.2.5 and 4.2.7. So, I will implement RatisObjectNameFactory.class to compatible with pre-version of ratis.

@SpriCoder
Copy link
Contributor Author

@szetszwo Thanks for your time! I have already fix naming problem.

@SpriCoder
Copy link
Contributor Author

@szetszwo I have already checked CI, it seems like there are some test out of time, can you take a look?

@szetszwo
Copy link
Contributor

@SpriCoder , thanks for update. Just have started re-running the failed jobs.

@szetszwo
Copy link
Contributor

@SpriCoder , TestLeaderElectionMetrics.testOnLeaderElectionCompletion failed again. Could you take a look?

@SpriCoder
Copy link
Contributor Author

@szetszwo I run this test locally and have passed in ten times, so can you help me to rerun this test?

@szetszwo
Copy link
Contributor

@SpriCoder , sure, let's try again.

@SpriCoder
Copy link
Contributor Author

@szetszwo , it's sad that the unit test is failed again. I need your help. What I am confused is that whether the LAST_LEADER_ELECTION_ELAPSED_TIME gauge can be zero?
image

@szetszwo
Copy link
Contributor

@SpriCoder , not sure. Let's print out the map and the value as below?

diff --git a/ratis-server/src/test/java/org/apache/ratis/server/metrics/TestLeaderElectionMetrics.java b/ratis-server/src/test/java/org/apache/ratis/server/metrics/TestLeaderElectionMetrics.java
index 62707845..af000f3a 100644
--- a/ratis-server/src/test/java/org/apache/ratis/server/metrics/TestLeaderElectionMetrics.java
+++ b/ratis-server/src/test/java/org/apache/ratis/server/metrics/TestLeaderElectionMetrics.java
@@ -23,6 +23,8 @@ import static org.apache.ratis.server.metrics.LeaderElectionMetrics.LEADER_ELECT
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertTrue;
 
+import com.codahale.metrics.Gauge;
+import org.apache.ratis.BaseTest;
 import org.apache.ratis.metrics.RatisMetricRegistry;
 import org.apache.ratis.protocol.RaftGroupId;
 import org.apache.ratis.protocol.RaftGroupMemberId;
@@ -30,10 +32,12 @@ import org.apache.ratis.protocol.RaftPeerId;
 import org.junit.BeforeClass;
 import org.junit.Test;
 
+import java.util.SortedMap;
+
 /**
  * Test for LeaderElectionMetrics.
  */
-public class TestLeaderElectionMetrics {
+public class TestLeaderElectionMetrics extends BaseTest {
 
   private static LeaderElectionMetrics leaderElectionMetrics;
   private static RatisMetricRegistry ratisMetricRegistry;
@@ -50,9 +54,11 @@ public class TestLeaderElectionMetrics {
   @Test
   public void testOnLeaderElectionCompletion() throws Exception {
     leaderElectionMetrics.onNewLeaderElectionCompletion();
-    Long leaderElectionLatency = (Long) ratisMetricRegistry.getGauges((s, metric) ->
-        s.contains(LAST_LEADER_ELECTION_ELAPSED_TIME)).values().iterator().next().getValue();
-    assertTrue(leaderElectionLatency > 0L);
+    final SortedMap<String, Gauge> gauges = ratisMetricRegistry.getGauges(
+        (s, metric) -> s.contains(LAST_LEADER_ELECTION_ELAPSED_TIME));
+    LOG.info("{} gauges: {}", LAST_LEADER_ELECTION_ELAPSED_TIME, gauges);
+    final Long leaderElectionLatency = (Long)gauges.values().iterator().next().getValue();
+    assertTrue("leaderElectionLatency = " + leaderElectionLatency, leaderElectionLatency > 0L);
   }
 
   @Test

@SpriCoder
Copy link
Contributor Author

@szetszwo Thanks for your suggestion, I have already modified.

@szetszwo
Copy link
Contributor

@SpriCoder , all the tests have passed now. Let's document the incompatibility of this change. I have posted some questions in https://issues.apache.org/jira/browse/RATIS-1391 , please take a look.

@SpriCoder
Copy link
Contributor Author

@szetszwo Great! So is there anything to do before merge?

@szetszwo
Copy link
Contributor

szetszwo commented May 4, 2022

@SpriCoder , unfortunately, there is no response to our questions from the mailing list. We are currently preparing the the 2.3.0 release. Let's merge this once the 2.3.0 has been released.

@szetszwo szetszwo changed the title RATIS-1391 and RATIS-1565: upgrade dropwizard-metric from 3.2.5 to 4.2.9(latest) and remain metr… RATIS-1391. Update library dropwizard.metrics version to 4.x Jun 6, 2022
@szetszwo szetszwo merged commit ec82b04 into apache:master Jun 6, 2022
codings-dan pushed a commit to codings-dan/incubator-ratis that referenced this pull request Jun 25, 2022
codings-dan pushed a commit that referenced this pull request Jun 25, 2022
SzyWilliam pushed a commit to SzyWilliam/ratis that referenced this pull request Jul 2, 2022
kaijchen added a commit to kaijchen/ratis that referenced this pull request Jul 11, 2022
captainzmc added a commit to captainzmc/ratis that referenced this pull request Aug 15, 2022
codings-dan added a commit to codings-dan/incubator-ratis that referenced this pull request Aug 16, 2022
codings-dan added a commit to codings-dan/incubator-ratis that referenced this pull request Aug 16, 2022
codings-dan added a commit to codings-dan/incubator-ratis that referenced this pull request Aug 17, 2022
codings-dan added a commit that referenced this pull request Aug 17, 2022
* Revert "RATIS-1601. Use the shaded dropwizard metrics and remove the dependency (#671)"

This reverts commit c41a60b.

* Revert "RATIS-1391. Update library dropwizard.metrics version to 4.x (#632)"

This reverts commit dbb4ffa

Co-authored-by: dragonyliu <dragonyliu@tencent.com>
chrisdutz added a commit to chrisdutz/ratis that referenced this pull request Sep 21, 2023
symious pushed a commit to symious/ratis that referenced this pull request Mar 5, 2024
RexXiong pushed a commit to apache/celeborn that referenced this pull request May 30, 2024
### What changes were proposed in this pull request?

Bump Ratis version from 2.5.1 to 3.0.1. Address incompatible changes:

- RATIS-589. Eliminate buffer copying in SegmentedRaftLogOutputStream.(apache/ratis#964)
- RATIS-1677. Do not auto format RaftStorage in RECOVER.(apache/ratis#718)
- RATIS-1710. Refactor metrics api and implementation to separated modules. (apache/ratis#749)

### Why are the changes needed?

Bump Ratis version from 2.5.1 to 3.0.1. Ratis has released v3.0.0, v3.0.1, which release note refers to [3.0.0](https://ratis.apache.org/post/3.0.0.html), [3.0.1](https://ratis.apache.org/post/3.0.1.html). The 3.0.x version include new features like pluggable metrics and lease read, etc, some improvements and bugfixes including:

- 3.0.0: Change list of ratis 3.0.0 In total, there are roughly 100 commits diffing from 2.5.1 including:
   - Incompatible Changes
      - RaftStorage Auto-Format
      - RATIS-1677. Do not auto format RaftStorage in RECOVER. (apache/ratis#718)
      - RATIS-1694. Fix the compatibility issue of RATIS-1677. (apache/ratis#731)
      - RATIS-1871. Auto format RaftStorage when there is only one directory configured. (apache/ratis#903)
      - Pluggable Ratis-Metrics (RATIS-1688)
      - RATIS-1689. Remove the use of the thirdparty Gauge. (apache/ratis#728)
      - RATIS-1692. Remove the use of the thirdparty Counter. (apache/ratis#732)
      - RATIS-1693. Remove the use of the thirdparty Timer. (apache/ratis#734)
      - RATIS-1703. Move MetricsReporting and JvmMetrics to impl. (apache/ratis#741)
      - RATIS-1704. Fix SuppressWarnings(“VisibilityModifier”) in RatisMetrics. (apache/ratis#742)
      - RATIS-1710. Refactor metrics api and implementation to separated modules. (apache/ratis#749)
      - RATIS-1712. Add a dropwizard 3 implementation of ratis-metrics-api. (apache/ratis#751)
      - RATIS-1391. Update library dropwizard.metrics version to 4.x (apache/ratis#632)
      - RATIS-1601. Use the shaded dropwizard metrics and remove the dependency (apache/ratis#671)
      - Streaming Protocol Change
      - RATIS-1569. Move the asyncRpcApi.sendForward(..) call to the client side. (apache/ratis#635)
   - New Features
      - Leader Lease (RATIS-1864)
      - RATIS-1865. Add leader lease bound ratio configuration (apache/ratis#897)
      - RATIS-1866. Maintain leader lease after AppendEntries (apache/ratis#898)
      - RATIS-1894. Implement ReadOnly based on leader lease (apache/ratis#925)
      - RATIS-1882. Support read-after-write consistency (apache/ratis#913)
      - StateMachine API
      - RATIS-1874. Add notifyLeaderReady function in IStateMachine (apache/ratis#906)
      - RATIS-1897. Make TransactionContext available in DataApi.write(..). (apache/ratis#930)
      - New Configuration Properties
      - RATIS-1862. Add the parameter whether to take Snapshot when stopping to adapt to different services (apache/ratis#896)
      - RATIS-1930. Add a conf for enable/disable majority-add. (apache/ratis#961)
      - RATIS-1918. Introduces parameters that separately control the shutdown of RaftServerProxy by JVMPauseMonitor. (apache/ratis#950)
      - RATIS-1636. Support re-config ratis properties (apache/ratis#800)
      - RATIS-1860. Add ratis-shell cmd to generate a new raft-meta.conf. (apache/ratis#901)
   - Improvements & Bug Fixes
      - Netty
         - RATIS-1898. Netty should use EpollEventLoopGroup by default (apache/ratis#931)
         - RATIS-1899. Use EpollEventLoopGroup for Netty Proxies (apache/ratis#932)
         - RATIS-1921. Shared worker group in WorkerGroupGetter should be closed. (apache/ratis#955)
         - RATIS-1923. Netty: atomic operations require side-effect-free functions. (apache/ratis#956)
      - RaftServer
         - RATIS-1924. Increase the default of raft.server.log.segment.size.max. (apache/ratis#957)
         - RATIS-1892. Unify the lifetime of the RaftServerProxy thread pool (apache/ratis#923)
         - RATIS-1889. NoSuchMethodError: RaftServerMetricsImpl.addNumPendingRequestsGauge apache/ratis#922 (apache/ratis#922)
         - RATIS-761. Handle writeStateMachineData failure in leader. (apache/ratis#927)
         - RATIS-1902. The snapshot index is set incorrectly in InstallSnapshotReplyProto. (apache/ratis#933)
         - RATIS-1912. Fix infinity election when perform membership change. (apache/ratis#954)
         - RATIS-1858. Follower keeps logging first election timeout. (apache/ratis#894)

- 3.0.1:This is a bugfix release. See the [changes between 3.0.0 and 3.0.1](apache/ratis@ratis-3.0.0...ratis-3.0.1) releases.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Cluster manual test.

Closes #2480 from SteNicholas/CELEBORN-1400.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants