[feat][broker] PIP-180 Part VI: Add ShadowManagedLedgerImpl by Jason918 · Pull Request #18265 · apache/pulsar

Jason918 · 2022-10-31T08:07:07Z

Master Issue: #16153

Motivation

After this PR, the basic function of PIP-180 shadow topic will be ready.

Modifications

Add ShadowManagedLedgerImpl

Verifying this change

Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

see ShadowTopicTest.java

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: Jason918#12

Jason918 · 2022-11-15T02:35:10Z

/pulsarbot run-failure-checks

Jason918 · 2022-11-15T02:36:17Z

@codelipenghui @eolivelli @michaeljmarshall @lhotari @mattisonchao PTAL

codecov-commenter · 2022-11-15T03:19:27Z

Codecov Report

Attention: Patch coverage is 15.62500% with 270 lines in your changes missing coverage. Please review.

Project coverage is 47.06%. Comparing base (b579bb8) to head (536f10a).
Report is 2591 commits behind head on master.

Files with missing lines	Patch %	Lines
...okkeeper/mledger/impl/ShadowManagedLedgerImpl.java	0.00%	199 Missing ⚠️
.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java	28.57%	17 Missing and 3 partials ⚠️
...ava/org/apache/pulsar/broker/service/Producer.java	27.27%	8 Missing and 8 partials ⚠️
...org/apache/bookkeeper/mledger/impl/OpAddEntry.java	26.31%	9 Missing and 5 partials ⚠️
...rg/apache/pulsar/broker/service/BrokerService.java	20.00%	6 Missing and 2 partials ⚠️
...sar/broker/service/persistent/PersistentTopic.java	20.00%	3 Missing and 1 partial ⚠️
...va/org/apache/pulsar/client/impl/ProducerImpl.java	0.00%	4 Missing ⚠️
...apache/bookkeeper/mledger/ManagedLedgerConfig.java	0.00%	2 Missing ⚠️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	90.47%	1 Missing and 1 partial ⚠️
...va/org/apache/pulsar/broker/service/ServerCnx.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18265      +/-   ##
============================================
- Coverage     47.51%   47.06%   -0.46%     
+ Complexity    10520    10457      -63     
============================================
  Files           698      698              
  Lines         68079    68323     +244     
  Branches       7280     7332      +52     
============================================
- Hits          32351    32157     -194     
- Misses        32151    32565     +414     
- Partials       3577     3601      +24

Flag	Coverage Δ
unittests	`47.06% <15.62%> (-0.46%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../org/apache/bookkeeper/mledger/impl/MetaStore.java	`100.00% <ø> (ø)`
...rg/apache/pulsar/broker/service/AbstractTopic.java	`65.06% <100.00%> (-0.21%)`	⬇️
...va/org/apache/pulsar/broker/service/ServerCnx.java	`48.97% <75.00%> (-0.86%)`	⬇️
...apache/bookkeeper/mledger/ManagedLedgerConfig.java	`71.60% <0.00%> (-0.90%)`	⬇️
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	`54.28% <90.47%> (+0.19%)`	⬆️
...sar/broker/service/persistent/PersistentTopic.java	`61.68% <20.00%> (-0.37%)`	⬇️
...va/org/apache/pulsar/client/impl/ProducerImpl.java	`17.00% <0.00%> (-0.02%)`	⬇️
...rg/apache/pulsar/broker/service/BrokerService.java	`57.40% <20.00%> (-0.55%)`	⬇️
...org/apache/bookkeeper/mledger/impl/OpAddEntry.java	`64.21% <26.31%> (-3.57%)`	⬇️
...ava/org/apache/pulsar/broker/service/Producer.java	`61.47% <27.27%> (-3.03%)`	⬇️
... and 2 more

... and 41 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

HQebupt

LGTM

AnonHxy · 2022-11-16T09:16:35Z

LGTM with minor comments

codelipenghui · 2022-11-23T13:56:33Z

        super(factory, bookKeeper, store, config, scheduledExecutor, name, mlOwnershipChecker);
-        this.shadowSource = TopicName.get(config.getShadowSource());
-        this.sourceMLName = shadowSource.getPersistenceNamingEncoding();
+        if (config.getTopicName().isPartitioned() && TopicName.getPartitionIndex(config.getShadowSource()) == -1) {


Why the name is not able to be used? It's a little confusing that we need to introduce the topic name in ManagedLedgerConfig

Or maybe we can introduce a properties?

Why the name is not able to be used? It's a little confusing that we need to introduce the topic name in ManagedLedgerConfig

Good point.

We need to know the TopicName of the source topic in the shadow topic. But the name here is in the format of PersistenceNamingEncoding.

I get that we'd better not introduce the concept of "topic" into managed-ledger. I will move this logic to the "broker" layer. So that the source topic name is in properties and the managedLedgerName of the source topic is stored in the new field ManagedLedgerConfig.shadowSourceName.

codelipenghui · 2022-11-24T01:30:41Z

+                    sourceLedgersStat, stat);
+        }
+
+        sourceLedgersStat = stat;


Do we need to check the data version?
Maybe the watcher gets a notification first, and then we get the response from get managed ledger info operation. Will we add the deleted ledger back?

Nice catch. Added a version check.

codelipenghui · 2022-11-24T01:35:02Z

+        sourceLedgersStat = stat;
+
+        if (mlInfo.hasTerminatedPosition()) {
+            state = State.Terminated;


Do you need Terminated state for a shadow ledger? If the source topic changes to the terminated state, the shadow topic will not get any new messages.

Not needed actually, just keeping the original logic at first.
After another thought, we should remove this, because the source topic can be terminated but the replication to the shadow topic should go on.
So comes these updates:

Removed Terminated state from ShadowManagedLedger.

Overrides asyncTerminate and return fail directly.

codelipenghui · 2022-11-24T01:45:48Z

+        if (state == State.Terminated) {
+            addOperation.failed(new ManagedLedgerException.ManagedLedgerTerminatedException(
+                    "Managed ledger was already terminated"));
+            return;
+        }


Looks like we don't need to check if the topic is terminated.
And if the broker receives the zookeeper notification first, then the shadow topic changes to the terminated state. But the shadow replicator hasn't reached the end of the source topic. In this case, the consumer connected to the shadow topic will have a different view.

Yes, exactly.

codelipenghui · 2022-11-24T01:51:48Z

+            currentLedgerEntries = position.getEntryId();
+            currentLedgerSize += addOperation.data.readableBytes();
+            addOperation.initiateShadowWrite();
+        }


How about the position's ledger ID that is not equal to the current ledger ID? It looks like the addOperation will never be complete.

Fixed, and add ut in ShadowManagedLedgerImplTest

Jason918 · 2022-11-25T15:13:36Z

@codelipenghui Comments all addressed. PTAL.

- finish processSourceManagedLedgerInfo - finish internalAsyncAddEntry - ShadowTopicTest pass - ShadowReplicatorTest pass Add test cases

…8265)

momo-jun · 2023-01-18T09:24:28Z

Hi @Jason918, do you have any updates for documenting this feature?
Since 2.11 has been released, now it's good timing to start the doc update in the next version.

Jason918 · 2023-02-17T02:26:16Z

Hi @Jason918, do you have any updates for documenting this feature?
Since 2.11 has been released, now it's good timing to start the doc update in the next version.

@momo-jun Thanks for the reminding, I will try to come up with the doc update before the 3.0.0 release.

StevenLuMT · 2023-02-17T02:42:29Z

Hi @Jason918, do you have any updates for documenting this feature?
Since 2.11 has been released, now it's good timing to start the doc update in the next version.

@momo-jun Thanks for the reminding, I will try to come up with the doc update before the 3.0.0 release.

@momo-jun I have communicated with @Jason918 , I'll help him update the documentation

momo-jun · 2023-02-17T02:57:48Z

Thanks @Jason918 @StevenLuMT. Feel free to ping me if you need any assistance with docs.

StevenLuMT · 2023-02-17T03:42:13Z

Thanks @Jason918 @StevenLuMT. Feel free to ping me if you need any assistance with docs.

Thanks @momo-jun ,my WeChat ID is 863199780, add me, keep fast communication

github-actions Bot added the doc-required Your PR changes impact docs and you will update later. label Oct 31, 2022

Jason918 self-assigned this Oct 31, 2022

Jason918 added type/feature The PR added a new feature or issue requested a new feature area/broker labels Oct 31, 2022

Jason918 added this to the 2.12.0 milestone Oct 31, 2022

Jason918 requested review from 315157973, AnonHxy, codelipenghui, eolivelli, hangc0276, mattisonchao, michaeljmarshall and zymap October 31, 2022 08:09

Jason918 mentioned this pull request Oct 31, 2022

[feat][broker] PIP-180 Part VI: Add ShadowManagedLedgerImpl Jason918/pulsar#12

Closed

Jason918 added the ready-to-test label Nov 6, 2022

AnonHxy reviewed Nov 7, 2022

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Producer.java

Jason918 requested review from AnonHxy, HQebupt and lhotari November 15, 2022 02:36

HQebupt approved these changes Nov 16, 2022

View reviewed changes

AnonHxy reviewed Nov 16, 2022

View reviewed changes

AnonHxy approved these changes Nov 16, 2022

View reviewed changes

Jason918 force-pushed the pip-180-dev branch from 9295d45 to eb300b8 Compare November 22, 2022 08:32

codelipenghui reviewed Nov 24, 2022

View reviewed changes

PIP-180 Part VI, Add ShadowManagedLedgerImpl

5d5189a

- finish processSourceManagedLedgerInfo - finish internalAsyncAddEntry - ShadowTopicTest pass - ShadowReplicatorTest pass Add test cases

Jason918 added 10 commits November 28, 2022 14:55

fix test

07d9649

fix testCompactionRetentionOnTopicCreationWithTopicPolicies

1f570e1

fix testBlockIfQueueFullWhenChunking

cb1fc40

address comments

f31e1c0

move topic name related logic in managed-ledger to broker layer.

9023336

Add version check when process mlInfo from sourceML

1dd5c7b

Removed Terminated state from ShadowManagedLedger.

48ebdeb

Add ShadowManagedLedgerImplTest

b9a2916

fix header

d11e502

fix compile

536f10a

Jason918 force-pushed the pip-180-dev branch from 69b8222 to 536f10a Compare November 28, 2022 06:56

codelipenghui approved these changes Nov 28, 2022

View reviewed changes

Jason918 merged commit 5b062f3 into apache:master Nov 29, 2022

lifepuzzlefun pushed a commit to lifepuzzlefun/pulsar that referenced this pull request Dec 9, 2022

[feat][broker] PIP-180 Part VI: Add ShadowManagedLedgerImpl (apache#1…

8cae91b

…8265)

lifepuzzlefun pushed a commit to lifepuzzlefun/pulsar that referenced this pull request Jan 10, 2023

[feat][broker] PIP-180 Part VI: Add ShadowManagedLedgerImpl (apache#1…

4f7cacc

…8265)

michaeljmarshall mentioned this pull request Feb 16, 2023

[cleanup][broker] Cleanup ManagedLedgerImpl's nouse method: isLedgersReadonly #19513

Merged

15 tasks

Jason918 deleted the pip-180-dev branch February 17, 2023 02:26

RobertIndie mentioned this pull request Apr 11, 2023

PIP-180: Shadow Topic, an alternative way to support readonly topic ownership. #16153

Open

Conversation

Jason918 commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

Uh oh!

Jason918 commented Nov 15, 2022

Uh oh!

Jason918 commented Nov 15, 2022

Uh oh!

codecov-commenter commented Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

HQebupt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AnonHxy commented Nov 16, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jason918 commented Nov 25, 2022

Uh oh!

momo-jun commented Jan 18, 2023

Uh oh!

Jason918 commented Feb 17, 2023

Uh oh!

StevenLuMT commented Feb 17, 2023

Uh oh!

momo-jun commented Feb 17, 2023

Uh oh!

StevenLuMT commented Feb 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Jason918 commented Oct 31, 2022 •

edited

Loading

codecov-commenter commented Nov 15, 2022 •

edited

Loading