[fix] [ml] Fix the incorrect total size if use ML interceptor

[poorbarcode]

Motivation

1. Issue

totalSize of managed ledger is less than 0

---

2. Analizy

Before writing to BK, OpAddEntry records the size of the data to dataLength, but if the data is changed by ManagedLedgerInterceptor.processPayloadBeforeLedgerWrite, the field dataLength of OpAddEntry is not changed, which will cause the field totalSize and currentLedgerSize of managed ledger to be incorrect.

For example:

1. write data [1], before write BK record the data size by OpAddEntry.dataLength.
   1-1. now OpAddEntry.dataLength is 1.
2. reset data of OpAddEntry to [1,2,3...21] by ManagedLedgerInterceptor.processPayloadBeforeLedgerWrite.
   2-1. now the actual size of data is 21, but OpAddEntry.dataLength still is 1.
3. do write BK and switch the ledger
   3-1. Create ledger info and put it into ML.ledgers, the size of ledger info is 21.
4. trim ledgers
   4-1. ML.totalSize -= ledgerInfo.size, then the totalSize of ML is 1-21 = -20

---

3.Reproduce

You can reproduce this issue by test testTotalSizeCorrectIfHasInterceptor and testCurrentLedgerSizeCorrectIfHasInterceptor

---

4. This issue comes with the new features by PR #12536

---

Modifications

Correct variables OpAddEntry.dataLength and ML.currentLedgerSize after executing ManagedLedgerInterceptor.processPayloadBeforeLedgerWrite

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

• [ci] [fix] [ml] Fix the incorrect total size if use ML interceptor poorbarcode/pulsar#62

[mattisonchao]

LGTM,
But I think that there are better ways to go than subtracting before adding. Can we move the currentLedgerSize calculation logic after the interceptor?

[codelipenghui]

Is it better to just reset the dataLength if the payload processor presents?

if (payloadProcessorHandle != null) {
    duplicateBuffer = payloadProcessorHandle.getProcessedPayload();
    dataLength = duplicateBuffer.readableBytes();
}

[poorbarcode]

Already fixed

[poorbarcode]

@mattisonchao

Can we move the currentLedgerSize calculation logic after the interceptor?

This will affect currentLedgerIsFull's checking and cause the switching ledger to delay one entry, which I'm not sure is appropriate

[codecov-commenter]

Codecov Report

Merging #19404 (2dec903) into master (39dd1cd) will increase coverage by 30.33%.
The diff coverage is 47.05%.

@@              Coverage Diff              @@
##             master   #19404       +/-   ##
=============================================
+ Coverage     32.36%   62.69%   +30.33%     
+ Complexity     6355     3472     -2883     
=============================================
  Files          1648     1826      +178     
  Lines        123990   133747     +9757     
  Branches      13523    14726     +1203     
=============================================
+ Hits          40129    83855    +43726     
+ Misses        77958    42160    -35798     
- Partials       5903     7732     +1829     

Flag	Coverage Δ
inttests	24.79% <47.05%> (-0.02%)	⬇️
systests	25.58% <0.00%> (-0.11%)	⬇️
unittests	59.93% <47.05%> (+42.39%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...org/apache/bookkeeper/mledger/impl/OpAddEntry.java	67.52% <25.00%> (+13.31%)	⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java	53.90% <53.84%> (+16.79%)	⬆️
...r/service/AbstractDispatcherMultipleConsumers.java	66.12% <0.00%> (-14.52%)	⬇️
...ker/loadbalance/impl/LeastLongTermMessageRate.java	73.33% <0.00%> (-11.12%)	⬇️
...r/service/schema/DefaultSchemaRegistryService.java	0.00% <0.00%> (-6.25%)	⬇️
.../apache/pulsar/client/impl/ClientCnxIdleState.java	68.88% <0.00%> (-2.23%)	⬇️
...in/java/org/apache/pulsar/common/api/AuthData.java	71.42% <0.00%> (ø)
...ava/org/apache/pulsar/client/api/schema/Field.java	80.00% <0.00%> (ø)
...g/apache/pulsar/common/naming/NamespaceBundle.java	60.71% <0.00%> (ø)
.../apache/pulsar/broker/namespace/LookupOptions.java	87.50% <0.00%> (ø)
... and 1215 more

[yaalsn]

In branch-2.10 and branch-2.11, it causes a compilation failure. @poorbarcode PTAL

pulsar/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java:[141,32] currentLedgerSize has private access in org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl

[yaalsn]

@codelipenghui @mattisonchao Please help to review and merge the above two PR otherwise it will affect 2.10 and 2.11 compilation.