[fix][client] Fix ConfigurationDataUtils loadConf strategy #13298

mgrenonville · 2021-12-14T14:37:29Z

Motivation

When using loadConf, ConfigurationDataUtils was loading existing values
in a Map using Jackson mapper. If there is a @JsonIgnore, Jackson
ignores theses fields, but there will be forgotten in the new instance.

Modifications

Using readerForUpdating(existingData).readValue(configJson) will
refresh only overriden values in config map.

Verifying this change

Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as org.apache.pulsar.client.impl.conf.ConfigurationDataUtilsTest.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: no (at least, it will no longer fails silently on @JsonIgnore properties)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

no-need-doc
This was a bug in loadConf code, it is intended to work as follow (I think)
doc-not-needed

codelipenghui · 2021-12-15T07:40:41Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConfigurationDataUtils.java


    public static <T> T loadData(Map<String, Object> config,
                                 T existingData,
                                 Class<T> dataCls) {


The dataCls becomes useless after the new change, we should remove it.

Thanks for review !
You're right, I've done the update.

Fixes apache#11646 When using `loadConf`, ConfigurationDataUtils was loading existing values in a Map using Jackson mapper. If there is a `@JsonIgnore`, Jackson ignores theses fields, but there will be forgotten in the new instance. Using `readerForUpdating(existingData).readValue(configJson)` will refresh only overriden values in config map. Since it's using existing object to write values, `dataCls` param becomes useless, thus removed.

mgrenonville · 2022-01-05T20:43:38Z

Hi @codelipenghui !
It's strange, I see that CI test are failing however, when executing these tests on my computer, they work. Are they known to be flaky ?

codelipenghui · 2022-01-07T05:09:58Z

@mgrenonville You can left a comment with content /pulsarbot run-failure-checks to trigger to rerun the failed tests.

gaoran10 · 2022-01-09T16:57:33Z

Maybe we could add a construct method for DeadLetterPolicy, then we could use the conf to set DeadLetterPolicy.

codelipenghui · 2022-01-18T05:51:51Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java


    private RegexSubscriptionMode regexSubscriptionMode = RegexSubscriptionMode.PersistentOnly;

+    @JsonIgnore


Will this introduce breaking change?

I think it's not needed anymore since #16487

michaeljmarshall

I am requesting changes because I think this is a breaking change. If that is incorrect, please let me know.

michaeljmarshall · 2022-02-11T21:58:26Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConfigurationDataUtils.java

    public static <T> T loadData(Map<String, Object> config,
-                                 T existingData,
-                                 Class<T> dataCls) {
+                                 T existingData) {


Unless I'm mistaken, I believe this is part of the public API and could be used by user applications to build the ClientConfigurationData. We would need to find a way to introduce this without breaking the API.

+1. We can keep the old method and just mark deprecated.

yes, I think it could have been used by users.
we should keep the old, make it work the same way as before and use the new method

Jason918 · 2022-02-12T03:38:50Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConfigurationDataUtils.java

    public static <T> T loadData(Map<String, Object> config,
-                                 T existingData,
-                                 Class<T> dataCls) {
+                                 T existingData) {


+1. We can keep the old method and just mark deprecated.

Jason918 · 2022-02-12T03:41:05Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ConsumerConfigurationData.java


    private boolean resetIncludeHead = false;

+    @JsonIgnore


Can you add some details about why we add @JsonIgnore here?

KeySharedPolicy is not deserializable:

@Test public void test() { Map<String, Object> config = new HashMap<>(); config.put("keySharedPolicy", KeySharedPolicy.autoSplitHashRange()); consumerBuilderImpl.loadConf(config); }

fails with com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of org.apache.pulsar.client.api.KeySharedPolicy

github-actions · 2022-03-19T01:56:40Z

The pr had no activity for 30 days, mark with Stale label.

github-actions · 2022-05-30T02:15:26Z

The pr had no activity for 30 days, mark with Stale label.

github-actions bot assigned mgrenonville Dec 14, 2021

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Dec 14, 2021

codelipenghui reviewed Dec 15, 2021

View reviewed changes

codelipenghui added this to the 2.10.0 milestone Dec 15, 2021

codelipenghui added release/2.8.3 release/2.9.2 labels Dec 15, 2021

mgrenonville force-pushed the fix_load_conf branch from 68ff2b1 to 6a6b30e Compare December 16, 2021 09:26

mgrenonville requested a review from codelipenghui December 20, 2021 14:39

codelipenghui approved these changes Dec 23, 2021

View reviewed changes

codelipenghui modified the milestones: 2.10.0, 2.11.0 Jan 18, 2022

codelipenghui added release/2.9.3 and removed release/2.9.2 labels Jan 18, 2022

codelipenghui reviewed Jan 18, 2022

View reviewed changes

michaeljmarshall requested changes Feb 11, 2022

View reviewed changes

michaeljmarshall added release/2.8.4 and removed release/2.8.3 labels Feb 11, 2022

Jason918 reviewed Feb 12, 2022

View reviewed changes

codelipenghui added the release/2.10.1 label Feb 16, 2022

github-actions bot added the lifecycle/stale label Mar 19, 2022

codelipenghui added release/2.10.2 and removed release/2.10.1 labels Jun 9, 2022

mattisonchao added release/2.9.4 and removed release/2.9.3 labels Jun 9, 2022

Technoboy- modified the milestones: 3.1.0, 3.2.0 Jul 31, 2023

poorbarcode added release/3.0.3 and removed release/3.0.2 labels Oct 24, 2023

Technoboy- modified the milestones: 3.2.0, 3.3.0 Dec 22, 2023

coderzc modified the milestones: 3.3.0, 3.4.0 May 8, 2024

lhotari added release/3.0.8 and removed release/3.0.3 labels Oct 10, 2024

lhotari modified the milestones: 4.0.0, 4.1.0 Oct 14, 2024

lhotari added release/3.0.9 and removed release/3.0.8 labels Dec 6, 2024

lhotari added release/3.0.10 and removed release/3.0.9 labels Jan 3, 2025

lhotari added release/3.0.11 and removed release/3.0.10 labels Feb 19, 2025

lhotari added release/3.0.12 and removed release/3.0.11 labels Mar 31, 2025

lhotari added release/3.0.13 and removed release/3.0.12 labels May 20, 2025

lhotari added release/3.0.14 and removed release/3.0.13 labels Jul 30, 2025

coderzc modified the milestones: 4.1.0, 4.2.0 Sep 1, 2025

lhotari added release/3.0.15 and removed release/3.0.14 labels Sep 27, 2025

lhotari added release/3.0.16 and removed release/3.0.15 labels Nov 11, 2025


		private RegexSubscriptionMode regexSubscriptionMode = RegexSubscriptionMode.PersistentOnly;

		@JsonIgnore

[fix][client] Fix ConfigurationDataUtils loadConf strategy #13298

Are you sure you want to change the base?

[fix][client] Fix ConfigurationDataUtils loadConf strategy #13298

Uh oh!

Conversation

mgrenonville commented Dec 14, 2021 • edited by tisonkun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mgrenonville commented Jan 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codelipenghui commented Jan 7, 2022

Uh oh!

gaoran10 commented Jan 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 19, 2022

Uh oh!

github-actions bot commented May 30, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

17 participants

mgrenonville commented Dec 14, 2021 •

edited by tisonkun

Loading

mgrenonville commented Jan 5, 2022 •

edited

Loading

gaoran10 commented Jan 9, 2022 •

edited

Loading