[issue 8337][Worker] Move initialize dlog namespace metadata to bin/pulsar #8781

nodece · 2020-12-02T09:01:20Z

Motivation

Currently, the starting function worker service will initialize the distributed log namespace. when initializing the distributed log namespace, the zookeeper will be connected. A better way is to implement a tool to initialize the distributed log namespace, which is similar to ./bin/pulsar initialize-cluster-metadata.

Modifications

Add init distributed log namespace metadata to ./bin/pulsar initialize-cluster-metadata
Add flag to control whether the distributed log namespace is initialized in Function worker
Add flag to conf/functions_worker.yml

Doc

If you want to initialize distributed log metadata by bin/pulsar, you need to perform the following steps:

Using ./bin/pulsar initialize-cluster-metadata to initialize cluster
Set initializedDlogMetadata to true in functions_worker.yml

Note: All the changes are compatible.

Verifying this change

Make sure that the change passes the CI checks.

sijie

Overall the change looks good to me. I left two comments. PTAL.

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java

pulsar-broker/src/main/java/org/apache/pulsar/PulsarInitialDlogNamespaceMetadataSetup.java

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java

tuteng · 2020-12-22T09:27:21Z

ping @nodece

nodece · 2021-01-04T08:16:11Z

@tuteng @sijie could you review the PR again?

tuteng · 2021-01-04T08:20:16Z

We may need to consider adding some documentation on how to use this command @jennifer88huang

Jennifer88huang-zz · 2021-01-04T09:54:41Z

@tuteng thank you for your reminder.
@nodece you can add related doc content in this PR or in a separate PR, or you can also ask a writer to add it. If you need any support, feel free to let us know. Thank you.

eolivelli

+1

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece · 2021-01-05T10:33:16Z

@tuteng @jennifer88huang thanks, I did update at #8781 (comment), but I'm not sure what to add document to the official website.

nodece · 2021-01-05T11:40:16Z

@tuteng @jennifer88huang once pr is merged, I will make a new PR for this doc.

tuteng · 2021-01-05T12:48:24Z

/pulsarbot run-failure-checks

Jennifer88huang-zz · 2021-01-05T15:23:58Z

@tuteng @jennifer88huang once pr is merged, I will make a new PR for this doc.

That's great, thank you.

Jennifer88huang-zz · 2021-01-05T15:24:17Z

/pulsarbot run-failure-checks

tuteng · 2021-01-07T09:06:07Z

/pulsarbot run-failure-checks

…space

tuteng · 2021-01-09T08:07:10Z

/pulsarbot run-failure-checks

tuteng · 2021-01-11T09:18:15Z

/pulsarbot run-failure-checks

nodece · 2021-01-12T07:23:13Z

@tuteng @sijie tests has beed passed. could you merge this to master?

nodece · 2021-01-12T09:02:21Z

@jennifer88huang I made a PR for document: #9185.

codelipenghui · 2021-01-14T03:10:34Z

Hi @tuteng I noticed this PR is based on #8560, #8560 has some big changes so it will not contains in the 2.7.x release. Do we need to cherry-pick this PR to branch-2.7?

tuteng · 2021-01-14T03:17:08Z

@codelipenghui Thanks, move this pr to 2.8.0, delete label release/2.7.1

sijia-w · 2021-03-10T10:05:19Z

@nodece Just a soft reminder. We still need to add doc for the update in this pr.

@tuteng @jennifer88huang once pr is merged, I will make a new PR for this doc.

Anonymitaet · 2021-03-15T00:05:10Z

@sijia-w does the author already add docs in #9185?

sijia-w · 2021-03-15T07:16:27Z

@Anonymitaet It seems so. Thanks for reminding me.

nodece mentioned this pull request Dec 2, 2020

ISSUE-8337: Create a tool to initialize dlog namespace streamnative/pulsar-archived#1591

Closed

sijie requested changes Dec 2, 2020

View reviewed changes

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java Outdated Show resolved Hide resolved

pulsar-broker/src/main/java/org/apache/pulsar/PulsarInitialDlogNamespaceMetadataSetup.java Outdated Show resolved Hide resolved

sijie assigned nodece Dec 2, 2020

sijie added the area/broker label Dec 2, 2020

sijie added this to the 2.8.0 milestone Dec 2, 2020

sijie added area/cli type/feature The PR added a new feature or issue requested a new feature and removed area/broker labels Dec 2, 2020

zymap approved these changes Dec 2, 2020

View reviewed changes

nodece force-pushed the move_init_dlog_namespace branch from 6423e6a to 2b1a216 Compare December 2, 2020 10:11

nodece requested a review from sijie December 2, 2020 10:23

nodece force-pushed the move_init_dlog_namespace branch 5 times, most recently from dcc634c to 4f5d8d5 Compare December 7, 2020 07:07

sijie reviewed Dec 9, 2020

View reviewed changes

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java Outdated Show resolved Hide resolved

pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java Outdated Show resolved Hide resolved

nodece force-pushed the move_init_dlog_namespace branch from 4f5d8d5 to 20bb187 Compare January 4, 2021 08:15

nodece requested a review from sijie January 4, 2021 08:16

tuteng added the doc-required Your PR changes impact docs and you will update later. label Jan 4, 2021

eolivelli approved these changes Jan 4, 2021

View reviewed changes

Add initialize dlog namespace metadata to bin/pulsar

c341217

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece force-pushed the move_init_dlog_namespace branch from 20bb187 to c341217 Compare January 5, 2021 10:16

Merge remote-tracking branch 'apache/master' into move_init_dlog_name…

544f61e

…space

sijie approved these changes Jan 12, 2021

View reviewed changes

sijie merged commit aca4cd7 into apache:master Jan 12, 2021

tuteng added the release/2.7.1 label Jan 12, 2021

sijie mentioned this pull request Jan 12, 2021

[docs] Add initializedDlogMetadata config to functions-worker #9185

Merged

tuteng removed the release/2.7.1 label Jan 14, 2021

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Apr 9, 2021

[issue 8337][Worker] Move initialize dlog namespace metadata to bin/pulsar #8781

[issue 8337][Worker] Move initialize dlog namespace metadata to bin/pulsar #8781

Uh oh!

Conversation

nodece commented Dec 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Doc

Verifying this change

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tuteng commented Dec 22, 2020

Uh oh!

nodece commented Jan 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tuteng commented Jan 4, 2021

Uh oh!

Jennifer88huang-zz commented Jan 4, 2021

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

nodece commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodece commented Jan 5, 2021

Uh oh!

tuteng commented Jan 5, 2021

Uh oh!

Jennifer88huang-zz commented Jan 5, 2021

Uh oh!

Jennifer88huang-zz commented Jan 5, 2021

Uh oh!

tuteng commented Jan 7, 2021

Uh oh!

tuteng commented Jan 9, 2021

Uh oh!

tuteng commented Jan 11, 2021

Uh oh!

nodece commented Jan 12, 2021

Uh oh!

nodece commented Jan 12, 2021

Uh oh!

codelipenghui commented Jan 14, 2021

Uh oh!

tuteng commented Jan 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sijia-w commented Mar 10, 2021

Uh oh!

Anonymitaet commented Mar 15, 2021

Uh oh!

sijia-w commented Mar 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

nodece commented Dec 2, 2020 •

edited

Loading

nodece commented Jan 4, 2021 •

edited

Loading

nodece commented Jan 5, 2021 •

edited

Loading

tuteng commented Jan 14, 2021 •

edited

Loading