fix(docs): add missing config in debezium mysql source #14590

ericsyh · 2022-03-08T00:25:40Z

Signed-off-by: Eric Shen ericshenyuhao@outlook.com

Motivation

The YAML sample of debezium mysql source missing a config of pulsar.service.url on io-debezium-source

Modifications

Add the missing config of pulsar.service.url

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

doc-required

(If you need help on updating docs, create a doc issue)
no-need-doc

(Please explain why)
doc

(If this PR contains doc changes)

Signed-off-by: Eric Shen <ericshenyuhao@outlook.com>

nodece

LGTM

nodece · 2022-03-08T03:00:44Z

site2/docs/io-cdc-debezium.md

        database.history.pulsar.topic: "history-topic"
        database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"
-
+        pulsar.service.url: "pulsar://127.0.0.1:6650"


I don't find any code to read pulsar.service.url, could you provide this code? The database.history.pulsar.service.url is correct.

Yes, i just checked the code and also not found the pulsar.service.url. In my test i used the sample YAML on https://pulsar.apache.org/docs/en/io-debezium-source/#configuration-1 but functions gave me an ERROR about the missing config of pulsar.service.url then i noticed the following sample

bin/pulsar-admin source localrun \ --archive connectors/pulsar-io-debezium-mysql-2.9.1.nar \ --name debezium-mysql-source --destination-topic-name debezium-mysql-topic \ --tenant public \ --namespace default \ --source-config '{"database.hostname": "localhost","database.port": "3306","database.user": "debezium","database.password": "dbz","database.server.id": "184054","database.server.name": "dbserver1","database.whitelist": "inventory","database.history": "org.apache.pulsar.io.debezium.PulsarDatabaseHistory","database.history.pulsar.topic": "history-topic","database.history.pulsar.service.url": "pulsar://127.0.0.1:6650","key.converter": "org.apache.kafka.connect.json.JsonConverter","value.converter": "org.apache.kafka.connect.json.JsonConverter","pulsar.service.url": "pulsar://127.0.0.1:6650","offset.storage.topic": "offset-topic"}'

Used the config of "pulsar.service.url": "pulsar://127.0.0.1:6650" and i also added it then passed the test.

I will do the test again to verify it.

nodece

I left a comment.

gaoran10 · 2022-03-08T04:08:08Z

Could you provide the error logs?

ericsyh · 2022-03-08T06:03:35Z

@gaoran10 @nodece Pulsar and debezium-mysql uses 2.8.1, the ERROR from functions is like below:

13:59:00.824 [public/default/debezium-mysql-source-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Source open produced uncaught exception:
java.lang.IllegalArgumentException: Pulsar service URL not provided.
        at org.apache.pulsar.io.debezium.DebeziumSource.open(DebeziumSource.java:84) ~[pulsar-io-debezium-core-2.8.1.jar:2.8.1]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:735) [org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:219) [org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:243) [org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
13:59:00.829 [public/default/debezium-mysql-source-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/default/debezium-mysql-source:0] Uncaught exception in Java Instance
java.lang.IllegalArgumentException: Pulsar service URL not provided.
        at org.apache.pulsar.io.debezium.DebeziumSource.open(DebeziumSource.java:84) ~[?:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:735) ~[org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:219) ~[org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:243) [org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]

The mysql.yaml config file is

tenant: "public"
namespace: "default"
name: "debezium-mysql-source"
topicName: "debezium-mysql-topic"
archive: "connectors/pulsar-io-debezium-mysql-2.8.1.nar"
parallelism: 1

configs:
    ## config for mysql, docker image: debezium/example-mysql:0.8
    database.hostname: "x.x.x.x"
    database.port: "3306"
    database.user: "ericsyh"
    database.password: "x.x.x.x.x.x"
    database.server.id: "1840"
    database.server.name: "streamnative"
    database.whitelist: "pulsar"
    database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory"
    database.history.pulsar.topic: "history-topic"
    database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"
    ## KEY_CONVERTER_CLASS_CONFIG, VALUE_CONVERTER_CLASS_CONFIG
    key.converter: "org.apache.kafka.connect.json.JsonConverter"
    value.converter: "org.apache.kafka.connect.json.JsonConverter"

    ## OFFSET_STORAGE_TOPIC_CONFIG
    offset.storage.topic: "offset-topic"

The status for the sources

{
  "numInstances" : 1,
  "numRunning" : 0,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : false,
      "error" : "UNAVAILABLE: io exception",
      "numRestarts" : 0,
      "numReceivedFromSource" : 0,
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "numSourceExceptions" : 0,
      "latestSourceExceptions" : [ ],
      "numWritten" : 0,
      "lastReceivedTime" : 0,
      "workerId" : "standalone"
    }
  } ]
}

gaoran10 · 2022-03-08T10:05:05Z

It seems that there is a new configuration database.history.pulsar.service.url after 2.9.1. Please take a look at the PR #11251. Maybe you need to modify the docs before 2.9.1. Could you try to use the 2.9.1?

ericsyh · 2022-03-09T02:19:46Z

It seems that there is a new configuration database.history.pulsar.service.url after 2.9.1. Please take a look at the PR #11251. Maybe you need to modify the docs before 2.9.1. Could you try to use the 2.9.1?

OK, i will test in on 2.9.1 later. Now seems the problem is debezium connectors docs is not fully accurate, the database.history.pulsar.service.url config only apply for 2.9.x and previously version should still use pulsar.service.url. But previsou docs also change this to use database.history.pulsar.service.url config.

nodece · 2022-03-09T02:31:31Z

We can add a compatible for old configuration.

github-actions · 2022-04-09T02:01:14Z

The pr had no activity for 30 days, mark with Stale label.

github-actions · 2022-05-28T02:09:39Z

The pr had no activity for 30 days, mark with Stale label.

tisonkun · 2022-12-06T13:40:34Z

Closed as no consensus.

fix(docs): add missing config in debezium mysql source

58ef257

Signed-off-by: Eric Shen <ericshenyuhao@outlook.com>

github-actions bot assigned ericsyh Mar 8, 2022

github-actions bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Mar 8, 2022

nodece approved these changes Mar 8, 2022

View reviewed changes

nodece reviewed Mar 8, 2022

View reviewed changes

nodece requested changes Mar 8, 2022

View reviewed changes

gaoran10 added the area/connector label Mar 8, 2022

github-actions bot added the lifecycle/stale label Apr 9, 2022

tisonkun closed this Dec 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(docs): add missing config in debezium mysql source #14590

fix(docs): add missing config in debezium mysql source #14590

Uh oh!

ericsyh commented Mar 8, 2022

Uh oh!

nodece left a comment

Uh oh!

nodece Mar 8, 2022

Uh oh!

ericsyh Mar 8, 2022

Uh oh!

nodece left a comment

Uh oh!

gaoran10 commented Mar 8, 2022

Uh oh!

ericsyh commented Mar 8, 2022

Uh oh!

gaoran10 commented Mar 8, 2022 •

edited

Loading

Uh oh!

ericsyh commented Mar 9, 2022

Uh oh!

nodece commented Mar 9, 2022

Uh oh!

github-actions bot commented Apr 9, 2022

Uh oh!

github-actions bot commented May 28, 2022

Uh oh!

tisonkun commented Dec 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix(docs): add missing config in debezium mysql source #14590

fix(docs): add missing config in debezium mysql source #14590

Uh oh!

Conversation

ericsyh commented Mar 8, 2022

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

nodece left a comment

Choose a reason for hiding this comment

Uh oh!

nodece Mar 8, 2022

Choose a reason for hiding this comment

Uh oh!

ericsyh Mar 8, 2022

Choose a reason for hiding this comment

Uh oh!

nodece left a comment

Choose a reason for hiding this comment

Uh oh!

gaoran10 commented Mar 8, 2022

Uh oh!

ericsyh commented Mar 8, 2022

Uh oh!

gaoran10 commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericsyh commented Mar 9, 2022

Uh oh!

nodece commented Mar 9, 2022

Uh oh!

github-actions bot commented Apr 9, 2022

Uh oh!

github-actions bot commented May 28, 2022

Uh oh!

tisonkun commented Dec 6, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gaoran10 commented Mar 8, 2022 •

edited

Loading