Debezium Oracle Source

[dlg99]

Motivation

Add Debezium Source for Oracle.

Modifications

Added Debezium Source for Oracle DB.
Added integration test.
Oracle JDBC is not included into nar (licensing issue) so the end user will have to package it.
Dropping jars into the pulsar/lib directory is not enough (connector classpath doesn't get it) so for the integration test I ended up updating the nar file with jar uf.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added integration test

$ $ mvn test -f tests/pom.xml -DintegrationTestSuiteFile=pulsar-io-ora-source.xml -DintegrationTests -Dtest=org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest  -DtestRetryCount=0
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 445.938 s - in org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Pulsar :: Tests 2.9.0-SNAPSHOT:
[INFO]
[INFO] Apache Pulsar :: Tests ............................. SUCCESS [  1.948 s]
[INFO] Apache Pulsar :: Tests :: Integration .............. SUCCESS [07:34 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:36 min

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): no
• The public API: no
• The schema: no
• The default values of configurations: no
• The wire protocol: no
• The rest endpoints: no
• The admin cli options: no
• Anything that affects deployment: no

Documentation

For contributor

For this PR, do we need to update docs?

Need to add description of the new connector; TBD (or will create an issue later).
Need to document requirement to download and package Oracle JDBC driver.

[Anonymitaet]

@dlg99 many thanks for providing doc info!
When adding docs, you can take the following guides as references:

• https://pulsar.apache.org/docs/en/next/io-connectors/#debezium-mysql
• https://hub.streamnative.io/connectors/sqs-source/v2.8.0.5/

[sijie]

before reviewing this, is there any license issues we should clear out?

[merlimat]

This is depending on Oracle JDBC driver https://mvnrepository.com/artifact/com.oracle.ojdbc/ojdbc8/19.3.0.0 that is licensed with "Oracle Free Use Terms and Conditions (FUTC)". It's not clear if this license is compatible with ASLv2.

[dlg99]

@sijie @merlimat I removed dependency on Oracle JDBC and updated tests/docker-images/latest-version-image/Dockerfile to download ojdbc jars for the integration test. Users will have to deal with this.
According to https://issues.apache.org/jira/browse/LEGAL-526 FUTC is a no-go for Apache projects.

[eolivelli]

What about creating one dedicated CI job only for Oracle CDC ?

[eolivelli]

great work
I left some comments about hardcoding 2.9.0-SNAPSHOT, PTAL

[eolivelli]

LGTM
great work !

[Anonymitaet]

Thanks for your great contribution.
Want users to know your awesome code changes? Do not forget to add docs accordingly! And you can ping me to review the docs. Let's grow Pulsar together!

[merlimat]

👍

[eolivelli]

@sijie are you okay with merging this PR now ? it looks like licensing issues have been addressed.

[dlg99]

ping @sijie

[lhotari]

@eolivelli @dlg99 The new integration test seems to be flaky. Here's one sample failure:
https://github.com/apache/pulsar/pull/12027/checks?check_run_id=3591596786#step:13:27471

Error:  testDebeziumOracleDbSource(org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest)  Time elapsed: 1,800.028 s  <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource() didn't finish within the time-out 1800000

[Anonymitaet]

Hi @dlg99 for the doc side, I think we already documented "the requirement to download and package Oracle JDBC driver" here(https://pulsar.apache.org/docs/en/next/io-debezium-source/#example-of-oracle)?

[dlg99]

@Anonymitaet correct, see #12213 for the docs