Skip to content

Conversation

@nicoloboschi
Copy link
Contributor

@nicoloboschi nicoloboschi commented Jul 7, 2022

Motivation

Currently JDBC sink will perform an insert query for most of the cases. The only way to execute an UPDATE or DELETE is to pass a specific property in the message properties. This is not handy and requires to all the sources to add this specific option.

The goal is to provide a way to upsert and delete records based on the record content while using key-value records.

  1. Upsert: Insert or update the record based on the key fields.
  2. Delete: if the value is NULL, delete the record for the given row.

Modifications

  • new config option insertMode with values
    • INSERT: only perform blindly insert (default, in order to not break upgrades)
    • UPDATE: only perform blindly update
    • UPSERT: use upsert query. Main issue is that there's no SQL standard so every sink has his own implementation (by default it throws a not supported exception)
  • new config option nullValueAction with values
    • FAIL: update the row with null values (default, in order to not break upgrades)
    • DELETE: delete the row with the the given key/pk

Note that in order to not break compatibility the message property ACTION has the precedence on other logics.

For the upsert mode, only the following sinks support it:

  • SqlLite
  • Postgres
  • MariaDB

Also upgraded SqlLite to 3.36.0.3 to support upsert

others will throw an exception and the message will be rejected

  • doc

@github-actions
Copy link

github-actions bot commented Jul 7, 2022

@nicoloboschi Please provide a correct documentation label for your PR.
Instructions see Pulsar Documentation Label Guide.

@nicoloboschi nicoloboschi added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. and removed doc-label-missing labels Jul 7, 2022
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work !

@nicoloboschi nicoloboschi reopened this Jul 12, 2022
@nicoloboschi nicoloboschi force-pushed the jdbc-sink-support-update-delete branch from cc7b7cd to 392bcc6 Compare July 12, 2022 20:34
@nicoloboschi nicoloboschi force-pushed the jdbc-sink-support-update-delete branch from 392bcc6 to 8faa374 Compare July 13, 2022 08:06
@nicoloboschi nicoloboschi merged commit ac41cfe into apache:master Jul 14, 2022
@nicoloboschi nicoloboschi added this to the 2.11.0 milestone Jul 14, 2022
nicoloboschi added a commit to datastax/pulsar that referenced this pull request Jul 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connector doc Your PR contains doc changes, no matter whether the changes are in markdown or code files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants