Add Kafka Connect artifact publish to release process#15212
Add Kafka Connect artifact publish to release process#15212rmoff wants to merge 4 commits intoapache:mainfrom
Conversation
|
Adding @jbonofre as a reviewer since the zip distribution is a new published artifact. I believe we already went through these for LICENSE/NOTICE, but I would love a double check on it. |
| Thanks to everyone for contributing! | ||
| ``` | ||
|
|
||
| #### Confluent Marketplace Submission |
There was a problem hiding this comment.
i dont think its a good idea to put the onus on the release manager to email confluent-hub
There was a problem hiding this comment.
asf also has guideline on where to direct users to the source distribution.
https://apache.org/legal/release-policy#publication
There was a problem hiding this comment.
@kevinjqliu this is not a official ASF publication here. The required by the ASF is to, at minimum, distribute the source distribution on dist.apache.org (used for download.apache.org).
dist can also be used to publish convenient artifacts.
The rest (dockerhub, pypi, etc) is out of the ASF scope, but it has to be ASF compliant from legal standpoint (LICENSE, NOTICE, etc).
There was a problem hiding this comment.
I'm fine to ask the release manager to email Confluent for publication as soon as the kafka-connect-runtime has been reviewed and voted as part of an official release.
According to line 316, we are "publishing" on Confluent a distribution that has been voted (as part of an official release).
There was a problem hiding this comment.
So, I took a deeper look on this.
I think we are fine here: according to line 316(in this file), we are publishing (on Confluent Marketplace) a kafka-runtime distribution (main) that has been reviewed/voted as part of a release. So it means that the "legal checks" should have been done as part of the release.
So, we are taking the kafka connect runtime distribution from Maven Central to "cross" publish on Confluent Marketplace.
However, we have a prerequisite here: we never published the kafka-connect-runtime distribution up to now. So, it means that we have to include kafka-connect-runtime distribution as part of the release process (stage on dist.apache.org for review, and stage on repository.apache.org).
It's not directly related to this PR, but it's a prerequisite.
|
I will take a look. Thanks ! As it's published artifact, it has to be "legal/foundation compliant". FYI, the kafka connect distribution is not fully correct today (see #15449). It's unrelated to this PR, but it has to be fixed before release including kafka-connect sink distribution. |
| Thanks to everyone for contributing! | ||
| ``` | ||
|
|
||
| #### Confluent Marketplace Submission |
There was a problem hiding this comment.
I'm fine to ask the release manager to email Confluent for publication as soon as the kafka-connect-runtime has been reviewed and voted as part of an official release.
According to line 316, we are "publishing" on Confluent a distribution that has been voted (as part of an official release).
| Thanks to everyone for contributing! | ||
| ``` | ||
|
|
||
| #### Confluent Marketplace Submission |
There was a problem hiding this comment.
So, I took a deeper look on this.
I think we are fine here: according to line 316(in this file), we are publishing (on Confluent Marketplace) a kafka-runtime distribution (main) that has been reviewed/voted as part of a release. So it means that the "legal checks" should have been done as part of the release.
So, we are taking the kafka connect runtime distribution from Maven Central to "cross" publish on Confluent Marketplace.
However, we have a prerequisite here: we never published the kafka-connect-runtime distribution up to now. So, it means that we have to include kafka-connect-runtime distribution as part of the release process (stage on dist.apache.org for review, and stage on repository.apache.org).
It's not directly related to this PR, but it's a prerequisite.
| 1. [Contact the Confluent Hub team](mailto:confluent-hub@confluent.io) with the Maven Central URL for the distribution zip: | ||
|
|
||
| ``` | ||
| https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-kafka-connect-runtime/<VERSION>/iceberg-kafka-connect-runtime-<VERSION>.zip |
There was a problem hiding this comment.
For the record, in kafka-connect/build.gradle (line 238-240), we can see:
// there are no Maven artifacts so disable publishing tasks...
project.afterEvaluate {
project.tasks.matching { it.group == 'publishing' }.each {it.enabled = false}
}
We can give link to Maven Central, but we can also completely publish the kafka-connect-runtime distribution on dist.apache.org (stage and then release), and give access via downloads.apache.org.
That's completely a viable option.
rmoff
force-pushed
the
confluent-hub-release-process
branch
from
b162f6e to
758cfff
Compare
|
@jbonofre @danielcweeks @kevinjqliu IIUC, this PR is now in a position to move forward. Or at least, I don't see any further unresolved discussion or open questions :) Thanks. |
The processResources filter for __VERSION__ replacement was applied to all resource files, corrupting the binary PNG. Scope it to manifest.json which is the only file that needs the substitution.
rmoff
force-pushed
the
confluent-hub-release-process
branch
from
758cfff to
1527202
Compare
5 participants
Background
There is currently no prebuilt Iceberg connector for Kafka Connect provided with Iceberg. This can make it more difficult than necessary for users who just want to build pipelines, not run gradle :)
Confluent Marketplace (previously "Confluent Hub") is a long-established platform for the distribution of connectors for Kafka Connect, both open-source and proprietary.
At the moment the 1.9.2 version of the connector has been uploaded to the site by Confluent staff as an ad-hoc process.
There are several existing issues from the community regarding: #10745 #14986 #12516
This PR
(replaces #15113)
This PR does two things: