Skip to content

Add Kafka Connect artifact publish to release process #15212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rmoff wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add Kafka Connect artifact publish to release process #15212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d2875cf
Add Kafka Connect artifact publish to release process
rmoff Jan 30, 2026
1226943
Rename file
rmoff Feb 9, 2026
6e9c350
Fix processResources corrupting iceberg.png binary
rmoff Feb 11, 2026
1527202
Remove optional note
rmoff Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 19 additions & 6 deletions kafka-connect/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,10 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
}

processResources {
filter {
it.replace('__VERSION__', project.version.toString())
filesMatching('manifest.json') {
filter {
it.replace('__VERSION__', project.version.toString())
}
Comment thread
rmoff marked this conversation as resolved.
}
}

Expand All @@ -212,6 +214,10 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
into('doc/') {
from "$projectDir/main/LICENSE"
from "$projectDir/main/NOTICE"
from "$projectDir/main/README.md"
}
into('etc/') {
from "$projectDir/main/etc"
}
into('assets/') {
from "${processResources.destinationDir}/iceberg.png"
Expand All @@ -235,10 +241,6 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
}
}

// there are no Maven artifacts so disable publishing tasks...
project.afterEvaluate {
project.tasks.matching { it.group == 'publishing' }.each {it.enabled = false}
}

tasks.jar.enabled = false

Expand All @@ -256,6 +258,17 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
assemble.dependsOn distZip, hiveDistZip

apply from: "${rootDir}/runtime-deps.gradle"

// Configure publishing for the distribution zip
afterEvaluate {
publishing {
publications {
apache(MavenPublication) {
artifact distZip
}
}
}
}
}

project(':iceberg-kafka-connect:iceberg-kafka-connect-transforms') {
Expand Down
26 changes: 26 additions & 0 deletions kafka-connect/kafka-connect-runtime/main/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-->

# Apache Iceberg Sink Connector

A Kafka Connect sink connector for writing data from Apache Kafka into Apache Iceberg tables.

Documentation: https://iceberg.apache.org/docs/latest/kafka-connect/

License: Apache License 2.0
27 changes: 27 additions & 0 deletions kafka-connect/kafka-connect-runtime/main/etc/iceberg-sink.properties.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Licensed to the Apache Software Foundation (ASF) under one
Comment thread
rmoff marked this conversation as resolved.
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Iceberg Sink Connector - Example Configuration
# https://iceberg.apache.org/docs/latest/kafka-connect/
name=iceberg-sink
connector.class=org.apache.iceberg.connect.IcebergSinkConnector
tasks.max=1
topics=events
iceberg.tables=db.table_name
iceberg.catalog.type=rest
iceberg.catalog.uri=https://your-catalog-uri
iceberg.catalog.warehouse=your-warehouse
2 changes: 1 addition & 1 deletion kafka-connect/kafka-connect-runtime/src/main/resources/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"logo": "assets/iceberg.png",
"type": "organization",
"url": "https://iceberg.apache.org",
"username": "iceberg"
"username": "apache"
Comment thread
rmoff marked this conversation as resolved.
},

"support": {
Expand Down
21 changes: 21 additions & 0 deletions site/docs/how-to-release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,11 @@ To build and publish the convenience binaries, run the `dev/stage-binaries.sh` s
dev/stage-binaries.sh
```

!!! Note
The staged binaries include the Kafka Connect runtime distribution
(`iceberg-kafka-connect-runtime-<VERSION>.zip`), which bundles the connector
with all required dependencies for Kafka Connect plugin installation.

Next, you need to close the staging repository:

1. Go to [Nexus](https://repository.apache.org/) and log in
Expand Down Expand Up @@ -297,6 +302,22 @@ Java artifacts are available from Maven Central.
Thanks to everyone for contributing!
```

#### Confluent Marketplace Submission
Copy link
Copy Markdown
Contributor

@kevinjqliu kevinjqliu Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont think its a good idea to put the onus on the release manager to email confluent-hub

Copy link
Copy Markdown
Contributor

@kevinjqliu kevinjqliu Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asf also has guideline on where to direct users to the source distribution.
https://apache.org/legal/release-policy#publication

Copy link
Copy Markdown
Member

@jbonofre jbonofre Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevinjqliu this is not a official ASF publication here. The required by the ASF is to, at minimum, distribute the source distribution on dist.apache.org (used for download.apache.org).

dist can also be used to publish convenient artifacts.

The rest (dockerhub, pypi, etc) is out of the ASF scope, but it has to be ASF compliant from legal standpoint (LICENSE, NOTICE, etc).

Copy link
Copy Markdown
Member

@jbonofre jbonofre Mar 25, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine to ask the release manager to email Confluent for publication as soon as the kafka-connect-runtime has been reviewed and voted as part of an official release.

According to line 316, we are "publishing" on Confluent a distribution that has been voted (as part of an official release).

Copy link
Copy Markdown
Member

@jbonofre jbonofre Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I took a deeper look on this.

I think we are fine here: according to line 316(in this file), we are publishing (on Confluent Marketplace) a kafka-runtime distribution (main) that has been reviewed/voted as part of a release. So it means that the "legal checks" should have been done as part of the release.

So, we are taking the kafka connect runtime distribution from Maven Central to "cross" publish on Confluent Marketplace.

However, we have a prerequisite here: we never published the kafka-connect-runtime distribution up to now. So, it means that we have to include kafka-connect-runtime distribution as part of the release process (stage on dist.apache.org for review, and stage on repository.apache.org).
It's not directly related to this PR, but it's a prerequisite.


After the release is published to Maven Central, the Kafka Connect runtime distribution
can be [submitted to](https://docs.confluent.io/platform/current/connect/confluent-hub/contributing.html)
[Confluent Marketplace](https://www.confluent.io/hub/) for broader distribution.

To submit to Confluent Hub:

1. [Contact the Confluent Hub team](mailto:confluent-hub@confluent.io) with the Maven Central URL for the distribution zip:

```
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-kafka-connect-runtime/<VERSION>/iceberg-kafka-connect-runtime-<VERSION>.zip
Copy link
Copy Markdown
Member

@jbonofre jbonofre Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, in kafka-connect/build.gradle (line 238-240), we can see:

  // there are no Maven artifacts so disable publishing tasks...                                                                                                                                                                                                               
  project.afterEvaluate {                                                                                                                                                                                                                                                      
    project.tasks.matching { it.group == 'publishing' }.each {it.enabled = false}                                                                                                                                                                                              
  }

We can give link to Maven Central, but we can also completely publish the kafka-connect-runtime distribution on dist.apache.org (stage and then release), and give access via downloads.apache.org.
That's completely a viable option.

```

2. Confluent will publish to the Marketplace.

#### Update revapi

Create a PR in the `iceberg` repo to make revapi run on the new release. For an example see [this PR](https://github.com/apache/iceberg/pull/6275).
Expand Down
Loading