SECURITY-51: Add GHA workflows by Xiangs18 · Pull Request #225 · kbase/index_runner

Xiangs18 · 2025-03-26T23:07:27Z

No description provided.

+    uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
+    with:
+      name: '${{ github.event.repository.name }}-develop'
+      tags: br-${{ github.ref_name }}
+    secrets: inherit


+    if: github.base_ref == 'develop' && github.event.pull_request.merged == false
+    uses: kbase/.github/.github/workflows/reusable_build.yml@main
+    secrets: inherit
+  build-develop-merge:


+    if: github.base_ref == 'develop' && github.event.pull_request.merged == true
+    uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
+    with:
+      name: '${{ github.event.repository.name }}-develop'
+      tags: pr-${{ github.event.number }},latest
+    secrets: inherit
+  build-main-open:


+    if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == false
+    uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
+    with:
+      name: '${{ github.event.repository.name }}'
+      tags: pr-${{ github.event.number }}
+    secrets: inherit
+  build-main-merge:


+    if: (github.base_ref == 'main' || github.base_ref == 'master') && github.event.pull_request.merged == true
+    uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
+    with:
+      name: '${{ github.event.repository.name }}'
+      tags: pr-${{ github.event.number }},latest-rc
+    secrets: inherit
+  trivy-scans:


+    uses: kbase/.github/.github/workflows/reusable_validate-branch.yml@main
+    with:
+      build_branch: '${{ github.event.release.target_commitish }}'
+  validate-release-tag:


+    needs: check-source-branch
+    uses: kbase/.github/.github/workflows/reusable_validate-release-tag.yml@main
+    with:
+      release_tag: '${{ github.event.release.tag_name }}'
+  build-push:


+    needs: validate-release-tag
+    uses: kbase/.github/.github/workflows/reusable_build-push.yml@main
+    with:
+      name: '${{ github.event.repository.name }}'
+      tags: '${{ github.event.release.tag_name }},latest'
+    secrets: inherit


Copilot

Pull Request Overview

This PR introduces several GitHub Actions workflows to support testing, releasing, pull request builds, manual builds, dependency updates, and code scanning for security improvements. Key changes include:

Addition of a multi-version Python test workflow (.github/workflows/test.yml)
Setup of release, pull request, and manual build workflows
Inclusion of Dependabot and CodeQL configurations for improved security and dependency management

Reviewed Changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
.github/workflows/test.yml	Adds a workflow for testing and building docker images
.github/workflows/release-main.yml	Implements a release workflow for building and pushing images
.github/workflows/pr_build.yml	Configures workflows for pull request builds and tag/push mechanisms
.github/workflows/manual-build.yml	Introduces a manual build and push workflow
.github/dependabot.yml	Configures monthly dependency update checks via Dependabot
.github/codeql.yml	Sets up CodeQL scanning for code security analysis

Files not reviewed (2)

.github/workflows/test-push-release.yml: Language not supported
scripts/run_unit_tests: Language not supported

Copilot · 2025-03-27T15:38:37Z

+name: Run tests and build docker iamges
+


There is a typo in 'iamges'. Please correct it to 'images' so that the workflow name is clear.

Suggested change

name: Run tests and build docker iamges

name: Run tests and build docker images

codecov · 2025-04-03T23:14:57Z

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

github-advanced-security · 2025-04-04T01:24:01Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

+    runs-on: ubuntu-latest
+
+    env:
+      WORKSPACE_TOKEN: ${{ secrets.WORKSPACE_TOKEN }}
+      RE_API_TOKEN: ${{ secrets.RE_API_TOKEN }}
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: 3.9.19
+
+    - name: Pip installation
+      run: python -m pip install --upgrade pip poetry
+
+    - name: Poetry installation
+      run: poetry install
+
+    - name: Run unit tests
+      run: scripts/run_unit_tests
+
+    - name: Run integration tests
+      run: |
+        docker compose up
+        scripts/run_integration_tests
+
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v5
+      with:
+        token: ${{ secrets.CODECOV_TOKEN }}
+        fail_ci_if_error: true


trigger build

…o dev-add_workflows

MrCreosote · 2025-04-14T23:31:57Z

I'm assuming again all the workflows are the standard boilerplate, LMK if that's not the case

yes, standard boilerplate

MrCreosote · 2025-04-14T23:37:39Z

-RUN apk --update add librdkafka librdkafka-dev && ldconfig /usr/lib
+RUN apk --update add build-base git cmake bash
+RUN git clone --branch v2.6.1 https://github.com/confluentinc/librdkafka.git /tmp/librdkafka
+RUN cd /tmp/librdkafka && \
+    ./configure && \
+    make && \
+    make install && \
+    ldconfig /usr/lib


Yikes, why do we need to build from source?

Because RUN apk --update add librdkafka only gives you librdkafka 2.4, which is not compatible with confluent-kafka 2.6.1.

Ok, so we might be able to install from a repo if we upgrade the docker image in the future

MrCreosote · 2025-04-14T23:43:01Z

+confluent-kafka = "2.6.1"

-[tool.poetry.dev-dependencies]
+[tool.poetry.group.dev.dependencies]


I don't know anything about poetry so I'm not sure what this change means

new formats for specifying dev dependencies in your pyproject.toml

MrCreosote · 2025-04-14T23:43:50Z

+    pip install --upgrade pip poetry==2.1.2 && \
    poetry config virtualenvs.create false && \
-    poetry install --no-dev --no-interaction --no-ansi && \
+    poetry install --no-root --without dev --no-interaction --no-ansi && \


Same deal as above re poetry

In Poetry ≥1.2, --no-dev is deprecated and replaced with dependency groups like --with and --without.

Line 42 basically installs only the main dependencies, skipping dev dependencies.

MrCreosote

LGTM, just needs @bio-boris's signoff

add gha workflows

21e0449

github-advanced-security AI found potential problems Mar 26, 2025

View reviewed changes

bio-boris requested a review from Copilot March 27, 2025 15:38

Copilot AI reviewed Mar 27, 2025

View reviewed changes

update python version

9a2f604

github-advanced-security AI found potential problems Mar 27, 2025

View reviewed changes

Comment thread .github/workflows/test.yml Fixed

simplify test.yml script

3b1ff3b

github-advanced-security AI found potential problems Mar 27, 2025

View reviewed changes

Comment thread .github/workflows/test.yml Fixed

Xiangs18 added 2 commits March 27, 2025 12:54

update test.yml

2b84efe

remove ref

0103fa2

github-advanced-security AI found potential problems Mar 27, 2025

View reviewed changes

Comment thread .github/workflows/test.yml Fixed

Xiangs18 added 7 commits April 2, 2025 10:27

update python version and confluent-kafka version

90c9f6e

update poetry version and remove libressl-dev

cbdfd3e

fix '--no-dev' option does not exist

f0fca9e

add missing librdkafka-dev dep

e423ea2

revert dep update

b8933dc

add pytest-cov dep

3b279f3

add .coveragerc file

8b59409

fix docker image

8706835

Xiangs18 changed the title ~~[WIP] SECURITY-51: Add GHA workflows~~ SECURITY-51: Add GHA workflows Apr 4, 2025

add run integration tests step

6ad9ec8

github-advanced-security AI found potential problems Apr 10, 2025

View reviewed changes

Xiangs18 added 6 commits April 10, 2025 12:44

start the services in the background

87e676d

switch to docker compose from docker-compose

e7a4f9d

fix docker-compose

a8e1652

run docker compose up -d in GHA

40eda2b

remove -d to produce verbose logs

32a8d40

reproduce logs

077cc78

Xiangs18 and others added 13 commits April 10, 2025 14:32

update docker-compose

8bad2af

rerun integration tests

7768ca8

add long timeout

79e7ab5

add timeout to integration tests

c5b707d

add a blank line

3553db5

retry with 1800s timeout

706e611

add display docker compose logs step

93410bf

add if failure()

6bdf92a

remove the logging: {driver: none}

e2963dd

Update README.md

8f0edaa

trigger build

use env vars in CI

6f1af73

Merge branch 'dev-add_workflows' of github.com:kbase/index_runner int…

277a70a

…o dev-add_workflows

revert docker-compose.yaml file && add comments

a39a4ab

bio-boris requested changes Apr 14, 2025

View reviewed changes

Comment thread docker-compose.yaml

Comment thread pyproject.toml Outdated

Comment thread .github/workflows/test.yml Outdated

fix docker-compose file obsolete version && pyproject.toml file

e075f63

MrCreosote reviewed Apr 14, 2025

View reviewed changes

Xiangs18 requested review from MrCreosote and bio-boris April 14, 2025 23:56

MrCreosote reviewed Apr 14, 2025

View reviewed changes

bio-boris approved these changes Apr 15, 2025

View reviewed changes

Xiangs18 merged commit e4f0360 into develop Apr 16, 2025
12 checks passed

Xiangs18 deleted the dev-add_workflows branch April 16, 2025 00:02

	name: Run tests and build docker iamges
	name: Run tests and build docker images

Conversation

Xiangs18 commented Mar 26, 2025

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 3, 2025

Welcome to Codecov 🎉

Uh oh!

github-advanced-security AI commented Apr 4, 2025

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xiangs18 Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MrCreosote left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Xiangs18 Apr 14, 2025 •

edited

Loading