Skip to content

Codecov improvements #3156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
webknjaz merged 7 commits into from
Dec 18, 2024
Merged

Codecov improvements #3156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
17af03b
remove erronously configured codecov flags
jakkdl Dec 17, 2024
02a8406
add .codecov.yml
jakkdl Dec 17, 2024
d418dc2
move token to yaml file, set fail_ci_if_error
jakkdl Dec 17, 2024
decfb73
restore flags, make alpine python flag dynamic
jakkdl Dec 17, 2024
2fd7fcf
skip codecov if check-formatting is set
jakkdl Dec 17, 2024
a7a2ff9
Apply suggestions from code review
jakkdl Dec 17, 2024
2c53ff2
Merge branch 'main' into codecov_stuff
jakkdl Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .codecov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# -- repository yaml --

# Explicitly wait for all jobs to finish, as wait_for_ci prematurely triggers.
# See https://github.com/python-trio/trio/issues/2689
codecov:
notify:
# This number needs to be changed whenever the number of runs in CI is changed.
# Another option is codecov-cli: https://github.com/codecov/codecov-cli#send-notifications
after_n_builds: 27
wait_for_ci: false
notify_error: true # if uploads fail, replace cov comment with a comment with errors.
require_ci_to_pass: false

# Publicly exposing the token has some small risks from mistakes or malicious actors.
# See https://docs.codecov.com/docs/codecov-tokens for correctly configuring it.
token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46

# only post PR comment if coverage changes
comment:
require_changes: true

coverage:
# required range
range: 99.6..100
status:
# require patches to be 100%
patch:
default:
target: 100%
20 changes: 14 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,12 @@ jobs:
uses: codecov/codecov-action@v3
with:
directory: empty
token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
name: Windows (${{ matrix.python }}, ${{ matrix.arch }}${{ matrix.extra_name }})
# multiple flags is marked as an error in codecov UI, but is actually fine
# https://github.com/codecov/feedback/issues/567
flags: Windows,${{ matrix.python }}
Copy link
Member

@webknjaz webknjaz Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd keep the flags despite the UI alerts.

Copy link
Member Author

@jakkdl jakkdl Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we could keep them, but nobody has pressed the button to enable analytics so currently it's entirely unused anyway https://app.codecov.io/gh/python-trio/trio/flags

I still find it somewhat questionable to track coverage for "python3.11" or "windows" as separate entities. Maybe you could diagnose issues by seeing big swings in coverage for one of them*, or any of them being very low, but the actual coverage % for each flag is mostly useless unless we'd do something like marking all platform/version-specific code somehow, exclude from coverage for that flag, and require 100% coverage on common code on all platforms... (which all sounds like way too much effort for close-to-no payoff).

* could use https://docs.codecov.com/docs/commit-status#threshold

Copy link
Contributor

@A5rocks A5rocks Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've used the flags once (when looking at updating the codecov action) but I don't remember how + given we haven't figured the issue out I assume it wasn't useful.

Copy link
Member Author

@jakkdl jakkdl Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess 1>0 and there's not really any downside to have them, so I'll add them back in. I'll go enable analytics too for fun.

Copy link
Member

@webknjaz webknjaz Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use them heavily in many projects. It's nice to see which likes are covered under which env or a combo of those. There's a way to filter it on the Codecov UI.

As for the checks and 100% coverage requirement, I usually don't make those depend on the flags, except for MyPy, perhaps. Codecov is able to track the combined coverage for those checks. I can look into improving this later.

Copy link
Member Author

@jakkdl jakkdl Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit of a pain that https://app.codecov.io/gh/python-trio/trio/flags shows all flags ever used, with 1/3 being for old versions we no longer support. I'm not sure there's a decent way to fix that without doing manual flag management

Copy link
Member

@webknjaz webknjaz Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you can clean them up. Although, to me, the most useful view with flags is the per-file line coverage pages.

# this option cannot be set in .codecov.yml
fail_ci_if_error: true

Ubuntu:
name: 'Ubuntu (${{ matrix.python }}${{ matrix.extra_name }})'
Expand Down Expand Up @@ -131,13 +134,15 @@ jobs:
env:
CHECK_FORMATTING: '${{ matrix.check_formatting }}'
NO_TEST_REQUIREMENTS: '${{ matrix.no_test_requirements }}'
- if: always()
- if: >-
always()
&& matrix.check_formatting != '1'
uses: codecov/codecov-action@v3
with:
directory: empty
token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
name: Ubuntu (${{ matrix.python }}${{ matrix.extra_name }})
flags: Ubuntu,${{ matrix.python }}
fail_ci_if_error: true

macOS:
name: 'macOS (${{ matrix.python }})'
Expand Down Expand Up @@ -171,9 +176,9 @@ jobs:
uses: codecov/codecov-action@v3
with:
directory: empty
token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
name: macOS (${{ matrix.python }})
flags: macOS,${{ matrix.python }}
fail_ci_if_error: true

# run CI on a musl linux
Alpine:
Expand All @@ -193,13 +198,16 @@ jobs:
run: python -m venv .venv
- name: Run tests
run: source .venv/bin/activate && ./ci.sh
- name: Get Python version for codecov flag
id: get-version
run: echo "version=$(python -V | cut -d' ' -f2 | cut -d'.' -f1,2)" >> "${GITHUB_OUTPUT}"
- if: always()
uses: codecov/codecov-action@v3
with:
directory: empty
token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
name: Alpine
flags: Alpine,3.12
flags: Alpine,${{ steps.get-version.outputs.version }}
fail_ci_if_error: true

Cython:
name: "Cython"
Expand Down