Behat tests with Github Actions#476
Conversation
📝 WalkthroughWalkthroughThis PR enables multi-version Behat testing with Docker containers in GitHub Actions. A new ChangesMulti-version Behat Testing Infrastructure
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## v2.x #476 +/- ##
=========================================
Coverage 98.72% 98.72%
Complexity 783 783
=========================================
Files 29 29
Lines 2275 2275
=========================================
Hits 2246 2246
Misses 29 29 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Art4
force-pushed
the
behat-tests-in-github-actions
branch
from
a43fc32 to
4c8ec52
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
.github/workflows/tests.yml (1)
96-103: ⚡ Quick winRun all Behat suites even if one fails.
With separate steps, the first failing suite prevents later suites from running. Consider aggregating exit codes in a single step so all suites execute and the job still fails at the end.
🔧 Proposed refactor
- - name: Run behat [Redmine 6.1] - run: vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_6_1 - - - name: Run behat [Redmine 6.0] - run: vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_6_0 - - - name: Run behat [Redmine 5.1] - run: vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_5_1 + - name: Run behat suites + run: | + status=0 + vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_6_1 || status=$? + vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_6_0 || status=$? + vendor/bin/behat --config tests/Behat/behat.yml --profile=github-actions --format=progress --suite=redmine_5_1 || status=$? + exit $status🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/tests.yml around lines 96 - 103, The three separate GitHub Actions steps running Behat (suites redmine_6_1, redmine_6_0, redmine_5_1) should be combined into a single step that runs all suites and aggregates their exit codes so later suites still execute; update the step that currently runs vendor/bin/behat --suite=redmine_6_1, --suite=redmine_6_0 and --suite=redmine_5_1 to invoke each suite sequentially, capture each command's exit status, and at the end exit with a non-zero code if any of the captured statuses are non-zero (so the job fails if any suite failed but all suites run).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/tests.yml:
- Around line 40-42: The behat job inherits broad GITHUB_TOKEN permissions;
restrict it by adding a permissions block under the job named "behat" that
grants least privilege (e.g., at minimum set contents: read and any other
specific read-only scopes your tests require) so the job no longer inherits wide
default permissions from the workflow root; update the job definition in the
.github/workflows/tests.yml for the behat job to include this permissions block.
- Around line 51-65: The workflow uses floating tags for actions
(actions/checkout@v4, shivammathur/setup-php@v2, ramsey/composer-install@v2) and
the checkout step lacks persist-credentials: false; fix by replacing each action
tag with its corresponding full 40-character commit SHA for supply-chain safety
and add persist-credentials: false under the Checkout step (the actions/checkout
usage) to prevent passing repo credentials to later steps; ensure the SHAs are
verified from each action's GitHub repository and update the workflow
accordingly.
---
Nitpick comments:
In @.github/workflows/tests.yml:
- Around line 96-103: The three separate GitHub Actions steps running Behat
(suites redmine_6_1, redmine_6_0, redmine_5_1) should be combined into a single
step that runs all suites and aggregates their exit codes so later suites still
execute; update the step that currently runs vendor/bin/behat
--suite=redmine_6_1, --suite=redmine_6_0 and --suite=redmine_5_1 to invoke each
suite sequentially, capture each command's exit status, and at the end exit with
a non-zero code if any of the captured statuses are non-zero (so the job fails
if any suite failed but all suites run).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c0426ce9-96d1-4653-84a9-be0502d17139
📒 Files selected for processing (5)
.github/workflows/tests.ymltests/Behat/Bootstrap/FeatureContext.phptests/Behat/behat.ymltests/RedmineExtension/BehatHookTracer.phptests/RedmineExtension/RedmineInstance.php
| behat: | ||
| name: Tests (Behat with PHP ${{ matrix.php }}) | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Restrict token scope for the new behat job.
Add a permissions block with least privilege so this job doesn’t inherit broad default GITHUB_TOKEN permissions.
🔧 Proposed fix
behat:
name: Tests (Behat with PHP ${{ matrix.php }})
runs-on: ubuntu-latest
+ permissions:
+ contents: read📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| behat: | |
| name: Tests (Behat with PHP ${{ matrix.php }}) | |
| runs-on: ubuntu-latest | |
| behat: | |
| name: Tests (Behat with PHP ${{ matrix.php }}) | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read |
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 40-107: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block
(excessive-permissions)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/tests.yml around lines 40 - 42, The behat job inherits
broad GITHUB_TOKEN permissions; restrict it by adding a permissions block under
the job named "behat" that grants least privilege (e.g., at minimum set
contents: read and any other specific read-only scopes your tests require) so
the job no longer inherits wide default permissions from the workflow root;
update the job definition in the .github/workflows/tests.yml for the behat job
to include this permissions block.
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 2 | ||
|
|
||
| - name: Setup PHP, with composer and extensions | ||
| uses: shivammathur/setup-php@v2 | ||
| with: | ||
| php-version: ${{ matrix.php }} | ||
| tools: phpunit | ||
| extensions: mbstring, xml, ctype, iconv, intl, pdo_sqlite, curl | ||
| coverage: xdebug | ||
|
|
||
| - name: Install Composer dependencies | ||
| uses: ramsey/composer-install@v2 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file=".github/workflows/tests.yml"
echo "Potentially unpinned uses: references (not full 40-char SHA):"
rg -n '^\s*uses:\s*[^@]+@(?![0-9a-fA-F]{40}\b).+$' "$file" || true
echo
echo "Checkout steps missing persist-credentials: false (manual check context shown):"
rg -n -A6 -B1 'uses:\s*actions/checkout@' "$file"Repository: kbsali/php-redmine-api
Length of output: 1278
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file=".github/workflows/tests.yml"
echo "=== All 'uses:' references in the file ==="
rg -n 'uses:' "$file"
echo
echo "=== Lines 51-65 (target of review) ==="
sed -n '51,65p' "$file" | cat -n
echo
echo "=== Count of unpinned vs pinned actions ==="
echo "Using version tags (unpinned):"
rg 'uses:.*`@v`[0-9]' "$file" | wc -l
echo "Using full SHAs (pinned):"
rg 'uses:.*@[0-9a-fA-F]{40}' "$file" | wc -lRepository: kbsali/php-redmine-api
Length of output: 1328
🌐 Web query:
GitHub Actions security best practices pin action SHA commit credentials
💡 Result:
Pinning GitHub Actions to a full-length commit SHA is a critical security best practice to prevent supply chain attacks, such as tag hijacking or malicious code injection [1][2][3]. Why Pinning to SHA is Required GitHub Actions referenced by tags (e.g., @v1) or branches (e.g., @main) are mutable, meaning the underlying code can be changed by the action maintainer or an attacker who gains access to the repository [2][3][4]. Pinning to a full-length (40-character) commit SHA ensures that your workflow always executes the exact, immutable version of the code you have verified [2][3][5]. This is currently the only way to use an action as an immutable release [6][7][3]. How to Pin Actions To pin an action, replace the tag or branch reference with the full-length commit SHA of the desired version [2][3]. Example: uses: actions/checkout@v4 # Vulnerable to tag manipulation uses: actions/checkout@11bd719 # Secure, pinned to SHA Best Practices for Maintenance 1. Use Version Comments: Always include the original version tag in a comment next to the SHA to make it easier to identify which version is being used and to facilitate manual updates [1][8][3]. 2. Automate Updates: Because SHA pinning requires manual updates to receive bug fixes and security patches, use tools like Dependabot or Renovate to automatically propose updates to your pinned SHAs [1][8][3][5]. 3. Verify Source: Ensure the SHA originates from the official action repository and not a fork [9][6][3]. 4. Enforce via Policy: Organizations can enforce this practice at the repository or organization level by enabling the "Require actions to be pinned to a full-length commit SHA" policy in GitHub Actions settings [6][10][11]. When enabled, any workflow using a non-SHA reference will fail [10][4][11]. 5. Use Security Auditing Tools: Tools like zizmor can automatically detect unpinned action references in your workflows [8][12][13]. Regarding Credentials While SHA pinning protects the integrity of the action code itself, it does not inherently protect against credential theft if the action is malicious or compromised [14]. Always follow the principle of least privilege by: - Setting default GITHUB_TOKEN permissions to read-only [15][7]. - Using OpenID Connect (OIDC) to access cloud resources instead of storing long-term secrets in GitHub [16]. - Auditing the permissions and network access required by third-party actions [15][17].
Citations:
- 1: Pin all GitHub Actions to immutable SHA commits for supply chain security github/gh-aw#5691
- 2: https://www.stepsecurity.io/blog/pinning-github-actions-for-enhanced-security-a-complete-guide
- 3: https://github.com/ihs7/action-sha-pinned-actions
- 4: https://www.romainlespinasse.dev/posts/github-actions-commit-sha-pinning/
- 5: https://www.codewrecks.com/post/github/github-sha-pinning/
- 6: https://docs.github.com/en/actions/reference/security/secure-use
- 7: https://docs.github.com/en/actions/reference/secure-use-reference
- 8: Pin GitHub Actions to commit SHAs in release workflow github/gh-aw#10021
- 9: https://stackoverflow.com/questions/78903499/how-do-i-pin-an-action-to-a-specific-sha
- 10: https://github.blog/changelog/2025-08-15-github-actions-policy-now-supports-blocking-and-sha-pinning-actions/
- 11: https://docs.github.com/en/enterprise-cloud@latest/admin/enforcing-policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-github-actions-in-your-enterprise
- 12: zizmorcore/zizmor@d9a98ea
- 13: unpinned-uses: don't auto fix non-version refs zizmorcore/zizmor#1860
- 14: https://gbhackers.com/compromised-github-action/
- 15: https://www.aikido.dev/blog/checklist-github-actions
- 16: https://www.stepsecurity.io/blog/github-actions-security-best-practices
- 17: https://dev.to/gitguardian/the-future-of-github-actions-security-and-what-you-can-do-right-now-32nj
🌐 Web query:
GitHub checkout action persist-credentials false security best practice
💡 Result:
Setting persist-credentials: false in the actions/checkout GitHub Action is considered a security best practice to minimize the risk of credential exposure [1][2]. By default, actions/checkout configures the local Git environment with the authentication token (such as the GITHUB_TOKEN) used to clone the repository [3][4]. This makes the token available to any subsequent steps or scripts running within the same job, as they can read it from the Git configuration [5][2]. If a workflow step is compromised or if the repository is inadvertently archived and uploaded as an artifact, this token could be exposed [6][2]. To mitigate this risk, you should explicitly disable this behavior when the token is not required for subsequent Git operations [1]: - uses: actions/checkout@v4 with: persist-credentials: false While recent updates to actions/checkout have improved security by moving these credentials to a separate file in $RUNNER_TEMP rather than directly in.git/config [3][7], setting persist-credentials: false remains the recommended approach to follow the principle of least privilege and prevent unnecessary credential persistence [1][2]. If your workflow does not need to perform authenticated Git operations (like git push or git submodule updates) after the initial checkout, there is no reason to keep the credentials active [1].
Citations:
- 1: https://cheatsheetseries.owasp.org/cheatsheets/GitHub_Actions_Security_Cheat_Sheet.html
- 2: https://yossarian.net/til/post/actions-checkout-can-leak-github-credentials/
- 3: https://github.com/actions/checkout/blob/main/README.md
- 4: https://github.com/actions/checkout
- 5: Remove
persist-credentialsor change the default tofalseactions/checkout#485 - 6: [security] Escalate concerning default
persist-credentials=trueplease actions/checkout#2312 - 7: Persist creds to a separate file actions/checkout#2286
Pin action references to full commit SHAs and add persist-credentials: false to checkout.
All actions in this workflow use version tags (@v4, @v2, etc.) instead of full 40-character commit SHAs, creating supply-chain attack risk. Additionally, the checkout step should set persist-credentials: false to avoid unnecessarily exposing credentials to subsequent steps.
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 51-54: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 52-52: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 57-57: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 65-65: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/tests.yml around lines 51 - 65, The workflow uses floating
tags for actions (actions/checkout@v4, shivammathur/setup-php@v2,
ramsey/composer-install@v2) and the checkout step lacks persist-credentials:
false; fix by replacing each action tag with its corresponding full 40-character
commit SHA for supply-chain safety and add persist-credentials: false under the
Checkout step (the actions/checkout usage) to prevent passing repo credentials
to later steps; ensure the SHAs are verified from each action's GitHub
repository and update the workflow accordingly.
1 participant
Another try to run the behat tests with Github Actions.
Summary by CodeRabbit