smoke test allow pass for flaky providers

[zanesq]

Summary

Goose created this or we could just remove the experimentals for now.

Investigation Results

I investigated the flaky smoke-tests-code-exec job using the GitHub CLI and found:

1. Root Cause: Two models have inconsistent tool-calling behavior:

   • google:gemini-3-pro-preview - Most frequent offender (~80% of failures). Sometimes returns empty responses without making any tool calls.
   • openrouter:nvidia/nemotron-3-nano-30b-a3b - Occasional failures with similar behavior.

2. Pattern: When these models fail, they return nothing within ~5 seconds. When they succeed, they take ~45 seconds and properly call tools. This is typical of preview/experimental models.

3. Timeline:

   • gemini-3-pro-preview added Nov 19, 2025
   • nvidia/nemotron-3-nano-30b-a3b added Dec 31, 2025

Fix Applied

I modified scripts/test_providers.sh to add an "allowed failures" mechanism:

1. Added an ALLOWED_FAILURES array listing the flaky models
2. Added an is_allowed_failure() function to check if a model is in the list
3. Modified the test logic to:
   • Mark flaky model failures with ⚠ FLAKY instead of ✗ FAILED
   • Track "hard failures" separately from allowed failures
   • Only exit with error code 1 if there are hard failures
   • Show a clear message when all required tests pass but some flaky tests failed

Expected Behavior After Fix

• If gemini-3-pro-preview fails: Test shows ⚠ google: gemini-3-pro-preview (flaky) and the job passes
• If a non-flaky model fails: Test shows ✗ provider: model and the job fails
• Summary clearly shows which tests were flaky vs hard failures

This approach:

• ✅ Keeps testing the flaky models (we still see if they pass/fail)
• ✅ Doesn't block PRs due to known flaky preview models
• ✅ Still fails on real regressions in stable models
• ✅ Provides clear visibility into flaky test status

[zanesq]

From conversation in Discord maybe we close this and look into the gemini issue since that seems concerning (-preview is latest) or maybe we add retries for flaky providers. Note its definitely flakiness because they all passed on this PR.

[michaelneale]

is ok for now - and will have a follow up to chase these

[rabi]

From conversation in Discord maybe we close this and look into the gemini issue since that seems concerning (-preview is latest)

Hi Zane! If the gemini3 failures are with code_execution, I've #6555 that would possibly fix the issue, as I don't see empty response issues with it locally.