feat: Add Windows Playwright testing to nightly builds

[vjgit96]

Summary

Adds Windows support for Playwright E2E tests in nightly builds to catch Windows-specific regressions early. Tests run in parallel on both Linux and Windows without increasing build time.

Changes

.github/workflows/typescript_test.yml

• Added windows-latest to runner options
• Made Playwright installation OS-aware (Windows doesn't support --with-deps flag)
• Added shell: bash to all script steps for cross-platform compatibility

.github/workflows/nightly_build.yml

• Added matrix strategy to run tests on both ubuntu-latest and windows-latest
• Added continue-on-error for Windows tests (non-blocking initially)
• Updated Slack notifications to indicate multi-platform testing

Testing

Manual test run: https://github.com/langflow-ai/langflow/actions/runs/23216451934

Results:

• ✅ Infrastructure works (servers start, tests execute on Windows)
• ✅ 51/57 test shards passed (89% success rate)
• ✅ Successfully identified Windows-specific issues:
  • File upload checkbox behavior differences
  • Database file locking differences
  • Performance/timing variations

Behavior

Current (After Merge)

• Linux tests: Blocking (failures block nightly build)
• Windows tests: Non-blocking (failures reported but don't block)
• Execution: Parallel (no increase in total build time)
• Frequency: Every nightly build (00:00 UTC)

Why Non-Blocking?

Windows tests use continue-on-error: true initially to:

• Provide visibility into Windows issues without blocking releases
• Allow incremental bug fixing
• Stabilize tests before making them blocking
• Plan: Remove continue-on-error once Windows tests are consistently passing

Cost Impact

• Additional cost: ~$7/month for Windows runners
• Build time: No increase (parallel execution)
• Value: Early detection of Windows-specific bugs

Benefits

1. Early Detection: Catch Windows-specific issues before release
2. Platform Parity: Ensure consistent behavior across platforms
3. User Confidence: Better support for Windows users
4. Proven Approach: Leverages existing cross-platform test patterns

Next Steps

• Monitor Windows test results for 1-2 weeks
• File issues for Windows-specific bugs found
• Fix Windows bugs incrementally
• Remove continue-on-error once tests are stable

Related

• JIRA: LE-566
• Cross-platform tests already prove Windows compatibility for installation/server startup
• This adds comprehensive E2E testing on Windows

Checklist

• Changes tested via manual workflow dispatch
• Playwright installation works on Windows
• Matrix strategy implemented correctly
• Slack notifications updated
• Non-blocking error handling added

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4a5e5613-42d3-404b-9a52-f32153f463d1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch add-windows-playwright-nightly

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can scan for known vulnerabilities in your dependencies using OSV Scanner.

OSV Scanner will automatically detect and report security vulnerabilities in your project's dependencies. No additional configuration is required.

[github-actions]

Frontend Unit Test Coverage Report

Coverage Summary

Lines	Statements	Branches	Functions
	24.37% (8698/35681)	17.19% (4832/28095)	17.05% (1275/7474)

Unit Test Results

Tests	Skipped	Failures	Errors	Time
2783	0 💤	0 ❌	0 🔥	47.718s ⏱️

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (release-1.9.0@f801a8d). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-1.9.0   #12221   +/-   ##
================================================
  Coverage                 ?   38.05%           
================================================
  Files                    ?     1646           
  Lines                    ?    82253           
  Branches                 ?    12219           
================================================
  Hits                     ?    31302           
  Misses                   ?    49169           
  Partials                 ?     1782           

Flag	Coverage Δ
backend	54.25% <ø> (?)
frontend	21.75% <ø> (?)
lfx	44.53% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Adam-Aghili]

LGTM

To test properly we need merge then run a nightly.

[Adam-Aghili]

This whole section has incorrect white space

[Adam-Aghili]

I pushed a commit to fix this now it is complaining about
Required property is missing: runs-on

on

  frontend-tests:
    if: github.repository == 'langflow-ai/langflow' && !inputs.skip_frontend_tests
    name: Run Frontend Tests - ${{ matrix.os }}
    needs: create-nightly-tag
    # Allow Windows tests to fail without blocking the nightly build
    # This gives us visibility into Windows-specific issues while we stabilize the tests
    # Remove this once Windows tests are consistently passing
    continue-on-error: ${{ matrix.os == 'windows-latest' }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest]
    uses: ./.github/workflows/typescript_test.yml
    with:
      tests_folder: "tests"
      release: true
      runs-on: ${{ matrix.os }}
    secrets:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      STORE_API_KEY: ${{ secrets.STORE_API_KEY }}
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}

it looks like reusable workflows do not support using matrix.

[Adam-Aghili]

I only noticed the white space issue because the github action is broken

[Adam-Aghili]

seeing

[Adam-Aghili]

I am still seeing errros locally but i think it is my linter that is messing up

runs-on is right there and correct from my understanding

[Adam-Aghili]

Also the name of the noghtly action is fixed which usually means it isn't a syntax error in the branch