Add VCR infrastructure for DSL functional tests

[mathiusj]

Add VCR Infrastructure for DSL Workflow Tests

Added VCR support for DSL workflow tests to enable reliable HTTP request recording and playback.

What changed?

• Created a new VCRURLRewriter module to handle API endpoint transformations for cassette portability
• Added a test workflow vcr_test_workflow.rb that makes a simple LLM query about the deepest lake
• Created a VCR cassette with a recorded response about Lake Baikal
• Added a new test case that verifies the workflow runs correctly with the VCR cassette
• Set up environment variables in test setup to ensure tests work in both recording and replay modes

How to test?

In replay mode (default):

bundle exec rake test

To record new cassettes:

OPENAI_API_KEY=your-key RECORD_VCR=true bundle exec rake test

Why make this change?

This change improves test reliability by eliminating dependencies on external API services during testing. By recording API interactions once and playing them back during subsequent test runs, we can:

• Make tests faster and more deterministic
• Avoid hitting rate limits or incurring costs from real API calls
• Test without requiring real API credentials
• Ensure consistent test behavior regardless of network conditions

[mathiusj]

• Add VCR infrastructure for DSL functional tests #655 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[mathiusj]

@juniper-shopify updated approach to add VCR support for the functional tests. I can iterate on this a bit more if this looks right to you 🙏

[juniper-shopify]

Run the new test: OPENAI_API_KEY=api-key-here RECORD_VCR=true bundle exec ruby -Itest test/dsl/functional/roast_dsl_examples_test.rb

This doesn't actually work. You need to also delete the cassette file that you want to re-record.

[juniper-shopify]

You're on the right track here. I think this is good.

A few functional issues to work through, but I like the direction

[mathiusj]

This doesn't actually work. You need to also delete the cassette file that you want to re-record.

@juniper-shopify I missed this comment earlier as I was focused on the inline comments, so would have to backtrack a bit to see what the issue was that you were getting. Mind sharing the stacktrace or response you were getting so I can see what the issue was? I'm also adding OPENAI_API_BASE=... to run the test with it pointing to our proxy, which does work with the rest of that line.

Please LMK if it works for you now! Also added a new class for rewriting the vcr config, with tests. Given it's EOD EDT I imagine this one may carry over to the new year, so can split this work into a stack for better reviewing/testing, when I get back

[dersam]

This PR removes a number of the useful test abstractions from the original functional test system, and reimplements several VCR features. Please refactor.

[dersam]

This doesn't actually work. You need to also delete the cassette file that you want to re-record.

@juniper-shopify I missed this comment earlier as I was focused on the inline comments, so would have to backtrack a bit to see what the issue was that you were getting. Mind sharing the stacktrace or response you were getting so I can see what the issue was? I'm also adding OPENAI_API_BASE=... to run the test with it pointing to our proxy, which does work with the rest of that line.

Please LMK if it works for you now! Also added a new class for rewriting the vcr config, with tests. Given it's EOD EDT I imagine this one may carry over to the new year, so can split this work into a stack for better reviewing/testing, when I get back

VCR will only record a new cassette if one doesn't already exist. If you need to replace the cassette and hit the proxy live, the old related cassette needs to be deleted. RECORD_VCR allows it to record if missing, without that flag it will simply fail if no cassette exists.

[mathiusj]

@dersam if I'm missing anything here now just lmk 🙏

[dersam]

What's the point of overwriting these credentials? When recording, real creds are necessary and will be filtered by filter_sensitive_data. Is there a reason this is needed during replay?