test(infra) Barrels of Fun scraper-pipeline integration tests#45
Merged
Conversation
Backfill of the PR #41 template across the family. 5 tests covering happy-path with full provenance + gate-vs-wire URL equality, per-page failure isolation, discovery failure, PolitenessException on both Acquire and Report. Single-yield (.Game only) WooCommerce storefront via /product-category/machines/. Pre-push audit: 7-item mechanical (all pass). /local-review deferred to human reviewer at merge time.
6fb268c to
f1ba40f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backfill of the PR #41 family-wide test-infra template for
BofProductScraper. 5 new scraper-pipeline integration tests pinning yield order, provenance propagation, polite-gate routing invariants, and failure-isolation behavior.BofProductScraperis a single-yield scraper (yields.Gameonly — no.Linkitems, since BoF's WooCommerce product pages host no firmware/document downloads), so the yield-order assertion is over.Gameonly. Otherwise the test class mirrors the CGC proof-of-concept exactly:ScrapeAsync_HappyPath_YieldsGamesInIndexOrderWithProvenance— 3-machine category page + 3 product pages; asserts category-page order, full provenance fields (SourceType.BarrelsOfFunProductPage,DiscoveryContext = "Barrels of Fun Machines Category",DiscoveryUrl),GameRecord.Source.ScrapedFrom/GamePageUrl,DiscoveredOncarries thebarrelsoffun_machines_categorysentinel, and gate-vs-wire URL equality (so a future re-canonicalisation refactor can't silently throttle a different origin).ScrapeAsync_PerPageFetchFailure_DoesNotAbortRun— middle product returns 500; siblings still yield; politeness invariants hold under failure (500 is reported back to the gate so the 429-streak detector can see it).ScrapeAsync_DiscoveryFailure_AbortsThisSourceOnly—/product-category/machines/returns 500; scraper yield-breaks cleanly without throwing.ScrapeAsync_PolitenessExceptionFromGate_PropagatesUp— gate throws on Acquire BEFORE any HTTP fires; assertions pin zero wire requests / zero reports so a regression that swallowed the exception would fail loudly.ScrapeAsync_GateThrowsOnReport_BubblesUp— symmetric report-side path.Touched files: 1 new test file + 1 CHANGELOG bullet (no production code changes).
Pre-push self-audit
Step 0 —
/local-review: deferred to the human reviewer at merge time per the PR description (this is a backfill of an already-vetted template; the mechanical checks below are the binding gate).Step 1 — 7-item mechanical checklist (all pass):
*Options).CgcGamePageScraperTests. Differences are intentional and called out in remarks: single-yield (no.Linkassertions), category-page path (/product-category/machines/) instead of menu, BoF source-type / discovery-context / sentinel strings, and the JSON-LD product schema in fixtures (instead of CGC's title / anchor extraction). Test structure, helper signatures, gate/wire assertions, andBuildScrapershape are byte-identical to the template.catch { }— confirmed; no catches in test code.ISourceScraper).Equal(2, games.Count)distinguishes "kept going" from "aborted on first failure";PolitenessExceptiontests pinEmpty(handler.Requests)so a regression that catches and swallows can't pass.dotnet build→0 Warning(s) / 0 Error(s).git log -1 --format='%an <%ae>'→Jim Keeley <94459922+jkeeley2073@users.noreply.github.com>(personal noreply).Test counts: 430 → 435 (+5).
Test plan
dotnet build— clean, 0 warnings.dotnet test --filter FullyQualifiedName~BofProductScraperTests— 5 passed.🤖 Generated with Claude Code