Skip to content

test(infra) Multimorphic scraper-pipeline integration tests#48

Merged
jkeeley2073 merged 1 commit into
mainfrom
Dev-MultimorphicScraperTestInfra
May 3, 2026
Merged

test(infra) Multimorphic scraper-pipeline integration tests#48
jkeeley2073 merged 1 commit into
mainfrom
Dev-MultimorphicScraperTestInfra

Conversation

@jkeeley2073
Copy link
Copy Markdown
Contributor

Summary

Backfill of the PR #41 family-wide test-infra template for MultimorphicProductScraper. 5 scraper-pipeline integration tests covering the audit-flagged invariants the unit-test surface cannot reach.

  • Discovery shape: WordPress sitemap walk — /wp-sitemap.xml -> wp-sitemap-posts-product-1.xml -> per-product HTML pages, filtered to /store/p3-game-kits/multimorphic-game-kits/{slug}/.
  • Yield kind: Single — .Game only (no .Link).
  • Source: SourceType.MultimorphicProductPage, discovery context "Multimorphic Game Kit".

Tests added

  1. ScrapeAsync_HappyPath_YieldsGamesInSitemapOrderWithProvenance — sitemap index XML -> product sub-sitemap XML (with third-party kit + circuit board mixed in to pin the prefix filter) -> 3 product HTML pages. Asserts: yield count + slug order, full provenance (SourceType, DiscoveryContext, DiscoveryUrl, Source.ScrapedFrom, ScrapedAt, DiscoveredOn), gate-vs-wire URL byte-equality, exact wire-trace (5 requests: index + sub-sitemap + 3 products, no third-party / circuit board).
  2. ScrapeAsync_PerPageFetchFailure_DoesNotAbortRun — middle product page returns 500; siblings still yield; politeness invariants hold under failure (the 500 is reported back).
  3. ScrapeAsync_DiscoveryFailure_AbortsThisSourceOnly — sitemap index returns 500; scraper yields nothing AND does not throw (clean yield-break for orchestrator).
  4. ScrapeAsync_PolitenessExceptionFromGate_PropagatesUpThrowOnAcquire injects PolitenessException; wire shows zero requests, gate shows zero reports.
  5. ScrapeAsync_GateThrowsOnReport_BubblesUp — symmetric: ThrowOnReport propagates.

Test count: 430 -> 435.

Pre-push self-audit

  • /local-review: deferred to the reviewer at merge time (per task instruction).
  • 7-item mechanical checklist: all pass.
    1. No new options added (tests-only PR).
    2. Sibling-diff vs CGC template: structure matches; adapted for XML sitemap fixtures, single-yield, sitemap-walk wire trace.
    3. No bare catch { }.
    4. CLI / orchestrator wiring: unchanged (multimorphic alias already wired in PR feat(multimorphic) Multimorphic scraper — Phase 1.3.c #39).
    5. Tests assert behavior — happy-path pins yield order + provenance content + URL equality; failure tests pin the broken slug is excluded AND the 500 is reported; politeness tests pin zero downstream requests.
    6. Build: 0 warnings, 0 errors.
    7. Identity: Jim Keeley <94459922+jkeeley2073@users.noreply.github.com> (personal noreply).

Test plan

  • dotnet build clean (0 warnings)
  • New tests pass (5/5)
  • Full suite passes (435/435)
  • Reviewer to run /local-review at merge time

Hard constraints honored

  • Touched only the new test file + a single CHANGELOG bullet.
  • No production code changed.
  • No scraper bugs found.

@jkeeley2073 jkeeley2073 added the claude-code Generated with Claude Code label May 3, 2026
Backfill of the PR #41 template across the family. 5 tests covering
happy-path with full provenance + gate-vs-wire URL equality, per-page
failure isolation, sitemap discovery failure, PolitenessException on
both Acquire and Report. Single-yield (.Game only) WordPress storefront
via /wp-sitemap.xml walk filtered to /store/p3-game-kits/multimorphic-game-kits/.

Pre-push audit: 7-item mechanical (all pass). /local-review deferred to
human reviewer at merge time.
@jkeeley2073 jkeeley2073 force-pushed the Dev-MultimorphicScraperTestInfra branch from e678e8f to 87daf96 Compare May 3, 2026 12:01
@jkeeley2073 jkeeley2073 merged commit e8e32cb into main May 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-code Generated with Claude Code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant