[ADR-206] Improve API tests

[jodavis]

Standardize API tests, make them easier to debug, and add more comprehensive coverage. That will make it easier to add tests and validate changes as we add services going forward.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR standardizes the backend API integration tests and supporting fixtures to improve debuggability (especially via log files) and expand coverage across authentication and pipeline scenarios.

Changes:

• Refactors API integration tests to reuse EndToEnd step bindings and adds broader endpoint/auth coverage in .feature files.
• Introduces new test fixtures/utilities for service processes (incl. LocalStack + JWT authority) and log capture/verification.
• Consolidates backend service logging into a shared AdaptiveRemote.Backend.Common project and adds simple file logging support.

Reviewed changes

Copilot reviewed 79 out of 84 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/AdaptiveRemote.TestUtilities/HttpClientExtensions.cs	Adds HTTP response content helper for tests (depends on shared wait helper).
test/AdaptiveRemote.TestUtilities/AdaptiveRemote.TestUtilities.csproj	Removes test SDK package reference from utilities project.
test/AdaptiveRemote.EndtoEndTests.TestServices/WaitHelpers.cs	Moves WaitHelpers into the AdaptiveRemote.TestUtilities namespace.
test/AdaptiveRemote.EndtoEndTests.TestServices/TestClient.cs	Adds a reusable HTTP test client with logging + JSON parsing helpers.
test/AdaptiveRemote.EndtoEndTests.TestServices/SimulatedTiVo/SimulatedTiVoDevice.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/SimulatedBroadlink/SimulatedBroadlinkDevice.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/SimulatedBroadlink/ISimulatedBroadlinkDeviceExtensions.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/Logging/TestResultFileHelper.cs	Adds helper to attach result files with retries.
test/AdaptiveRemote.EndtoEndTests.TestServices/Logging/HostApplicationLoggerProvider.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/IUITestServiceExtensions.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/ITestEndpointExtensions.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/IApplicationTestServiceExtensions.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/Host/SimulatedEnvironment.cs	Adds lazy-start fixtures for backend services and LocalStack + exposes log paths.
test/AdaptiveRemote.EndtoEndTests.TestServices/Host/ISimulatedEnvironment.cs	Extends simulated environment contract for backend service fixtures + logs.
test/AdaptiveRemote.EndtoEndTests.TestServices/Host/AdaptiveRemoteHost.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/Host/AdaptiveRemoteHost.Builder.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/BlazorWebViewUITestService.cs	Adjusts imports to use shared test utilities.
test/AdaptiveRemote.EndtoEndTests.TestServices/Backend/ServiceFixture.cs	Adds process-based backend service fixture with health polling + log file support.
test/AdaptiveRemote.EndtoEndTests.TestServices/AdaptiveRemote.EndtoEndTests.TestServices.csproj	Adds AWS/JWT deps and references shared test utilities.
test/AdaptiveRemote.EndToEndTests.Steps/UISteps.cs	Moves application steps into Steps.Application namespace.
test/AdaptiveRemote.EndToEndTests.Steps/StepsBase.cs	Adds TestClient access to step base.
test/AdaptiveRemote.EndToEndTests.Steps/SpeechSteps.cs	Moves application steps into Steps.Application namespace.
test/AdaptiveRemote.EndToEndTests.Steps/SimulatedTiVoSteps.cs	Moves application steps + imports shared test utilities.
test/AdaptiveRemote.EndToEndTests.Steps/SimulatedBroadlinkSteps.cs	Moves application steps + imports shared test utilities.
test/AdaptiveRemote.EndToEndTests.Steps/LogVerificationSteps.cs	Expands log assertions to support multiple service logs and incremental reading.
test/AdaptiveRemote.EndToEndTests.Steps/ISpeechTestServiceExtensions.cs	Moves application steps + imports shared test utilities.
test/AdaptiveRemote.EndToEndTests.Steps/Hooks/EnvironmentSetupHooks.cs	Attaches multiple service logs to test results.
test/AdaptiveRemote.EndToEndTests.Steps/DebugSteps.cs	Moves application steps into Steps.Application namespace.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/ServiceSteps.cs	Adds step transformations to map service names to fixtures/URIs.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/RawLayoutSteps.cs	Adds assertions for RawLayout responses.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/HttpResponseSteps.cs	Adds reusable HTTP response assertions (status/body/JSON/property checks).
test/AdaptiveRemote.EndToEndTests.Steps/Backend/HttpRequestSteps.cs	Adds reusable HTTP request steps and layout creation helper.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/HealthResponseSteps.cs	Adds JsonTypeInfo transform for HealthResponse.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/CompiledLayoutSteps.cs	Adds assertions for CompiledLayout contents.
test/AdaptiveRemote.EndToEndTests.Steps/Backend/AuthenticationSteps.cs	Adds auth token setup steps via test JWT authority.
test/AdaptiveRemote.EndToEndTests.Steps/AdptiveRemoteHostSteps.cs	Moves application steps into Steps.Application namespace.
test/AdaptiveRemote.EndToEndTests.Steps/AccessibilitySteps.cs	Moves application steps into Steps.Application namespace.
test/AdaptiveRemote.Backend.ApiTests/reqnroll.json	Configures ApiTests to load step bindings from EndToEnd step assembly.
test/AdaptiveRemote.Backend.ApiTests/Support/TestJwtAuthority.cs	Relocates TestJwtAuthority into shared backend test-services namespace.
test/AdaptiveRemote.Backend.ApiTests/Support/StubCompiledLayoutService.cs	Removes in-process stub compiled layout service.
test/AdaptiveRemote.Backend.ApiTests/Support/ServiceFixture.cs	Removes old ApiTests service fixture implementation.
test/AdaptiveRemote.Backend.ApiTests/Support/ServiceContext.cs	Removes old ApiTests per-scenario context.
test/AdaptiveRemote.Backend.ApiTests/Support/PipelineServiceFixture.cs	Removes old pipeline fixture implementation.
test/AdaptiveRemote.Backend.ApiTests/Support/PipelineContext.cs	Removes old pipeline per-scenario context.
test/AdaptiveRemote.Backend.ApiTests/Support/LocalStackFixture.cs	Refactors LocalStack fixture to sync APIs + logger injection.
test/AdaptiveRemote.Backend.ApiTests/StepDefinitions/RawLayoutSteps.cs	Removes old ApiTests step definitions (now using shared step assembly).
test/AdaptiveRemote.Backend.ApiTests/StepDefinitions/LayoutProcessingServiceSteps.cs	Removes old ApiTests step definitions (now using shared step assembly).
test/AdaptiveRemote.Backend.ApiTests/StepDefinitions/CommonSteps.cs	Removes old ApiTests step definitions (now using shared step assembly).
test/AdaptiveRemote.Backend.ApiTests/StepDefinitions/AuthenticationSteps.cs	Removes old ApiTests step definitions (now using shared step assembly).
test/AdaptiveRemote.Backend.ApiTests/Hooks/ApiTestHooks.cs	Adds ApiTests hook to register host settings in DI container.
test/AdaptiveRemote.Backend.ApiTests/Features/RawLayoutEndpoints.feature	Expands RawLayout endpoint coverage, auth cases, and log assertions.
test/AdaptiveRemote.Backend.ApiTests/Features/LayoutProcessingServiceEndpoints.feature	Updates health + pipeline scenarios to use shared steps and log assertions.
test/AdaptiveRemote.Backend.ApiTests/Features/HealthEndpoints.feature	Updates health tests to shared steps + response assertions.
test/AdaptiveRemote.Backend.ApiTests/Features/CompiledLayoutEndpoints.feature	Adds auth scenarios + compiled layout assertions + log checks.
test/AdaptiveRemote.Backend.ApiTests/Features/AuthenticationEndpoints.feature	Updates auth tests to shared steps + log checks.
test/AdaptiveRemote.Backend.ApiTests/AdaptiveRemote.Backend.ApiTests.csproj	Switches ApiTests to reference shared steps/utilities projects.
src/AdaptiveRemote.Backend.RawLayoutService/Services/SqsLayoutProcessingTrigger.cs	Switches logging to shared backend common logging.
src/AdaptiveRemote.Backend.RawLayoutService/Program.cs	Adds --logFile support + switches to common logging messages.
src/AdaptiveRemote.Backend.RawLayoutService/Logging/MessageLogger.cs	Removes service-specific message logger (moved to common).
src/AdaptiveRemote.Backend.RawLayoutService/Endpoints/LayoutEndpoints.cs	Replaces per-endpoint logs with request scope logging.
src/AdaptiveRemote.Backend.RawLayoutService/Endpoints/HealthEndpoints.cs	Replaces per-endpoint logs with request scope logging.
src/AdaptiveRemote.Backend.RawLayoutService/AdaptiveRemote.Backend.RawLayoutService.csproj	Adds reference to backend common project.
src/AdaptiveRemote.Backend.LayoutProcessingService/Services/StubLayoutValidationClient.cs	Changes how tests force validation failures.
src/AdaptiveRemote.Backend.LayoutProcessingService/Services/StubLayoutCompilerClient.cs	Injects special marker to trigger validation failure in tests.
src/AdaptiveRemote.Backend.LayoutProcessingService/Services/LayoutProcessingOrchestrator.cs	Switches logging to shared backend common logging.
src/AdaptiveRemote.Backend.LayoutProcessingService/Program.cs	Adds --logFile support + switches to common logging messages.
src/AdaptiveRemote.Backend.LayoutProcessingService/Logging/MessageLogger.cs	Removes service-specific message logger (moved to common).
src/AdaptiveRemote.Backend.LayoutProcessingService/Endpoints/HealthEndpoints.cs	Replaces per-endpoint logs with request scope logging.
src/AdaptiveRemote.Backend.LayoutProcessingService/AdaptiveRemote.Backend.LayoutProcessingService.csproj	Adds reference to backend common project.
src/AdaptiveRemote.Backend.CompiledLayoutService/Program.cs	Adds --logFile support + switches to common logging messages.
src/AdaptiveRemote.Backend.CompiledLayoutService/Logging/MessageLogger.cs	Removes service-specific message logger (moved to common).
src/AdaptiveRemote.Backend.CompiledLayoutService/Endpoints/LayoutEndpoints.cs	Adds stub POST endpoint for compiled layouts + request scope logging.
src/AdaptiveRemote.Backend.CompiledLayoutService/Endpoints/HealthEndpoints.cs	Replaces per-endpoint logs with request scope logging.
src/AdaptiveRemote.Backend.CompiledLayoutService/AdaptiveRemote.Backend.CompiledLayoutService.csproj	Adds reference to backend common project.
src/AdaptiveRemote.Backend.Common/Logging/RequestHandlerScope.cs	Adds request scope helper that logs start/end consistently.
src/AdaptiveRemote.Backend.Common/Logging/MessageLogger.cs	Adds shared source-generated logging messages for multiple services.
src/AdaptiveRemote.Backend.Common/Logging/FileLoggerExtensions.cs	Adds a simple file logger provider for test-time log capture.
src/AdaptiveRemote.Backend.Common/AdaptiveRemote.Backend.Common.csproj	Introduces shared backend common library project.
AdaptiveRemote.sln	Adds backend common project to solution.

Files not reviewed (5)

• test/AdaptiveRemote.Backend.ApiTests/Features/AuthenticationEndpoints.feature.cs: Language not supported
• test/AdaptiveRemote.Backend.ApiTests/Features/CompiledLayoutEndpoints.feature.cs: Language not supported
• test/AdaptiveRemote.Backend.ApiTests/Features/HealthEndpoints.feature.cs: Language not supported
• test/AdaptiveRemote.Backend.ApiTests/Features/LayoutProcessingServiceEndpoints.feature.cs: Language not supported
• test/AdaptiveRemote.Backend.ApiTests/Features/RawLayoutEndpoints.feature.cs: Language not supported

Comments suppressed due to low confidence (10)

test/AdaptiveRemote.EndtoEndTests.TestServices/TestClient.cs:1

• The structured logging placeholders {ClientID} and {RequestNumber} don’t match the argument order (line 46 passes requestNumber where {ClientID} is expected). Swap the first two arguments so ClientID receives _clientID and RequestNumber receives requestNumber, otherwise logs will be misleading when debugging tests.
  test/AdaptiveRemote.EndtoEndTests.TestServices/TestClient.cs:1
• This re-reads the response content and blocks on .Result, which can deadlock and/or consume a non-buffered content stream. Prefer logging the already-captured _lastResponseBody (or LastResponseBody) instead of calling ReadAsStringAsync().Result again.
  test/AdaptiveRemote.EndtoEndTests.TestServices/Backend/ServiceFixture.cs:1
• The log filename includes an extra trailing ) which will produce an incorrect path (and makes log attachment/reading fail). Remove the stray ) so the generated file path ends with .log.
  test/AdaptiveRemote.EndtoEndTests.TestServices/Backend/ServiceFixture.cs:1
• Standard output/error are redirected but never read. If the child process writes enough output, it can block on a full buffer and hang the test run. Either disable redirection, or start draining both streams (e.g., BeginOutputReadLine/BeginErrorReadLine with handlers, or background tasks reading ReadLineAsync) immediately after Start().
  test/AdaptiveRemote.EndtoEndTests.TestServices/Backend/ServiceFixture.cs:1
• Standard output/error are redirected but never read. If the child process writes enough output, it can block on a full buffer and hang the test run. Either disable redirection, or start draining both streams (e.g., BeginOutputReadLine/BeginErrorReadLine with handlers, or background tasks reading ReadLineAsync) immediately after Start().
  test/AdaptiveRemote.EndToEndTests.Steps/LogVerificationSteps.cs:1
• This assertion message has two format placeholders ({0}, {1}) but only provides one argument (serviceName), which can throw FormatException and mask the real test failure. Pass both serviceName and expectedMessagePart.
  test/AdaptiveRemote.EndToEndTests.Steps/StepsBase.cs:1
• This property never caches the resolved TestClient into _testClient, so it re-queries the container every time. Use assignment (e.g., _testClient ??= ...) to keep the behavior consistent with Logger/Environment caching and reduce repeated container lookups.
  test/AdaptiveRemote.EndToEndTests.Steps/Backend/HttpResponseSteps.cs:1
• The step text says 'does not contain', but StringAssert.DoesNotMatch treats the pattern as a regex (so special characters in unexpectedContent will change semantics). Use a non-regex contains check (e.g., StringAssert.DoesNotContain / Assert.IsFalse(...Contains...)) to match the step’s intent.
  test/AdaptiveRemote.TestUtilities/HttpClientExtensions.cs:1
• HttpClientExtensions (in AdaptiveRemote.TestUtilities) depends on WaitHelpers, but the diff shows WaitHelpers.cs still lives under the AdaptiveRemote.EndtoEndTests.TestServices project path. If WaitHelpers is not compiled into the AdaptiveRemote.TestUtilities assembly, this will fail to build. Consider moving WaitHelpers.cs into the AdaptiveRemote.TestUtilities project (or otherwise ensuring the type is available without introducing an undesirable project reference direction).
  test/AdaptiveRemote.EndToEndTests.Steps/Hooks/EnvironmentSetupHooks.cs:1
• This hook attaches log files directly, but the PR also introduces TestResultFileHelper.AttachResultFileIfExists (with retry logic for in-flight writes). To avoid duplicated logic and improve reliability, use the helper here (and include the service name in the helper’s logging if needed).