Add wait conditions to check when setup is ready

[smola]

Description

Replace setup sleep with precondition checks, which should generally be faster:

• Wait for traces: Issues a final request and waits until its trace is received. It is assumed that traces for previous request will have been received by the time the last trace is received.
• Wait for telemetry: Wait to receive 2 heartbeats after the last trace. This should be enough to have received metrics or other events generated by previous requests.
• Wait for remote config: Wait until the number of required remote config messages for each scenario have been received.
• Wait for DSM events (scenario: INTEGRATIONS)
• Wait for profiling events (scenario: PROFILING)
• Fallback timeouts
• Wait for OTel
• Wait for backend in E2E tests
• Move most logic to wait_conditions.py
• Make sure tests using /make_distant_call verify the secondary request, then make sure it's compatible with the wait conditions.
• Add further documentation here about how all conditions work, how they relate to each other, and why they actually work.

Supersedes:

• [POC] Add wait conditions for setup stage #1150

Related:

• Fix getting rid from user-agent for cpp tracer #1453
• [Java][Ratpack] Avoid blocking in /make_distant_call #1460

Motivation

Reviewer checklist

• If this PR modifies anything else than strictly the default scenario, then remove the run-default-scenario label
• CI is green
  • If not, failing jobs are not related to this change (and you are 100% sure about this statement)

Workflow

1. ⚠️⚠️ Create your PR as draft
2. Follow the style guidelines of this project (See how to easily lint the code)
3. Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
4. Mark it as ready for review

NOTE: By default in PR only default scenario tests will be launched. Remove the run-default-scenario label to run all scenarios (more info)

Once your PR is reviewed, you can merge it! ❤️