[C++] Fix and re-enable Asof Join Backpresure test flakiness

[icexelloss]

Describe the enhancement requested

See #36020 (comment)

The backpressure test in asof join is flaky. We should fix it.

Update (20230707):

In #36331 we decided to temporarily disable the flaky asof join backpressure test. So this jira is meant to track the work to fix the test and reenable it.

Component(s)

C++

[rtpsw]

@bkietz, do you have any more info about this issue? In which CI jobs did you observe the test fail? Did you always observe the same assertion fail? Did you notice how frequently the failure happens?

[bkietz]

So far I've only seen it once in this run of AMD64 Ubuntu 22.04 C++ ASAN UBSAN. I don't have a lot of information, sorry.

For trying to diagnose issues like this it'd be neat to have a label or tag or something which can be grepped for across all PRs. @amol- what if we had something like @github-actions register-flaky $link_to_CI_run, which saves the whole log to longer-term storage? That'd both make it more formal/obvious when you ignore a flaky failure in a PR and would make it possible for devs hunting those failures to find them. Even just a had-flaky-failure label for the PR would be pretty helpful

[raulcd]

@icexelloss is this a duplicate for: #36331
Should this be closed?

[icexelloss]

@raulcd Not really. #36331 merely disabled the test. This issue is meant to fixing/reenable the test. Let me update the description here.