openlineage: add support for system tests, add support for example_bigquery_queries system test

[mobuchowski]

The idea for system tests is as follows:

• add VariableTransport that will push OpenLineage events to Variable with key <DAG_ID>.<TASK_ID>.event.<EVENT_TYPE>.
• System tests DAGs will be configured to use this transport
• then, newly added OpenLineageTestOperator will compare the expected events with the ones that are stored in Variables.

This PR adds those components, as well as OL support for BigQueryGetDataOperator and OL support for system test utilizing it.

PR is based on previous PR: #31293 - only second commit is relevant to this PR.

Closes: #29676

[pankajkoti]

I am wondering if this is the right place to keep OpenLineage system tests or should we have it in its own home tests/system/providers/openlineage/...? WDYT?

Wondering if some config disables OpenLineage or uses a different lineage backend then would having this here affect the system test run?

[sunank200]

+1

[mobuchowski]

I believe it's advantageous to have OL system tests as an addition to regular ones, because they are just an addition of one operator, and actively use rest of the dag.

Those aren't only "OpenLineage system tests" in vacuum, but test OL integration of this particular provider.

Having separate dags for system tests would result in fragmentation - we'd have to maintain the same dag, or at least subset of it, with just the addition of a single OpenLineageTestOperator.

However, I believe option to disable OpenLineage and make this special operator always pass is a very valid need, and will add such option.

[sunank200]

+1

[mobuchowski]

Will reopen later