Make iceberg integration tests optional

[andygrove]

What is the problem the feature request solves?

It is great that we now have Iceberg integration tests, but each run takes almost 2 hours and if one run fails (which happens often) then we have to wait for the first run to complete before being able to restart the second, resulting in an almost 4 hour delay before a PR is green. This can be annoying when the PR is updating something that would not impact the Iceberg support.

I'd like to discuss options for being able to skip these tests on some PRs or run them on demand.

Describe the potential solution

No response

Additional context

No response

[andygrove]

@hsiang-c @kazuyukitanimura Do you have any thoughts on this? One idea I have is to only run these workflows if the PR title contains [iceberg].

[kazuyukitanimura]

Is it possible to split the test make each run shorter and easier to retry? @hsiang-c
There were issues with Iceberg integrations, so I think CI is important.

[parthchandra]

One idea I have is to only run these workflows if the PR title contains [iceberg].

I like this idea. Run either the Spark SQL tests or the Iceberg tests depending on whether the PR is tagged [iceberg]. @kazuyukitanimura do you think this would mitigate any iceberg issues we have.

Separately, since we have contributed the Comet integration to Iceberg, I feel it might be a good idea to contribute a CI to run Iceberg/Comet/Spark tests to the Iceberg project as well. (The issues we have seen could have been caught on the Iceberg side, particularly since we cannot test any changes unless both Comet and Iceberg are updated).

[hsiang-c]

@andygrove @parthchandra We can make it optional for now and meanwhile I'll try to split the tests into smaller chunks to speed up CI.

[andygrove]

this was implemented