Spark : The iceberg-spark3-extensions module CI test failed

[zhangjun0x01]

I found that ,the iceberg-spark3-extensions module CI test often failed.

the exception like this

org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete > testDeleteNonExistingRecords[catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, clients=1, parquet-enabled=false, cache-enabled=false}, format = avro, vectorized = false] FAILED
    java.lang.RuntimeException: Failed to get table info from metastore default.table

        Caused by:
        org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)

            Caused by:
            java.net.SocketException: Broken pipe (Write failed)

[kbendick]

I too have found that this test fails. However, it does not appear to be deterministic. I'm guessing it's possible that the tests are run in such an order that the table gets dropped during one of the other tests.

There's also multiple levels of inheritance used for this test suite, but I don't necessarily think that's the issue. However, it does offer some clues, as the non-abstract test suites hat inherit from the higher up classes such as TestExtensionsTestBase all seem to pass. The issue appears to only be in TestCopyOnWriteDelete, which is possibly conflicting in its DROP TABLE statements with the other test suite that extends SparkRowLevelOperationsTestBase, TestCopyOnWrtieMerge, or is conflicting with tests of its own.

Woud making the names of the tables used in each test somewhat random prevent this scenario? I think this would entail updating createAndInitTable to take in the table name and then use that table name instead of a class level variable. This is not exactly a trivial refactor though.

[kbendick]

Additionally, another option seems to be that we could use TestSparkTableUtilWithInMemoryCatalog, or essentially update the test suite to not use the Hive metastore for all of these tests given that its prone to connection leaks at times.

[pan3793]

+1 I also can reproduce in local with ./gradlew :iceberg-spark3-extensions:test, but passed in IDEA.

[zhangjun0x01]

I found it is random, there are many times the test passed on my computer, but CI test failed

[kbendick]

I've also been able to get CI to pass recently (as in today), but it's sporadic & non-deterministic.

[kbendick]

Locally, when I increase the catalog config value for clients from 1 to 2 or higher, this has passed on the past 3 times I've tried it. This is obviously kind of a hack, and having a value for clients that is > 1 won't necessarily work for catalogs backed by derby. However, it would be helpful if other people can try upping these values and see if that helps.

I upped the value of clients from 1 in the following test parameters (in the immutable config map, using "clients", "2"). I did not update it for the one value that is hard coded as 1.

iceberg/spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java

Lines 57 to 87 in 74bb38d

	@Parameters(name = "catalogName = {0}, implementation = {1}, config = {2}, format = {3}, vectorized = {4}")
	public static Object[][] parameters() {
	return new Object[][] {
	{ "testhive", SparkCatalog.class.getName(),
	ImmutableMap.of(
	"type", "hive",
	"default-namespace", "default"
	),
	"orc",
	true
	},
	{ "testhadoop", SparkCatalog.class.getName(),
	ImmutableMap.of(
	"type", "hadoop"
	),
	"parquet",
	RANDOM.nextBoolean()
	},
	{ "spark_catalog", SparkSessionCatalog.class.getName(),
	ImmutableMap.of(
	"type", "hive",
	"default-namespace", "default",
	"clients", "1",
	"parquet-enabled", "false",
	"cache-enabled", "false" // Spark will delete tables using v1, leaving the cache out of sync
	),
	"avro",
	false
	}
	};
	}

[stevenzwu]

I started a thread titled as "test flakiness with SocketException of broken pipe in HiveMetaStoreClient" [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.

[1] https://www.mail-archive.com/dev@iceberg.apache.org/msg01789.html

[kbendick]

I started a thread titled as "test flakiness with SocketException of broken pipe in HiveMetaStoreClient" [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.

[1] https://www.mail-archive.com/dev@iceberg.apache.org/msg01789.html

Cool thank you @stevenzwu. Connection leak would potentially explain why I’m much less able to reproduce when I increase the catalog’s clients config value. Also, knowing that it is (was?) happening in the flink tests helps a ton with guidance for how much time to put into investigating from the POV of a potential test inheritance pattern issue vs connection leak issue.

This is very helpful. I should be more active on the dev list, though I’m only able to contribute somewhat sporadically given my current work is not exactly iceberg related. I’ll follow up in that thread too with anything I find while investigating. 👍

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'