Flink Go XVR tests fail on TestXLang_...: Insufficient number of network buffers

[damccorm]

When running the cross-language test suites () Flink fails on TestXLang_Multi with the following error:

19:29:14 2021/08/27 02:29:14  (): java.io.IOException: Insufficient number of network buffers: required
17, but only 16 available. The total number of network buffers is currently set to 2048 of 32768 bytes
each. You can increase this number by setting the configuration keys 'taskmanager.memory.network.fraction',
'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
19:29:14 2021/08/27 02:29:14
Job state: FAILED
19:29:14 --- FAIL: TestXLang_Multi (6.26s)

This doesn't seem to be a parallelism problem (go test is run with "-p 1" as expected) and is only happening on this specific test.

Imported from Jira BEAM-12815. Original Jira may contain additional context.
Reported by: danoliveira.

[damccorm]

This hasn't happened for a long time

[Abacn]

It happens again after switched to use Flink 1.15 for testing and being permared:

Task :runners:flink:1.15:job-server:validatesCrossLanguageRunnerGoUsingJava FAILED

05:30:04 Caused by: java.io.IOException: Insufficient number of network buffers: required 17, but only 0 available. The total number of network buffers is currently set to 2048 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
05:30:04 	at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.internalCreateBufferPool(NetworkBufferPool.java:483)
05:30:04 	at org.apache.flink.runtime.io.network.buffer.NetworkBufferPool.createBufferPool(NetworkBufferPool.java:461)
05:30:04 	at org.apache.flink.runtime.io.network.partition.ResultPartitionFactory.lambda$createBufferPoolFactory$0(ResultPartitionFactory.java:279)
05:30:04 	at org.apache.flink.runtime.io.network.partition.ResultPartition.setup(ResultPartition.java:160)
05:30:04 	at org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition.setup(BufferWritingResultPartition.java:97)
05:30:04 	at org.apache.flink.runtime.taskmanager.Task.setupPartitionsAndGates(Task.java:959)
05:30:04 	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:652)
05:30:04 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
05:30:04 	at java.lang.Thread.run(Thread.java:750)
05:30:04 2022/11/16 10:30:03  (): java.io.IOException: Insufficient number of network buffers: required 17, but only 0 available. The total number of network buffers is currently set to 2048 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.memory.network.fraction', 'taskmanager.memory.network.min', and 'taskmanager.memory.network.max'.
05:30:04 2022/11/16 10:30:03 Job state: FAILED
05:30:04     ptest.go:108: Failed to execute job: job go0testxlang0partition0717-jenkins-1116102953-b57d468_c2e367a3-5ff1-4ff6-a475-1faea47d2c99 failed
05:30:04 --- FAIL: TestXLang_Partition (13.36s)
05:30:04 FAIL
05:30:04 FAIL	github.com/apache/beam/sdks/v2/go/test/integration/xlang	82.614s