[Bug]: JDBCIO Write freeze at getConnection() in WriteFn

[Abacn]

What happened?

Re-raise BEAM-9629 as this seems not fully resolved.

Affect versions: at least >= 2.25, likely >=2.18 as in BEAM-9629.

log using 2.35.0

Operation ongoing in step Write Blacklisted Posts/ParDo(Write) for at least 45m00s without outputting or completing in state finish at ...
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:581) at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:437) at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:354) at 
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) at 
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.getConnection(JdbcIO.java:2029) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.executeBatch(JdbcIO.java:2087) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.finishBundle(JdbcIO.java:2048) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)

log using 2.25.0:

warnings:
Operation ongoing in step Write Blacklisted Posts/ParDo(Write) for at least 45m00s without outputting or completing in state finish at ... at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at 
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:581) at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:437) at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:354) at 
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) at 
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.getConnection(JdbcIO.java:2029) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.executeBatch(JdbcIO.java:2087) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.finishBundle(JdbcIO.java:2048) at 
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)

Issue Priority

Priority: 1

Issue Component

Component: io-java-jdbc

[Abacn]

A possible cause is that a racing condition could occur here:

beam/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

Line 2323 in 2f13855

private Connection getConnection() throws SQLException {

getConnection() gets called by many process element at the same time, with connection == null, and all of them trying to establish the dataSource.getConnection at the same time. Marking this async may solve the issue.

However, verifying this is blocked by broken JdbcIOIT, where the fix is currently in progress (#21796)

[johnjcasey]

is there a particular reason to not just put connection setup logic into @setup?

[johnjcasey]

looking at this more, it can't be in @setup, but it may be possible to use @startbundle instead

[Abacn]

Context of last change: #12209