[Feature Request]: Flink runner savepoint backward compatibility

[omkardeshpande8]

What would you like to happen?

Flink savepoints are backward compatible. But when using beam flink runner, this is not the case. It prevents seamless migration versions for stateful applications.

Issue Priority

Priority: 1

Issue Component

Component: runner-flink

[je-ik]

What do you mean by backward compatible savepoint in Flink? Beam's Flink runner uses Flink's checkpoint/savepoint mechanism, thus it should be compatible under the same circumstances as native Flink. The problem is that Beam adds additional layer of abstraction that can cause state incompatibility - e.g. when a translation of a Beam PTransform changes, when a Coder changes, etc.
There currently is not any mechanism that can transfer incompatible savepoints between versions, the only option is to run a "bootstrap" version of your job, that creates the state from some historical data.

Reading the a savepoint and converting it to preserve compatibility across beam versions could be possible under some circumstances, but is not (currently) possible in general.

[jainikvora]

@je-ik From our experience, just beam version upgrade without any processor code change is not compatible and this is Beam's restriction, not Flink's. Here's a mailing thread with more details. Any plans to support this in Beam? Also interested to know if this restriction exists for dataflow as well or only non-dataflow runners?

[je-ik]

Generally this is problem of Java serialization. Beam could (and should) pay more attention to not breaking it, but it nevertheless can happen from time to time. Moreover, the PTransform expansion can change and a lot of other things can go wrong, so - currently - the ability to recreate the pipeline into meaningful state from scratch is vital.

I think it could be theoretically possible to create a runner and beam-version (into some extent) agnostic snapshot of a pipeline, which could then be used to overcome such restrictions and even port a (running) pipeline between runners. But that would be a large project so even if something like this will come into existence in the future it cannot be expected soon.

[je-ik]

Related: #30385