Handle VM shutdown better

[galamit86]

Currently, if the VM shuts down while a flow is being run, the flow-run remains active for a long time - up to a 12 or more hours on some occasions.

Considering our setup is built on a preemptible machine, the recovery and restart time should be much faster.

Not sure how to implement.

[dkapitan]

I think this is harder than I thought initially. See PrefectHQ/prefect#2942. Currently, cancelling flows is only implemented from the cloud backend to the agent, and not the other way around.

Perhaps easiest for now is to use non-premptible machines and leave this for later.

Since standard machines only cost around one euro per dag to run, that's fine for now.

[galamit86]

Yes, I got the same feeling that it's more complicated that we thought - otherwise I'd expect Prefect "to handle it".

I think that's a good solution for now - when we start reaching actual costs, we might think of a different setup that might be cheaper - maybe from the GCP environment, "catch" the shutdown, then cancel the flow, restart the machine and restart the flow?

[galamit86]

I think this will be relevant.

[dkapitan]

indeed, good catch!

…

On Wed, Mar 17, 2021 at 5:31 PM Amit Gal ***@***.***> wrote: I think this <https://docs.prefect.io/core/concepts/persistence.html> will be relevant. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GYDW7BILRUA3BDFSBH5TTEDKN7ANCNFSM4ZKWLVTA> .