Prevent breaking scheduler on task_instance.priority_weight column has overflow value

[kosteev]

closes: #22781

[eladkal]

The DAG shared on the issue runs just fine with Sqlite:

I'm not sure if this is the right fix. Why should the model be concerned with issues originated from a specific DB?
I suspect that this could be also reported on retries, pool_slots etc.
I think the fix should be in the scheduler level (at least by not crashing) / raising broken DAG?

[uranusjr]

The fix is specific to Postgres I believe. Every database has different integer ranges so this will need to contain some if-else branches to cover each of them. Although honestly maybe it’s even better to just document this is a database limitation and you just can’t go over the limit.

[eladkal]

The fix is specific to Postgres I believe. Every database has different integer ranges so this will need to contain some if-else branches to cover each of them. Although honestly maybe it’s even better to just document this is a database limitation and you just can’t go over the limit.

I agree but the point of concern is that this cause the scheduler to crash which is not good.
It means that one dag can cause cluster wide problems. I think we need to find a way to locallize the effect so only the problematic dag will be effected.

[uranusjr]

Yeah the scheduler should not crash, but perhaps we could catch the exception or something like that. I seem to recall there’s a similar situation for something else a while ago (datetime out of range?)

[eladkal]

I recall only #17003 which also caused scheduler crash with divide by zero

[ashb]

Is silently clamping it right, or should this be a parse time DAG import error?

[uranusjr]

Ah found it, 2213635

So I guess the better fix here is to do something similar, check for Postgres, and prevent an out-of-bounds value to be assigned for Postgres specifically.

+1 to an import error.

[kosteev]

All right, validating it during DAG import and throwing error makes sense.
What about different databases? Should we handle all the databases taking into account different constraints of each?

[uranusjr]

Let’s start with adding just a check for Postgres now, we can add more later.