AIP-84 Refactor Handling of Insert Duplicates

[jason810496]

related: #44121 (comment)

It seems that only the Post Connection endpoint has an extra query to handle duplicate entries. Other 409_CONFLICT cases don't handle inserting duplicate entries on unique constraint columns.

[jason810496]

In summary, this PR will only add orig and statement to the response error details.

Regexp can be not super reliable, also it will depends on the error message of the database (which can change between releases and versions).

I will still explore the regex or string-matching approach in a follow-up PR.
From my perspective, database error messages do not change frequently ( also we won't update the database driver version very often I think ). Even if the error message changes after upgrading the database driver, fixing it in one global handler would not be a significant issue.

[jason810496]

Hi @pierrejeambrun, @ephraimbuddy,

I’ve refactored the _UniqueConstraintErrorHandler to provide unified error details across all database dialects by parsing errors with re2. If the parsing fails, it defaults to returning the original database error.

I’ve also added tests for the unique constraint handler, covering the following cases:

• Inserting a duplicate entry for a single-column unique constraint (e.g., Pool, Variable).
• Inserting a duplicate entry for a multi-column unique constraint (e.g., dag_id and run_id pair in DagRun).
• Handling cases where parsing the database error message fails.

Looking forward to your feedback and suggestions!

[jason810496]

Rebase to latest main, looking forward to feedback.
cc @pierrejeambrun , @ephraimbuddy

[pierrejeambrun]

I think there are a lot of hardcoded values, regexp and manual string manipulation etc.
This code will likely break when the db version update the message or when a column name / values contains special character that we did not account for. The parsing could still fail.

Also that's a lot of code to maintain, and it's not super welcoming.

I'm not in favour of this change.

I still think that at first just returning the plain orig and statement is enough for a user to understand. And for tests we just assert that those keys are present in the response.

[kaxil]

I think there are a lot of hardcoded values, regexp and manual string manipulation etc. This code will likely break when the db version update the message or when a column name / values contains special character that we did not account for. The parsing could still fail.

Also that's a lot of code to maintain, and it's not super welcoming.

I'm not in favour of this change.

I still think that at first just returning the plain orig and statement is enough for a user to understand. And for tests we just assert that those keys are present in the response.

Yup, I agree, it looks too brittle unfortunately.

[pierrejeambrun]

Thanks for the effort @jason810496 in having these error messages parsed.

I don't think that's the best approach and I wouldn't invest more time trying to make this more reliable because it's very unlikely we do.

The best we can do at this point is to return the plain orig and maybe also statement in the response details. This will enable power users to actually understand why the resquest is conflicting. (parameters fields should not be exposed in the response)

[jason810496]

Thank you for the feedback, @pierrejeambrun and @kaxil!
I realize I overlooked the maintenance effort required for directly parsing errors from different dialects.
( Hope we’ll see a PEP someday that standardizes the error details across various DBAPI drivers, then we could adopt the parsing solution. )

Just refactored the implementation to include only the statement and orig in the error response.

[pierrejeambrun]

Thanks