Remove Database Isolation feature flag and run DB isolation tests

[potiuk]

This PR removes AIP-44 feature flag and replaces "in-progress-disabled" test with dedicated "DatabaseIsolation" one.

The DatabaseIsolation test will run all "db-tests" with enabled DB isolation mode and running internal-api component - groups of tests marked with "skip-if-database-isolation" will be skipped.

• tests/models/test_taskinstance.py (32 tests failing)
• tests/operators/test_python.py (2 tests failing)
• tests/sensors/test_external_task_sensor.py (2 tests failing)

Some of those tests might need to be excluded as they make no sense to be run (for example if those are tests that are testing scheduler or webserver code). It might also be that we need to improve our test harness for isolation mode in some cases.

---

How to run the tests? It's fairly easy to run the tests with breeeze (you can checkout this pr with gh pr checkout 41067 and work on top of it.

You can run tests using DB isolation already using database isolation. It works in the way that you need to enter breeze shell --database-isolation, split the terminals with tmux, run internal-api component in one of them and then you can run pytest tests in the other terminal. Again - sources are mounted to inside the container so you can easily iterate on tests (ctrl-c restart internal-api to pick up changes).

See: https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#running-tests-with-database-isolation for detailed description.

One tests that fully works for now is dag_processing/test_job_runner.py for example and a good way to see how it works - you will see that the test initializes DB direcly, but then the dag_processing code calls the internal-api server and you will see messages going back-forth in the "internal-api" output.

You should make sure that you use pytest --run-db-tests-only flag - and all non-db tests will be skipped, that should speed up your tests when you run whole module or directory.

---

How to submit fixes?

After you make a fix - submit a PR explaining which tests are fixed with Related: #41067 in the commit message, That's it. I will be continuously rebasing this PR on top of main and keeping the inventory of what has been fixed.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

This PR will have quite a number of failing tests still - some of them because they should just be disabled - on a module level in most cases (with adding pytest.mark.skuip-if-database-isolation to pytest.mark.db-test.

Some of them because there are still some missing pieces to fix in the internal API.

The "special" DatabaseIsolation test that replaces "InProgressDisabled" runs the special "Database Isolation" mode that allows all our test fixtures and generally "test code" to access the database, while any code that is being tested from "airflow" should get special TracingSession assigned that will fail if the "airflow" code will attempt to use the database directly rather than via internal API.

I will make an extra pass today - once it completes - and disable all the "big" modules that should be disabled from testing, and maybe fix some obvious cases. But anyone (especially @jscheffl @vincbeck and @dstandish ) are most welcome to sign-up to fix some of those in parallel (please comment here when you pick something - I will be merging my fixes individually as separate PRs and will keep on rebasing that PR until we will get it hopefully green before 2.10rc1.

[aritra24]

I'll try taking a look at some failures tomorrow my time

[jscheffl]

I LOVE pull-requests which delete more lines than adding. But as being DRAFT, looking forward for review (if family time permits tomorrow)

[potiuk]

We still might remove the feature flag in 2.11. This will be easy - we do not need to keep that PR around for that.