Production-level support for MSSQL

[potiuk]

MSSQL has been somewhat experimental in the main branch, but as
we near releasing for 2.2.0 version, the image should support
the mssql at the level as it supports other databases.

This PR adds proper support for both PROD and CI images.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[mik-laj]

Could MSSQL be used as a celery backend?

airflow/scripts/in_container/prod/entrypoint_prod.sh

Lines 98 to 106 in f76eaec

	if [[ ${BACKEND} == "postgres"* ]]; then
	detected_port=5432
	elif [[ ${BACKEND} == "mysql"* ]]; then
	detected_port=3306
	elif [[ ${BACKEND} == "redis"* ]]; then
	detected_port=6379
	elif [[ ${BACKEND} == "amqp"* ]]; then
	detected_port=5672
	fi

[potiuk]

Could MSSQL be used as a celery backend?

Very good point! Indeed I believe it can

[potiuk]

Could MSSQL be used as a celery backend?

Added support for the entrypoint . Good catch @mik-laj !

[mik-laj]

I am missing one more element. On the database configuration page, we should add an example URL address.

For SQLite, we have the following fragment:

An example URI for the sqlite database:

sqlite:////home/airflow/airflow.db

For MySQL, we have the following fragment:

We recommend using the mysqlclient driver and specifying it in your SqlAlchemy connection string.

mysql+mysqldb://:@[:]/
But we also support the mysql-connector-python driver, which lets you connect through SSL without any cert options provided.

mysql+mysqlconnector://:@[:]/

For. PostgresSQL, we have the following fragment:

We recommend using the psycopg2 driver and specifying it in your SqlAlchemy connection string.

postgresql+psycopg2://:@/

Only MSSQL does not include an example URI address for me to be able to. configure the database. SQLAlchemy supports multiple drivers. Which one is the best for production deployment?

# pyodbc
mssql+pyodbc://scott:tiger@mydsn

# pymssql
mssql+pymssql://scott:tiger@hostname:port/dbname

# mxODBC
mssql+mxodbc://<username>:<password>@<dsnname>

[potiuk]

Only MSSQL does not include an example URI address for me to be able to. configure the database. SQLAlchemy supports multiple drivers. Which one is the best for production deployment?

@aneesh-joseph ?

[potiuk]

Only MSSQL does not include an example URI address for me to be able to. configure the database. SQLAlchemy supports multiple drivers. Which one is the best for production deployment?

I added mssql+pyodbc as this is the one a) we support in the the image b) we test during CI. Since this is rather fresh - I will leave it at that until we have an evidence that other drivers should be preferred (but then we should also change our tests and image to use them).

[potiuk]

good comments - All solved @mik-laj

[mik-laj]

@potiuk SQLAlchemy published some recommendations about preferred drivers:
PyODBC has the best Unicode support.
https://docs.sqlalchemy.org/en/14/dialects/mssql.html#driver-unicode-support

Deprecated since version 1.4: The mxODBC DBAPI is deprecated and will be removed in a future version. Please use one of the supported DBAPIs to connect to mssql.

https://docs.sqlalchemy.org/en/14/dialects/mssql.html#dialect-mssql-mxodbc-connect

pymssql is currently not included in SQLAlchemy’s continuous integration (CI) testing.

https://docs.sqlalchemy.org/en/14/dialects/mssql.html#module-sqlalchemy.dialects.mssql.pymssql

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[mik-laj]

On CI, we use a little more complex URI. This leads me to suppose that this example may be incomplete.

airflow/scripts/ci/docker-compose/backend-mssql.yml

Lines 23 to 24 in c686241

	- AIRFLOW__CORE__SQL_ALCHEMY_CONN=mssql+pyodbc://sa:Airflow123@mssql:1433/airflow?driver=ODBC+Driver+17+for+SQL+Server
	- AIRFLOW__CELERY__RESULT_BACKEND=db+mssql+pyodbc://sa:Airflow123@mssql:1433/airflow?driver=ODBC+Driver+17+for+SQL+Server

[potiuk]

This is the default one from the sqlalchemy docs (I took it from there):

https://docs.sqlalchemy.org/en/14/core/engines.html#microsoft-sql-server

We use mor complex one, but those are optional parameters, and I wouldn't dive into details of it (especially not before we have first users using 'for real' and making comments about it.

[mik-laj]

Should we add a database name to the example? I think it is a commonly used parameter. It needed to set up a database created in the paragraph above.

mssql+pyodbc://<user>:<password>@<host>
mssql+pyodbc://<user>:<password>@<host>/<database_name>

[mik-laj]

I created a PR: #18404

[ashb]

How much does this add to the docker images size btw?

[potiuk]

How much does this add to the docker images size btw?

It increases the size of the image by 4MB - from 972 MB to 976 MB (~ 0.5%) See the discussion here: https://apache-airflow.slack.com/archives/CQAMHKWSJ/p1632131354042400