Skip to content

Prepare Breeze2 for prime time :) #22713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Apr 3, 2022
Merged

Prepare Breeze2 for prime time :) #22713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ jobs:
- name: "Free space"
run: airflow-freespace
- name: "Build CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
run: Breeze2 build-ci-image
run: Breeze2 build-image
- name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_push_ci_images.sh
- name: >
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ jobs:
run: airflow-freespace
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Build CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
run: Breeze2 build-ci-image
run: Breeze2 build-image
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Push CI image ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_push_ci_images.sh
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ default_stages: [commit, push]
default_language_version:
# force all unspecified python hooks to run python3
python: python3
minimum_pre_commit_version: "1.20.0"
minimum_pre_commit_version: "2.0.0"
repos:
- repo: meta
hooks:
Expand Down
21 changes: 0 additions & 21 deletions BREEZE.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1336,13 +1336,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.

--skip-installing-airflow-providers-from-sources
By default 'pip install' in Airflow 2.0 installs only the provider packages that
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --use-packages-from-dist flag.

--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
Expand Down Expand Up @@ -2031,13 +2024,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.

--skip-installing-airflow-providers-from-sources
By default 'pip install' in Airflow 2.0 installs only the provider packages that
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --use-packages-from-dist flag.

--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
Expand Down Expand Up @@ -2632,13 +2618,6 @@ This is the current syntax for `./breeze <./breeze>`_:
--image-tag TAG
Additional tag in the image.

--skip-installing-airflow-providers-from-sources
By default 'pip install' in Airflow 2.0 installs only the provider packages that
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --use-packages-from-dist flag.

--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,7 +644,7 @@ and not packaged together with the core, unless you set ``INSTALL_PROVIDERS_FROM
variable to ``true``.

In Breeze - which is a development environment, ``INSTALL_PROVIDERS_FROM_SOURCES`` variable is set to true,
but you can add ``--skip-installing-airflow-providers-from-sources`` flag to Breeze to skip installing providers when
but you can add ``--install-providers-from-sources=true`` flag to Breeze to skip installing providers when
Copy link
Contributor

@eladkal eladkal Apr 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be True?

Copy link
Member Author

@potiuk potiuk Apr 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dockls are not yet updated. I leave it for the switch :) (next PR)

Copy link
Member Author

@potiuk potiuk Apr 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be quite many changes in the docs :)

building the images.

One watch-out - providers are still always installed (or rather available) if you install airflow from
Expand Down
50 changes: 26 additions & 24 deletions Dockerfile.ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -640,10 +640,11 @@ export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then

echo
echo "Airflow home: ${AIRFLOW_HOME}"
echo "Airflow sources: ${AIRFLOW_SOURCES}"
echo "Airflow core SQL connection: ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"

echo "${COLOR_BLUE}Running Initialization. Your basic configuration is:${COLOR_RESET}"
echo
echo " * ${COLOR_BLUE}Airflow home:${COLOR_RESET} ${AIRFLOW_HOME}"
echo " * ${COLOR_BLUE}Airflow sources:${COLOR_RESET} ${AIRFLOW_SOURCES}"
echo " * ${COLOR_BLUE}Airflow core SQL connection:${COLOR_RESET} ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"
echo

RUN_TESTS=${RUN_TESTS:="false"}
Expand All @@ -653,7 +654,7 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
if [[ ${USE_AIRFLOW_VERSION} == "" ]]; then
export PYTHONPATH=${AIRFLOW_SOURCES}
echo
echo "Using already installed airflow version"
echo "${COLOR_BLUE}Using airflow version from current sources${COLOR_RESET}"
echo
if [[ -d "${AIRFLOW_SOURCES}/airflow/www/" ]]; then
pushd "${AIRFLOW_SOURCES}/airflow/www/" >/dev/null
Expand All @@ -667,38 +668,38 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
mkdir -p "${AIRFLOW_SOURCES}"/tmp/
elif [[ ${USE_AIRFLOW_VERSION} == "none" ]]; then
echo
echo "Skip installing airflow - only install wheel/tar.gz packages that are present locally"
echo "${COLOR_BLUE}Skip installing airflow - only install wheel/tar.gz packages that are present locally.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
elif [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
echo
echo "Install airflow from wheel package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers."
echo "${COLOR_BLUE}Install airflow from wheel package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
install_airflow_from_wheel "[${AIRFLOW_EXTRAS}]"
uninstall_providers
elif [[ ${USE_AIRFLOW_VERSION} == "sdist" ]]; then
echo
echo "Install airflow from sdist package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers."
echo "${COLOR_BLUE}Install airflow from sdist package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
install_airflow_from_sdist "[${AIRFLOW_EXTRAS}]"
uninstall_providers
else
echo
echo "Install airflow from PyPI without extras"
echo "${COLOR_BLUE}Install airflow from PyPI without extras"
echo
install_released_airflow_version "${USE_AIRFLOW_VERSION}"
fi
if [[ ${USE_PACKAGES_FROM_DIST=} == "true" ]]; then
echo
echo "Install all packages from dist folder"
echo "${COLOR_BLUE}Install all packages from dist folder"
if [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
echo "(except apache-airflow)"
fi
if [[ ${PACKAGE_FORMAT} == "both" ]]; then
echo
echo "${COLOR_RED}ERROR:You can only specify 'wheel' or 'sdist' as PACKAGE_FORMAT not 'both'${COLOR_RESET}"
echo "${COLOR_RED}ERROR:You can only specify 'wheel' or 'sdist' as PACKAGE_FORMAT not 'both'.${COLOR_RESET}"
echo
exit 1
fi
Expand Down Expand Up @@ -782,7 +783,7 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then

cd "${AIRFLOW_SOURCES}"

if [[ ${START_AIRFLOW:="false"} == "true" ]]; then
if [[ ${START_AIRFLOW:="false"} == "true" || ${START_AIRFLOW} == "True" ]]; then
export AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS}
export AIRFLOW__CORE__LOAD_EXAMPLES=${LOAD_EXAMPLES}
# shellcheck source=scripts/in_container/bin/run_tmux
Expand Down Expand Up @@ -998,10 +999,11 @@ export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then

echo
echo "Airflow home: ${AIRFLOW_HOME}"
echo "Airflow sources: ${AIRFLOW_SOURCES}"
echo "Airflow core SQL connection: ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"

echo "${COLOR_BLUE}Running Initialization. Your basic configuration is:${COLOR_RESET}"
echo
echo " * ${COLOR_BLUE}Airflow home:${COLOR_RESET} ${AIRFLOW_HOME}"
echo " * ${COLOR_BLUE}Airflow sources:${COLOR_RESET} ${AIRFLOW_SOURCES}"
echo " * ${COLOR_BLUE}Airflow core SQL connection:${COLOR_RESET} ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"
echo

RUN_TESTS=${RUN_TESTS:="false"}
Expand All @@ -1011,7 +1013,7 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
if [[ ${USE_AIRFLOW_VERSION} == "" ]]; then
export PYTHONPATH=${AIRFLOW_SOURCES}
echo
echo "Using already installed airflow version"
echo "${COLOR_BLUE}Using airflow version from current sources${COLOR_RESET}"
echo
if [[ -d "${AIRFLOW_SOURCES}/airflow/www/" ]]; then
pushd "${AIRFLOW_SOURCES}/airflow/www/" >/dev/null
Expand All @@ -1025,38 +1027,38 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
mkdir -p "${AIRFLOW_SOURCES}"/tmp/
elif [[ ${USE_AIRFLOW_VERSION} == "none" ]]; then
echo
echo "Skip installing airflow - only install wheel/tar.gz packages that are present locally"
echo "${COLOR_BLUE}Skip installing airflow - only install wheel/tar.gz packages that are present locally.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
elif [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
echo
echo "Install airflow from wheel package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers."
echo "${COLOR_BLUE}Install airflow from wheel package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
install_airflow_from_wheel "[${AIRFLOW_EXTRAS}]"
uninstall_providers
elif [[ ${USE_AIRFLOW_VERSION} == "sdist" ]]; then
echo
echo "Install airflow from sdist package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers."
echo "${COLOR_BLUE}Install airflow from sdist package with [${AIRFLOW_EXTRAS}] extras but uninstalling providers.${COLOR_RESET}"
echo
uninstall_airflow_and_providers
install_airflow_from_sdist "[${AIRFLOW_EXTRAS}]"
uninstall_providers
else
echo
echo "Install airflow from PyPI without extras"
echo "${COLOR_BLUE}Install airflow from PyPI without extras"
echo
install_released_airflow_version "${USE_AIRFLOW_VERSION}"
fi
if [[ ${USE_PACKAGES_FROM_DIST=} == "true" ]]; then
echo
echo "Install all packages from dist folder"
echo "${COLOR_BLUE}Install all packages from dist folder"
if [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
echo "(except apache-airflow)"
fi
if [[ ${PACKAGE_FORMAT} == "both" ]]; then
echo
echo "${COLOR_RED}ERROR:You can only specify 'wheel' or 'sdist' as PACKAGE_FORMAT not 'both'${COLOR_RESET}"
echo "${COLOR_RED}ERROR:You can only specify 'wheel' or 'sdist' as PACKAGE_FORMAT not 'both'.${COLOR_RESET}"
echo
exit 1
fi
Expand Down Expand Up @@ -1140,7 +1142,7 @@ if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then

cd "${AIRFLOW_SOURCES}"

if [[ ${START_AIRFLOW:="false"} == "true" ]]; then
if [[ ${START_AIRFLOW:="false"} == "true" || ${START_AIRFLOW} == "True" ]]; then
export AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS}
export AIRFLOW__CORE__LOAD_EXAMPLES=${LOAD_EXAMPLES}
# shellcheck source=scripts/in_container/bin/run_tmux
Expand Down
2 changes: 1 addition & 1 deletion airflow/www/ask_for_recompile_assets_if_needed.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ NO_COLOR='\033[0m'
md5sum=$(find package.json yarn.lock static/css static/js -type f | sort | xargs md5sum)
old_md5sum=$(cat "${MD5SUM_FILE}" 2>/dev/null || true)
if [[ ${old_md5sum} != "${md5sum}" ]]; then
if [[ ${START_AIRFLOW:="false"} == "true" && ${USE_AIRFLOW_VERSION:=} == "" ]]; then
if [[ ( ${START_AIRFLOW:="false"} == "true" || ${START_AIRFLOW} == "True" ) && ${USE_AIRFLOW_VERSION:=} == "" ]]; then
echo
echo -e "${YELLOW}Recompiling assets as they have changed and you need them for 'start_airflow' command${NO_COLOR}"
echo
Expand Down
12 changes: 0 additions & 12 deletions breeze
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,11 +1047,6 @@ function breeze::parse_arguments() {
echo "Extras : ${AIRFLOW_EXTRAS}"
shift 2
;;
--skip-installing-airflow-providers-from-sources)
export INSTALL_PROVIDERS_FROM_SOURCES="false"
echo "Install all Airflow Providers: false"
shift
;;
--additional-extras)
export ADDITIONAL_AIRFLOW_EXTRAS="${2}"
echo "Additional extras : ${ADDITIONAL_AIRFLOW_EXTRAS}"
Expand Down Expand Up @@ -2700,13 +2695,6 @@ ${FORMATTED_DEFAULT_PROD_EXTRAS}
--image-tag TAG
Additional tag in the image.

--skip-installing-airflow-providers-from-sources
By default 'pip install' in Airflow 2.0 installs only the provider packages that
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --use-packages-from-dist flag.

--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
Expand Down
2 changes: 1 addition & 1 deletion breeze-complete
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ postgres-version: mysql-version: mssql-version:
version-suffix-for-pypi: version-suffix-for-svn:
additional-extras: additional-python-deps: additional-dev-deps: additional-runtime-deps: image-tag:
disable-mysql-client-installation disable-mssql-client-installation constraints-location: disable-pip-cache install-from-docker-context-files
additional-extras: additional-python-deps: disable-pypi-when-building skip-installing-airflow-providers-from-sources
additional-extras: additional-python-deps: disable-pypi-when-building
dev-apt-deps: additional-dev-apt-deps: dev-apt-command: additional-dev-apt-command: additional-dev-apt-env:
runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-runtime-apt-command: additional-runtime-apt-env:
load-default-connections load-example-dags
Expand Down
7 changes: 4 additions & 3 deletions dev/breeze/setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,15 @@ package_dir=
packages = find:
install_requires =
click
inputimeout
pendulum
psutil
pytest
pytest-xdist
pyyaml
requests
rich
rich_click
requests
psutil
inputimeout

[options.packages.find]
where=src
Expand Down
21 changes: 21 additions & 0 deletions dev/breeze/src/airflow_breeze/branch_defaults.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,27 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Default configuration for this branch. Those two variables below
should be the only one that should be changed when we branch off
different airflow branch.

This file is different in every branch (`main`, `vX_Y_test' of airflow)
The _stable branches have the same values as _test branches.

Examples:

main:

AIRFLOW_BRANCH = "main"
DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = "constraints-main"

v2-2-test:

AIRFLOW_BRANCH = "v2-2-test"
DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = "constraints-2-2"

"""

AIRFLOW_BRANCH = "main"
DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH = "constraints-main"
Loading