Skip to content

Improve 'start-airflow' experience for users and ui developers#26341

Merged
potiuk merged 1 commit intoapache:mainfrom
potiuk:faster-start-airflow
Sep 16, 2022
Merged

Improve 'start-airflow' experience for users and ui developers#26341
potiuk merged 1 commit intoapache:mainfrom
potiuk:faster-start-airflow

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Sep 12, 2022

Since we moved the "compile-www-assets" to host (and deferred it to pre-commit to prepare and manage appropriate node virtualenv), starting airflow via start-airflow command had become slower, adding quite a few seconds always when you run the command. Also it was a bit convoluted to start dev mode - you had to stop and restart webserver and run dev asset recompilation.

This change improves the start-airflow experience in several ways:

  • the compile-www-assets pre-commit in start airflow runs in a background, daemon thread. This allows start-airflow command to continue even if asset compilation takes quite some time. By the time webserver starts and database initializes and the user launches browser it should be completed anyway.
  • the assets are not recompiled if none of the "www" files changed which for most people who run airflow without modifying the UI part will make it really fast to re-run start-airflow
  • we have a new flag --skip-assets-compilation added that will skip asset compilation even if the www files were modified
  • finally we have a new "--dev-mode" when starting airflow that will run the "dev" mode of assets compilation. And thanks to the background, daemon thread, assets compilation in dev mode will continue running as long as the start-airflow command is running

All this should naturally plug in the exisitng start-airflow usage and add new capabilities for those who develop UI (you can run dev mode now without manually restarting anything).

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member Author

potiuk commented Sep 12, 2022

cc: @bbovenzi @pierrejeambrun - apart of improving start-airflow, you might find it really useful too as it will allow to enter "dev mode" with start-airflow much more easily --dev-mode with launch yarn dev in the background as well as -d when starting webserver.

potiuk
potiuk commented Sep 12, 2022
View reviewed changes
potiuk
potiuk commented Sep 12, 2022
View reviewed changes
@potiuk potiuk requested a review from ephraimbuddy September 12, 2022 11:25
@potiuk potiuk force-pushed the faster-start-airflow branch 2 times, most recently from b30d534 to b40c72b Compare September 14, 2022 19:27
@potiuk
Copy link
Member Author

potiuk commented Sep 14, 2022

Looking forward to merging this one :). Should be good improvement for many users.

@potiuk
Copy link
Member Author

potiuk commented Sep 14, 2022

(rebased it to include PEP-563)

@potiuk potiuk force-pushed the faster-start-airflow branch 2 times, most recently from 3efcc1a to 8cf3bcf Compare September 14, 2022 21:59
jedcunningham
jedcunningham approved these changes Sep 14, 2022
View reviewed changes
@potiuk potiuk force-pushed the faster-start-airflow branch from 36083dc to ac9640c Compare September 15, 2022 13:15
@potiuk potiuk mentioned this pull request Sep 15, 2022
Merged
@potiuk potiuk force-pushed the faster-start-airflow branch from ac9640c to 9a7ec38 Compare September 15, 2022 15:38
josh-fell
josh-fell approved these changes Sep 15, 2022
View reviewed changes
Copy link
Contributor

@josh-fell josh-fell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a small comment. This will definitely be an improvement!

potiuk
potiuk commented Sep 15, 2022
View reviewed changes
@potiuk potiuk force-pushed the faster-start-airflow branch 3 times, most recently from 246827c to 4ff8b1c Compare September 15, 2022 19:01
@potiuk
Improve 'start-airflow' experience for users and ui developers
0ad03c0 
Since we moved the "compile-www-assets" to host (and deferred it
to pre-commit to prepare and manage appropriate node virtualenv),
starting airflow via `start-airflow` command had become slower,
adding quite a few seconds always when you run the command. Also
it was a bit convoluted to start dev mode - you had to stop
and restart webserver and run dev asset recompilation.

This change improves the `start-airflow` experience in several ways:

* the compile-www-assets pre-commit in start airflow runs in a
  background, daemon thread. This allows start-airflow command
  to continue even if asset compilation takes quite some time.
  By the time webserver starts and database initializes and the user
  launches browser it should be completed anyway.
* the assets are not recompiled if none of the "www" files changed which
  for most people who run airflow without modifying the UI part
  will make it really fast to re-run start-airflow
* we have a new flag --skip-assets-compilation added that will skip
  asset compilation even if the www files were modified
* finally we have a new "--dev-mode" when starting airflow that will
  run the "dev" mode of assets compilation. And thanks to the
  background, daemon thread, assets compilation in dev mode will
  continue running as long as the `start-airflow` command is running

All this should naturally plug in the exisitng start-airflow usage
and add new capabilities for those who develop UI (you can run
dev mode now without manually restarting anything).
@potiuk potiuk force-pushed the faster-start-airflow branch from 4ff8b1c to 0ad03c0 Compare September 15, 2022 20:29
@potiuk potiuk merged commit 7661f34 into apache:main Sep 16, 2022
@potiuk potiuk deleted the faster-start-airflow branch September 16, 2022 07:45
@jedcunningham jedcunningham mentioned this pull request Sep 16, 2022
Merged
ephraimbuddy pushed a commit that referenced this pull request Nov 10, 2022
@potiuk @ephraimbuddy
Improve 'start-airflow' experience for users and ui developers (#26341)
8a67569 
Since we moved the "compile-www-assets" to host (and deferred it
to pre-commit to prepare and manage appropriate node virtualenv),
starting airflow via `start-airflow` command had become slower,
adding quite a few seconds always when you run the command. Also
it was a bit convoluted to start dev mode - you had to stop
and restart webserver and run dev asset recompilation.

This change improves the `start-airflow` experience in several ways:

* the compile-www-assets pre-commit in start airflow runs in a
  background, daemon thread. This allows start-airflow command
  to continue even if asset compilation takes quite some time.
  By the time webserver starts and database initializes and the user
  launches browser it should be completed anyway.
* the assets are not recompiled if none of the "www" files changed which
  for most people who run airflow without modifying the UI part
  will make it really fast to re-run start-airflow
* we have a new flag --skip-assets-compilation added that will skip
  asset compilation even if the www files were modified
* finally we have a new "--dev-mode" when starting airflow that will
  run the "dev" mode of assets compilation. And thanks to the
  background, daemon thread, assets compilation in dev mode will
  continue running as long as the `start-airflow` command is running

All this should naturally plug in the exisitng start-airflow usage
and add new capabilities for those who develop UI (you can run
dev mode now without manually restarting anything).

(cherry picked from commit 7661f34)
ephraimbuddy pushed a commit that referenced this pull request Nov 10, 2022
@potiuk @ephraimbuddy
Improve 'start-airflow' experience for users and ui developers (#26341)
f5c3028 
Since we moved the "compile-www-assets" to host (and deferred it
to pre-commit to prepare and manage appropriate node virtualenv),
starting airflow via `start-airflow` command had become slower,
adding quite a few seconds always when you run the command. Also
it was a bit convoluted to start dev mode - you had to stop
and restart webserver and run dev asset recompilation.

This change improves the `start-airflow` experience in several ways:

* the compile-www-assets pre-commit in start airflow runs in a
  background, daemon thread. This allows start-airflow command
  to continue even if asset compilation takes quite some time.
  By the time webserver starts and database initializes and the user
  launches browser it should be completed anyway.
* the assets are not recompiled if none of the "www" files changed which
  for most people who run airflow without modifying the UI part
  will make it really fast to re-run start-airflow
* we have a new flag --skip-assets-compilation added that will skip
  asset compilation even if the www files were modified
* finally we have a new "--dev-mode" when starting airflow that will
  run the "dev" mode of assets compilation. And thanks to the
  background, daemon thread, assets compilation in dev mode will
  continue running as long as the `start-airflow` command is running

All this should naturally plug in the exisitng start-airflow usage
and add new capabilities for those who develop UI (you can run
dev mode now without manually restarting anything).

(cherry picked from commit 7661f34)
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Nov 10, 2022
@ephraimbuddy ephraimbuddy added this to the Airflow 2.4.3 milestone Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pierrejeambrun pierrejeambrun pierrejeambrun approved these changes

@josh-fell josh-fell josh-fell approved these changes

@jedcunningham jedcunningham jedcunningham approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@mik-laj mik-laj Awaiting requested review from mik-laj

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

@eladkal eladkal Awaiting requested review from eladkal

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

Assignees

No one assigned

Labels

area:dev-tools changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Milestone

Airflow 2.4.3

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk @pierrejeambrun @josh-fell @jedcunningham @ephraimbuddy