Skip to content

Conversation

@jscheffl
Copy link
Contributor

@jscheffl jscheffl commented Oct 14, 2022

This PR adds a trigger UI based on ideas and discussion from AIP-50 Trigger DAG UI Extension with Flexible User Form Concept based on current FAB UI infrastructure. (Note: Initially it was a PoC PR, now functionally complete).

Due to the feedback received on AIP-50 the PR is based on the Params toolings already available and leverages with minimal changes 80% of desired scope. Also the parallel implementation of #27805 was integrated.

Coverage/Scope of this PR:

  • AIP-50 Part 1+3 (fully covered): Specifying Trigger UI on DAG Level + Standard Implementation for Forms
    • By defining params or usage of existing params dict a form as UI is generated dynamically
    • Existing JSON schema is used as good as possible with slight extensions for purpose of form rendering
    • Compared to the initial idea to have a "new" form as dialog, the proposed PR now extends the existing JSON entry UI by:
      • If params are defined on a DAG a form is generated
      • If no params are defined the previous JSON based entry is presented
      • Why? Benefits? Part 2 was implemented slightly different than proposed in AIP-50 - see below.
    • Small derivations compared to current AIP-50:
      • Field type and rendering of field is not only made depending on Schema but also auto-discovered for common data types (string, number, list, object, boolean) which adds an easy entry for existing DAGs w/o needs of change == for free.
      • Field type hidden was implemented by using the JSON schema attribute constant which assumes that only one constant value is permitted, no additional flag needed.
      • Advanced section of a entry form was generalized allowing to define any/multiple sections of the form (not limited to one "Advanced")
  • AIP-50 Part 2: not included, will be a future PR.
  • AIP-50 Part 4 (fully covered): Examples:
    • Two examples added within this PR
      • One simple example as "showcase"
      • One "Tutorial" with a lot of examples for large coverage and template for users to be able to copy
  • AIP-50 Part 5 (fully covered): Documentation
    • Extended the existing page Params to add details about UI, options and supported parameters etc.

related: #26215 #27805
closes: #11054

FYI @clellmann @wolfdn @AutomationDev85

@boring-cyborg boring-cyborg bot added area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Oct 14, 2022
@jscheffl
Copy link
Contributor Author

Note that Pytests are not adjusted and are even failing for the WIP PR at the moment, will be fixed before the PR leaves the WIP phase. Still the function is working and can be inspected.

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 29, 2022
@pierrejeambrun
Copy link
Member

pierrejeambrun commented Nov 29, 2022

@jens-scheffler-bosch
I couldn’t find time to try this yet. I will take a look this week, thanks for this preview :)

@pierrejeambrun pierrejeambrun removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 29, 2022
@jscheffl
Copy link
Contributor Author

@jens-scheffler-bosch I couldn’t find time to try this yet. I will take a look this week, thanks for this preview :)

Great, looing forward to complete the missing pieces if the direction/proposal is accepted!

@pierrejeambrun
Copy link
Member

@jens-scheffler-bosch Tested that locally and looks very cool. I think it is worth pushing that further with the addition you mentioned. (Doc, tests, cleaning the code etc.)

I would mention that it would be great if this could integrate with the newly added retrieval of recent config feature #27805.

@pierrejeambrun
Copy link
Member

To address your questions, (imho):

  • What you mentioned is I believe a good start to have something mergeable. (test, doc, etc.)
  • I believe this could make it in the next minor release -> 2.6.0
  • Adding this feature via FAB is perfectly fine for me. AIP-38 is a long term ongoing work. At the moment I am not aware of a change that would bring the trigger page into 'react world' so we might have this under FAB for quite a while before it gets updated.
  • Part 2: We could fetch these forms on the trigger.html page, not before (dag level). And ask the person to chose which form/format he wants to use via extra buttons/select. (just an idea to not have to get all forms for all dags on the home page).
  • I know there has been a few questions about this behavior (slack) and we might benefit from updating it. Today I see it like this, a dag 'can' always run without config, this is what happens when it is not manually triggered. Manually triggering the dag without config would mimic a 'scheduled' run in terms of config i.e not using params 'at all'. Wondering what others think about this. Maybe the empty dict is misleading and a plain 'None' would be more appropriate. We woudn't want scheduled run to use default params values.

@jscheffl jscheffl changed the title [WIP/Preview] Feature/26215 Proposal for AIP-50 Trigger UI based on FAB Feature/26215 AIP-50 Trigger UI based on FAB Dec 23, 2022
@jscheffl
Copy link
Contributor Author

Hi @pierrejeambrun - thanks again for the review! I now closed the gaps and made the pipeline green - functional-wise the AIP-50 implementation is completed now. Now time for a final review, then (in my view) ready to be merged.

Recent commits changed:

  • Added part 2 of the AIP-50 implementation
  • Added documentation
  • Fixed unit tests and needed some commits to make pipeline green

(Default) value that should be displayed when showing/loading the form.</li>
<li>Note: If you have elements changing a value, call <code>updateJSONconf()</code>.</li>
</ul>
Example: <code>&lt;input name='{name}' value='{value}' onchange='updateJSONconf()' /&gt;</code>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than needing custom form's to use onchange='updateJSONconf()' can we change the even to pick up all inputs under the form automatically?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to apply this comment but when looking to my own example I realized that a "generic" solution might be hard. I fear this generates more problems than it helps users, because adding the event listeners globally might generate side effects when a custom form has other event listeners and the order is not right.

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python side looks okay to me now (though lets separate out trigger_dag_url change so we can continue discussing that without blocking this.)

I'd like @bbovenzi To take a look at the JS side of things before merging.

Copy link
Contributor Author

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adedd screen shot snippets for Trigger form

Comment on lines +32 to +43
<td class="col-lg-2">
<label for="element_{{ form_key }}" control-label="">
{%- if "title" in form_details.schema and form_details.schema.title -%}
{{ form_details.schema.title }}
{%- else -%}
{{ form_key }}
{%- endif -%}
{%- if form_details.schema and form_details.schema.type and not "null" in form_details.schema.type and not "boolean" in form_details.schema.type -%}
<strong style="color: red"> *</strong>
{%- endif -%}
: </label>
</td>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block renders the form label, either using the key or the display label if given. A red star is added if field is required to have content before submit:
image

Comment on lines +48 to +52
<label for="element_{{ form_key }}" control-label="">
<input class="switch-input" name="element_{{ form_key }}" id="element_{{ form_key }}" type="checkbox"
{%- if form_details.value %} checked="checked"{%- endif -%}/>
<span class="switch" aria-hidden="true"></span>
</label>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines render a boolean switch.
image

Comment on lines +54 to +58
<div class="input-group datetime datetimepicker">
<span class="input-group-addon"><span class="material-icons cursor-hand">calendar_today</span></span>
<input class="form-control" name="element_{{ form_key }}" id="element_{{ form_key }}" type="text" valuetype="date"
{%- if not "null" in form_details.schema.type %} required=""{%- endif %} value="{% if form_details.value %}{{ form_details.value }}{% endif %}" />
</div>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines render a date-time picker (Unfortunately I'm not able to screen-capture the pupp of the date/time picker but it is the same like on other UI parts:
image

Comment on lines +60 to +64
<div class="input-group datetime datepicker">
<span class="input-group-addon"><span class="material-icons cursor-hand">calendar_today</span></span>
<input class="form-control" name="element_{{ form_key }}" id="element_{{ form_key }}" type="text" valuetype="date"
{%- if not "null" in form_details.schema.type %} required=""{%- endif %} value="{% if form_details.value %}{{ form_details.value }}{% endif %}" />
</div>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Date picker (without time):
image

Comment on lines +66 to +71
<div class="input-group datetime timepicker">
<span class="input-group-addon"><span class="material-icons cursor-hand">calendar_today</span></span>
<input class="form-control" name="element_{{ form_key }}" id="element_{{ form_key }}" type="text" valuetype="date"
{%- if not "null" in form_details.schema.type %} required=""{%- endif %}
value="{% if form_details.value %}{{ form_details.value[0:2] }}00-01-01 {{ form_details.value }}{% endif %}" />
</div>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time picker (w/o date):
image

Comment on lines +99 to +103
<input class="form-control" name="element_{{ form_key }}" id="element_{{ form_key }}" placeholder="" type="text"
value="{% if form_details.value %}{{ form_details.value }}{% endif %}"
{%- if form_details.schema and form_details.schema.minLength %} minlength="{{ form_details.schema.minLength }}"{% endif %}
{%- if form_details.schema and form_details.schema.maxLength %} maxlength="{{ form_details.schema.maxLength }}"{% endif %}
{%- if form_details.schema and form_details.schema.type and not "null" in form_details.schema.type %} required=""{% endif %} />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...aaaand final leftover, if not any specific type then a simple text field is rendered incl. optional min/max length:
image

{%- if form_details.schema and form_details.schema.type and not "null" in form_details.schema.type %} required=""{% endif %} />
{% endif %}
{% if form_details.description -%}
<span class="help-block">{{ form_details.description }}</span>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the help text/description either as markup (if it is HTML) or plain text:
image

Comment on lines -52 to 140
<div class="col-md-2">
<label for="recent_configs">Recent Configurations</label>
<label for="recent_configs">Select Recent Configurations</label>
<select class="form-control" name="recent_configs" id="recent_configs">
<option value="">empty config</option>
{% for conf in recent_confs %}
<option value="{{ conf }}">{{ conf }}</option>
<option value="{{ conf }}">Default parameters</option>
{% for run_id, recent_conf in recent_confs.items() %}
<option value="{{ recent_conf }}">{{ run_id }}: {{ recent_conf }}</option>
{% endfor %}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is the select of the recent configs which can be applied to the form:
image

Comment on lines +172 to +177
<div class="panel-heading">
<h4 class="panel-title">
<a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion_section_{{ form_section | lower() | replace(' ', '_') }}" href="#accordion_section_{{ form_section | lower() | replace(' ', '_') }}_href"
data-original-title="" title="" id="{{ form_section | lower() | replace(' ', '_') }}_toggle">{{ form_section }}<span class="caret"></span></a>
</h4>
</div>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the form defines multiple sections, this generate a new section header - each section is folded by default and can be un-folded:
image

Comment on lines +196 to +209
<div class="panel-group" id="accordion_json_conf">
<div class="panel panel-default">
<div class="panel-heading">
<h4 class="panel-title">
<a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion_json_conf" href="#accordion_json_conf_href" data-original-title="" title="" id="generated_json_toggle">Generated Configuration JSON<span class="caret"></span></a>
</h4>
</div>
<div id="accordion_json_conf_href" class="panel-collapse collapse">
<div class="panel-body">
<textarea name="conf" id="json">{{ conf }}</textarea>
</div>
</div>
</div>
</div>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below the form the generated JSON can be seen - if you want to re-use this for future reference:
image

@jscheffl
Copy link
Contributor Author

Hi @ashb && @bbovenzi applied feedback, resolved merge conflict, added screen snippets, reverted part 2 of AIP-50 - ready for next stage of review!

Copy link
Contributor

@bbovenzi bbovenzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I played around with the example dags too. They were really helpful.

@bbovenzi bbovenzi merged commit 5e470c1 into apache:main Jan 30, 2023
@pierrejeambrun pierrejeambrun added changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) AIP-50 Trigger DAG UI user Form labels Feb 27, 2023
@mayeulk
Copy link
Contributor

mayeulk commented Mar 29, 2023

Thank you for this!!
Is there a way to have this new function in a docker stack?
What would be the process? Taking the version of the latest commit from main (like: 26 January, aaf54fc )? (with git pull?)
Or is it planned to be in 2.5.3 or 2.5.4?
(I cannot see it in https://github.com/apache/airflow/milestone/70 ).

Or the only reasonable way would be waiting until it goes into a stable release?

I'm running Airflow 2.5.1 with Docker Compose in dev mode, following:
https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
(I can see there is a slightly different docker-compose.yaml for 2.5.2)

Thank you.
Mayeul

@pierrejeambrun
Copy link
Member

pierrejeambrun commented Mar 29, 2023

Hello @mayeulk,

  • Main is a 'development' branch, not stable
  • There are in progress changes not finalized that you do not want in your production env
  • Most importantly, there are potentially breaking changes marked for Airflow 3.0.0 that could break your install.

I would say that you could cherry pick only the changes related to AIP-50 and apply that to your own version (if you really can't wait). This is a new feature, so it will be part of the next minor release. 2.6.0 is just around the corner as it will most probably be the next released version of airflow. (right after 2.5.3)

I would recommend to wait just a little bit longer. 😄

@mayeulk
Copy link
Contributor

mayeulk commented Mar 30, 2023

Hello @pierrejeambrun,
Thank you so much for the swift and detailed answer; it helps me a lot!
I'll keep an eye on https://airflow.apache.org/announcements/ and the 2.6.0 milestone https://github.com/apache/airflow/milestone/67

@jscheffl jscheffl deleted the feature/26215-poc-on-fab-dag-trigger-ui branch October 28, 2023 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIP-50 Trigger DAG UI user Form area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improvement: Replace 'Configuration JSON' free-text area with key-value form in Trigger DAG UI

7 participants