Add Advance Search Support and Integrate with Get Variables #44516

shubhamraj-git · 2024-11-30T19:35:31Z

Add the filter variables support

Request:

Response:

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

airflow/api_fastapi/core_api/routes/public/variables.py

bbovenzi · 2024-12-09T16:19:01Z

airflow/api_fastapi/common/parameters.py

+    STARTS_WITH = "starts_with"
+    ENDS_WITH = "ends_with"
+    CONTAINS = "contains"
+    EQUALS = "equals"
+    NOT_STARTS_WITH = "not_starts_with"
+    NOT_ENDS_WITH = "not_ends_with"
+    NOT_CONTAINS = "not_contains"


I wonder what's the best way to pass all these filter options to the UI so we can build the proper form for it.

we can have a dropdown and multiple lines of search can be given from user with add and delete functionality for each search options.
Similar behaviour from legacy UI

const filters = [ { key: "STARTS_WITH", label: "Starts With", value: "starts_with" }, { key: "ENDS_WITH", label: "Ends With", value: "ends_with" }, { key: "CONTAINS", label: "Contains", value: "contains" }, { key: "EQUALS", label: "Equals", value: "equals" }, { key: "NOT_STARTS_WITH", label: "Does Not Start With", value: "not_starts_with" }, { key: "NOT_ENDS_WITH", label: "Does Not End With", value: "not_ends_with" }, { key: "NOT_CONTAINS", label: "Does Not Contain", value: "not_contains" } ];

I think we need a private endpoint for that ?

Also an endpoint to list resource fields, and their types, so we can appropriately display the UI form.

For instance DAGModel has a field called is_active which is a boolean etc.. etc.... (So we can basically list fields, operators, and then compos fields + operators do construct a dynamic query)

by private, do you mean ui endpoint?

Yes, I completely agree we should have a endpoint to list the applicable columns.

Yes. We need an endpoint to list possible operations (in, or, and, >= etc...) and the columns that can be used and maybe the 'type' as well. (boolean, number, etc.)

pierrejeambrun

Nice

pierrejeambrun · 2024-12-10T16:54:27Z

airflow/api_fastapi/common/parameters.py

    return depends_search


+class SearchPatternEnum(Enum):


Maybe we should reuse FilterOptionEnum and rename it to a more generic name. Maybe move it to the DB package with something like ORMOperatorEnum or something else.

DB operations supported by the query parameters and the one from advanced search will basically be very close. (and this will avoid duplication).

As we do for states task_states dag_run_states, etc.. we can split FilterOperator AdvancedSearchOperator, etc...

FilterOptionEnum have bit different types of support, such as numeric, etc.
Mixing both could create confusion?
Maybe i can differentiate this with filter, such that it should actually look like advance search, like chnaging filter param to advance_search in APIs.

Also do you mean something like:
Parent : child?
ORMOperator -> FilterOperator, AdvancedSearchOperator ?

I think the enumeration of all ORM operation could live in the db module of FastAPI.

Then other enums FilterOperator, AdvancedSearchOperator can use that, but basically Filter and Search are just query params, the difference is the db operator that is used airflow/utils/states.py combines enums as well, could be interesting to take a look.

That's not of big importance at this stage I think

pierrejeambrun · 2024-12-10T16:56:37Z

airflow/api_fastapi/common/parameters.py

+    STARTS_WITH = "starts_with"
+    ENDS_WITH = "ends_with"
+    CONTAINS = "contains"
+    EQUALS = "equals"
+    NOT_STARTS_WITH = "not_starts_with"
+    NOT_ENDS_WITH = "not_ends_with"
+    NOT_CONTAINS = "not_contains"


I think we need a private endpoint for that ?

pierrejeambrun · 2024-12-10T17:00:09Z

airflow/api_fastapi/common/parameters.py

+        pattern_map = {
+            SearchPatternEnum.STARTS_WITH: f"{self.value}%",
+            SearchPatternEnum.ENDS_WITH: f"%{self.value}",
+            SearchPatternEnum.CONTAINS: f"%{self.value}%",
+            SearchPatternEnum.EQUALS: self.value,
+            SearchPatternEnum.NOT_STARTS_WITH: f"{self.value}%",
+            SearchPatternEnum.NOT_ENDS_WITH: f"%{self.value}",
+            SearchPatternEnum.NOT_CONTAINS: f"%{self.value}%",
+        }


All that is really focused around search (text search) while I imagined advanced search to actually be on any field and any db operation.

For instance I want a custom query to display dags with the specific status that is SUCCESS and with the dag_id that ends with something and is_paused True.

Basically serializing and deserializing via a query parameter &query='xxxxx'&query='' dyanamic Filters.

We can do that,
Maybe I might be wrong, I would need some clarity about the differences between filter and advance_search operations.
If advance_search starts taking for booleans or special types, wouldn't it contradict the work of filters?

I think what we eventually want is the ability to create a full custom query such as:

I want dags that are in success or running state, and that have a dag_id that starts with dag_id_prefix and with at least 1 tag among [tag1, tag2].

Its basically being able to create a custom query.

Also I think we need to be able to support 'AND' and 'OR' and maybe composition (cd1 | cd2) & (cd3).

pierrejeambrun · 2024-12-10T17:01:01Z

airflow/api_fastapi/common/parameters.py

+    STARTS_WITH = "starts_with"
+    ENDS_WITH = "ends_with"
+    CONTAINS = "contains"
+    EQUALS = "equals"
+    NOT_STARTS_WITH = "not_starts_with"
+    NOT_ENDS_WITH = "not_ends_with"
+    NOT_CONTAINS = "not_contains"


Also an endpoint to list resource fields, and their types, so we can appropriately display the UI form.

For instance DAGModel has a field called is_active which is a boolean etc.. etc.... (So we can basically list fields, operators, and then compos fields + operators do construct a dynamic query)

pierrejeambrun

I think we need to extend this to 'any' column of the resource and 'any' operation. (>=, text contains start with end with etc.., IN, array contain etc..)

Maybe the serialization form should be something like

?query="((key, START_WITH, 'something') or (value, EQUAL, 'some_value')) and (value, END_WITH, 'suffix')"

Maybe we don't need OR | AND support and expressions (nested boolean operators). Maybe we don't need full flexibility about fields and filters used. Because if it's just some predetermined filters, we can just develop them as we do currently and add an endpoint to retrieve them (types, possible values etc...) and display that directly in the UI, constructing the current query normaly with in the front-end ?state=success&active=true.

Maybe @bbovenzi can confirm the use case.

pierrejeambrun · 2024-12-13T15:19:57Z

airflow/api_fastapi/common/parameters.py

+    STARTS_WITH = "starts_with"
+    ENDS_WITH = "ends_with"
+    CONTAINS = "contains"
+    EQUALS = "equals"
+    NOT_STARTS_WITH = "not_starts_with"
+    NOT_ENDS_WITH = "not_ends_with"
+    NOT_CONTAINS = "not_contains"


Yes. We need an endpoint to list possible operations (in, or, and, >= etc...) and the columns that can be used and maybe the 'type' as well. (boolean, number, etc.)

pierrejeambrun · 2024-12-13T15:24:32Z

airflow/api_fastapi/common/parameters.py

+        pattern_map = {
+            SearchPatternEnum.STARTS_WITH: f"{self.value}%",
+            SearchPatternEnum.ENDS_WITH: f"%{self.value}",
+            SearchPatternEnum.CONTAINS: f"%{self.value}%",
+            SearchPatternEnum.EQUALS: self.value,
+            SearchPatternEnum.NOT_STARTS_WITH: f"{self.value}%",
+            SearchPatternEnum.NOT_ENDS_WITH: f"%{self.value}",
+            SearchPatternEnum.NOT_CONTAINS: f"%{self.value}%",
+        }


I think what we eventually want is the ability to create a full custom query such as:

I want dags that are in success or running state, and that have a dag_id that starts with dag_id_prefix and with at least 1 tag among [tag1, tag2].

Its basically being able to create a custom query.

Also I think we need to be able to support 'AND' and 'OR' and maybe composition (cd1 | cd2) & (cd3).

pierrejeambrun · 2024-12-13T15:25:52Z

airflow/api_fastapi/common/parameters.py

+# Variables
+VARIABLE_COLUMN_MAP = {
+    "key": Variable.key,
+    "value": Variable._val,


I think we need to use the val proxy not directly _val

jscheffl · 2025-01-01T00:04:04Z

Note: As PR #45312 has been merged, the code formatting rules have changed for new UI. Please rebase and re-run pre-commit checks to ensure that formatting in folder airflow/ui is adjusted.

github-actions · 2025-02-15T00:15:02Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

shubhamraj-git added 3 commits November 30, 2024 09:03

add key search

4a900a6

advance search

895e647

refactor

62357ed

boring-cyborg bot added the area:UI Related to UI/UX. For Frontend Developers. label Nov 30, 2024

multi filters support

857075e

jscheffl reviewed Dec 4, 2024

View reviewed changes

airflow/api_fastapi/core_api/routes/public/variables.py Outdated Show resolved Hide resolved

shubhamraj-git and others added 4 commits December 7, 2024 00:32

Merge branch 'main' into filter-variables-support

9959eef

Merge branch 'main' into filter-variables-support

74b3a9e

refactor to string

d78c164

Merge branch 'main' into filter-variables-support

c32d1fc

shubhamraj-git changed the title ~~Add the filter variables support~~ Added Advance Search Support and Integrated with Get Variables Dec 7, 2024

shubhamraj-git requested a review from jscheffl December 7, 2024 18:48

shubhamraj-git marked this pull request as ready for review December 7, 2024 18:48

shubhamraj-git requested review from bbovenzi, ephraimbuddy, pierrejeambrun and ryanahamilton as code owners December 7, 2024 18:48

shubhamraj-git changed the title ~~Added Advance Search Support and Integrated with Get Variables~~ Add Advance Search Support and Integrated with Get Variables Dec 7, 2024

shubhamraj-git changed the title ~~Add Advance Search Support and Integrated with Get Variables~~ Add Advance Search Support and Integrate with Get Variables Dec 7, 2024

Merge branch 'main' into filter-variables-support

b523965

bbovenzi reviewed Dec 9, 2024

View reviewed changes

shubhamraj-git requested a review from bbovenzi December 9, 2024 17:02

Merge branch 'main' into filter-variables-support

517ec9a

pierrejeambrun reviewed Dec 10, 2024

View reviewed changes

shubhamraj-git requested a review from pierrejeambrun December 12, 2024 17:21

pierrejeambrun reviewed Dec 13, 2024

View reviewed changes

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Feb 15, 2025

github-actions bot closed this Feb 20, 2025

shubhamraj-git deleted the filter-variables-support branch September 7, 2025 19:37

Add Advance Search Support and Integrate with Get Variables #44516

Add Advance Search Support and Integrate with Get Variables #44516

Uh oh!

Conversation

shubhamraj-git commented Nov 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shubhamraj-git Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shubhamraj-git Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jscheffl commented Jan 1, 2025

Uh oh!

github-actions bot commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shubhamraj-git commented Nov 30, 2024 •

edited

Loading

shubhamraj-git Dec 9, 2024 •

edited

Loading

pierrejeambrun Dec 10, 2024 •

edited

Loading

pierrejeambrun Dec 13, 2024 •

edited

Loading

pierrejeambrun Dec 10, 2024 •

edited

Loading

pierrejeambrun Dec 10, 2024 •

edited

Loading

shubhamraj-git Dec 10, 2024 •

edited

Loading

pierrejeambrun Dec 10, 2024 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading

pierrejeambrun Dec 13, 2024 •

edited

Loading