Expand KPO template_fields, fix Spark k8s operator tests #46268
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
insomnes
requested review from
hussein-awala and
jedcunningham
as code owners
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
Contributor
Author
|
I was interested in this feature so I've bring back staled PR with some extra tests, maybe we can succeed with it this time |
Contributor
Author
|
I think, I've found the bug in spark k8s operator test code. Will check and try to fix for affected test and bring the bug issue to fix any other occurrences |
insomnes
force-pushed
the
kpo-name-template
branch
from
2aeff58 to
0e5df15
Compare
Contributor
Author
|
The mentioned bug in the spark k8s operator test is related to mocking and new changes. Previously, tests were failed because the new KPO This led me to understand that calls to Any method call on instance of I believe that all affected tests are fixed now. So there is no need for a new issue. |
insomnes
force-pushed
the
kpo-name-template
branch
from
0e5df15 to
4016c3d
Compare
insomnes
changed the title
insomnes
changed the title
insomnes
changed the title
shahar1
approved these changes
Feb 1, 2025
Contributor
|
LGTM! |
amoghrajesh
pushed a commit
to astronomer/airflow
that referenced
this pull request
Feb 3, 2025
* Add name and hostname to KPO template_fields * Add fetch container mock to name normalization test * Fix bugged tests * Run execute in test_pod_name to get validation fail * Add long name case to KPO unit tests
dabla
pushed a commit
to dabla/airflow
that referenced
this pull request
Feb 3, 2025
* Add name and hostname to KPO template_fields * Add fetch container mock to name normalization test * Fix bugged tests * Run execute in test_pod_name to get validation fail * Add long name case to KPO unit tests
niklasr22
pushed a commit
to niklasr22/airflow
that referenced
this pull request
Feb 8, 2025
* Add name and hostname to KPO template_fields * Add fetch container mock to name normalization test * Fix bugged tests * Run execute in test_pod_name to get validation fail * Add long name case to KPO unit tests
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
* Add name and hostname to KPO template_fields * Add fetch container mock to name normalization test * Fix bugged tests * Run execute in test_pod_name to get validation fail * Add long name case to KPO unit tests
brouberol
added a commit
to brouberol/airflow
that referenced
this pull request
Mar 17, 2025
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
brouberol
added a commit
to brouberol/airflow
that referenced
this pull request
Mar 17, 2025
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
brouberol
added a commit
to brouberol/airflow
that referenced
this pull request
Mar 17, 2025
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
jscheffl
pushed a commit
that referenced
this pull request
Mar 22, 2025
#47864) I would like to propose adding `base_container_name` to the `KubernetesPodOperator` templated fields. The rationale is that the base container name is part of the log lines emitted by the KubernetesPodManager, which is a good opportunity to have it give as much context as possible. For example, in a Wikimedia DAG, we defined the following operators: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki # Name of the "dumps" container (default is base, which isn't super telling) self.base_container_name = f"mediawiki-{self.dump_type}-dump" # name of the pod itself # made templated in #46268 self.name = f"{self.base_container_name}-{wiki}" class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator): """Operator class running the sql/xml wiki dumps as Kubernetes Pods""" dump_type = "sql-xml" class WikimediaWikidataDumpsOperator(WikimediaDumpOperator): """Operator class running the wikidata dumps as Kubernetes Pods""" dump_type = "wikidata" ``` Adding `base_container_name` to the templated fields would allow us to rewrite the `WikimediaDumpOperator` to the following: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki ``` and we could invoke the operator as such: ```python WikimediaSqlXmlOperator( ..., base_container_name='mediawiki-{{ task.dump_type }}-dump', name='{{ task.base_container_name }}-{{ task.wiki }}' ... ) ``` The endgame would be to have our logs contain as much context as possible while avoiding mixing passing both keyword args to the conttructor _and_ infering some attributes _within_ the `__init__` method itself.
shubham-pyc
pushed a commit
to shubham-pyc/airflow
that referenced
this pull request
Apr 2, 2025
apache#47864) I would like to propose adding `base_container_name` to the `KubernetesPodOperator` templated fields. The rationale is that the base container name is part of the log lines emitted by the KubernetesPodManager, which is a good opportunity to have it give as much context as possible. For example, in a Wikimedia DAG, we defined the following operators: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki # Name of the "dumps" container (default is base, which isn't super telling) self.base_container_name = f"mediawiki-{self.dump_type}-dump" # name of the pod itself # made templated in apache#46268 self.name = f"{self.base_container_name}-{wiki}" class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator): """Operator class running the sql/xml wiki dumps as Kubernetes Pods""" dump_type = "sql-xml" class WikimediaWikidataDumpsOperator(WikimediaDumpOperator): """Operator class running the wikidata dumps as Kubernetes Pods""" dump_type = "wikidata" ``` Adding `base_container_name` to the templated fields would allow us to rewrite the `WikimediaDumpOperator` to the following: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki ``` and we could invoke the operator as such: ```python WikimediaSqlXmlOperator( ..., base_container_name='mediawiki-{{ task.dump_type }}-dump', name='{{ task.base_container_name }}-{{ task.wiki }}' ... ) ``` The endgame would be to have our logs contain as much context as possible while avoiding mixing passing both keyword args to the conttructor _and_ infering some attributes _within_ the `__init__` method itself.
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Apr 4, 2025
apache#47864) I would like to propose adding `base_container_name` to the `KubernetesPodOperator` templated fields. The rationale is that the base container name is part of the log lines emitted by the KubernetesPodManager, which is a good opportunity to have it give as much context as possible. For example, in a Wikimedia DAG, we defined the following operators: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki # Name of the "dumps" container (default is base, which isn't super telling) self.base_container_name = f"mediawiki-{self.dump_type}-dump" # name of the pod itself # made templated in apache#46268 self.name = f"{self.base_container_name}-{wiki}" class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator): """Operator class running the sql/xml wiki dumps as Kubernetes Pods""" dump_type = "sql-xml" class WikimediaWikidataDumpsOperator(WikimediaDumpOperator): """Operator class running the wikidata dumps as Kubernetes Pods""" dump_type = "wikidata" ``` Adding `base_container_name` to the templated fields would allow us to rewrite the `WikimediaDumpOperator` to the following: ```python class WikimediaDumpOperator(KubernetesPodOperator): """ Base class for all types of wiki dumps run as Kubernetes Pods. """ dump_type = "generic" def __init__(self, wiki: str, *args, **kwargs): super().__init__(*args, **kwargs) self.wiki = wiki ``` and we could invoke the operator as such: ```python WikimediaSqlXmlOperator( ..., base_container_name='mediawiki-{{ task.dump_type }}-dump', name='{{ task.base_container_name }}-{{ task.wiki }}' ... ) ``` The endgame would be to have our logs contain as much context as possible while avoiding mixing passing both keyword args to the conttructor _and_ infering some attributes _within_ the `__init__` method itself.
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
May 28, 2025
…s (#47864)
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache/airflow#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
GitOrigin-RevId: 204020a329d954dca14ef30ea7f72c25782da85b
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Sep 24, 2025
…s (#47864)
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache/airflow#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
GitOrigin-RevId: 204020a329d954dca14ef30ea7f72c25782da85b
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Oct 22, 2025
…s (#47864)
I would like to propose adding `base_container_name` to the
`KubernetesPodOperator` templated fields.
The rationale is that the base container name is part of the log lines
emitted by the KubernetesPodManager, which is a good opportunity to have
it give as much context as possible.
For example, in a Wikimedia DAG, we defined the following operators:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
# Name of the "dumps" container (default is base, which isn't super telling)
self.base_container_name = f"mediawiki-{self.dump_type}-dump"
# name of the pod itself
# made templated in apache/airflow#46268
self.name = f"{self.base_container_name}-{wiki}"
class WikimediaSqlXmlDumpsOperator(WikimediaDumpOperator):
"""Operator class running the sql/xml wiki dumps as Kubernetes Pods"""
dump_type = "sql-xml"
class WikimediaWikidataDumpsOperator(WikimediaDumpOperator):
"""Operator class running the wikidata dumps as Kubernetes Pods"""
dump_type = "wikidata"
```
Adding `base_container_name` to the templated fields would allow us to
rewrite the `WikimediaDumpOperator` to the following:
```python
class WikimediaDumpOperator(KubernetesPodOperator):
"""
Base class for all types of wiki dumps run as Kubernetes Pods.
"""
dump_type = "generic"
def __init__(self, wiki: str, *args, **kwargs):
super().__init__(*args, **kwargs)
self.wiki = wiki
```
and we could invoke the operator as such:
```python
WikimediaSqlXmlOperator(
...,
base_container_name='mediawiki-{{ task.dump_type }}-dump',
name='{{ task.base_container_name }}-{{ task.wiki }}'
...
)
```
The endgame would be to have our logs contain as much context as
possible while avoiding mixing passing both keyword args to the
conttructor _and_ infering some attributes _within_ the `__init__`
method itself.
GitOrigin-RevId: 204020a329d954dca14ef30ea7f72c25782da85b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding name and hostname to KubernetesPodOperator template_fields
kubernetes_tests/test_kubernetes_pod_operator.pytoo long name testing flow and add this case to unit testscloses: #43480
Based on previously staled PR
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.