Adding property files option in the Spark Submit command #34894
Conversation
eladkal
changed the title
|
can this be tested with unit test? |
Hi @eladkal, As I mentioned above, I am a newcomer to open source and a also the development as a whole. So I haven't done any testing yet. However I would love to do so if you can guide me on how to do it. |
| self._hook = self._get_hook() | ||
| self._hook.property_files() | ||
|
|
||
| def _get_hook(self) -> SparkSubmitHook: |
There was a problem hiding this comment.
you have to accept the property_files param value as one of the parameters that can be used when creating SparkSubmitOperator. This code doesn't involve those changes.
There was a problem hiding this comment.
Okay will try to add that as well
There was a problem hiding this comment.
I've added properties-file param to Operator and Hook files. However, I haven't added the default file path handling logic as it seems to be already handled by spark-submit command as per the docs. I will follow up with the test case changes in the next commit. let me know if there is any feedback or concern.
you have to accept the property_files param value as one of the parameters that can be used when creating SparkSubmitOperator. This code doesn't involve those changes.
Please check if you can add relevant test in the test_spark_submit.py file for the changes you have done in spark_submit.py file. @ghostp13409 |
|
hi @ghostp13409 any update on this? if you are struck in this let me know... i will see if i can support you. also some static checks are failing. you may need to fix those parts too. |
HI @Bowrna, thanks for reaching out. A possible way to make the text more professional is: I am working on setting up the dev environment, but I am encountering an error while installing breeze. I have tried different platforms (Windows and Linux) and versions of python and pyenv, but the error persists. I would appreciate your guidance on how to resolve this issue, as I have already asked about it on the slack community. |
|
can you point out where you posted the question? @ghostp13409 |
I posted my question on #troubleshooting and #random. |
|
Hi, |
|
yeah sure, that would be very helpful.
…________________________________
From: Ashish Patel ***@***.***>
Sent: Tuesday, November 28, 2023 2:41 PM
To: apache/airflow ***@***.***>
Cc: ghostp13409 ***@***.***>; Mention ***@***.***>
Subject: Re: [apache/airflow] Adding property files option in the Spark Submit command (PR #34894)
Hi,
I have a related issue #35911<#35911>,
@ghostp13409<https://github.com/ghostp13409> if you are stuck, I can pull your branch into mine and merge both changes and raise a PR.
let me know if that works.
cc. @eladkal<https://github.com/eladkal> @Bowrna<https://github.com/Bowrna>
—
Reply to this email directly, view it on GitHub<#34894 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BDGJQGQJJU2PGWOY6TQI4OTYGWTF5AVCNFSM6AAAAAA55Y7DG2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRZGM4TGOJZGU>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@pateash @ghostp13409 if you need any help to push the fix for this issue, let me know. |
|
#36164 |
@pateash thanks |
4 participants
Hi @Bowrna
closes: #34838
I've added property-files operator in the spark submit operator.
As I'm a newcomer to this industry I'm not sure if I need to do any other changes other than this. please let me know if there is a need to do any additional changes.