Singularity Operator with Airflow

[SiftingSands]

I'm putting this question here because the Singularity operator for Apache Airflow uses spython - didn't think I would get the attention on the airflow repo.

I followed the example DAG https://github.com/apache/airflow/tree/main/airflow/providers/singularity/example_dags and slightly modified the operator :

    t3 = SingularityOperator(
        command='ls -l', 
        image=<path to my image>, 
        options=['--pwd /home/'],
        task_id='singularity_op_tester', 
        dag=dag
    )

The container runs without an issue but the logs show the command operating on '/' instead of '/home/' corresponding to the defined option. I also observe similar behavior with other options that don't seem to translate over the the airflow runs (such as binds, --no-nv, --no-home). These options work fine outside of airflow as expected, so I don't think it's my singularity build. I'm probably missing something obvious, but I couldn't figure it out after a serious debugging effort.

Thanks!

[vsoch]

I'm not an expert with airflow, but I'd try for options having that be a list instead of a single string in a list.

[SiftingSands]

@vsoch Unfortunately, that didn't fix it. I tried ['--pwd', '/home/'] and [['--pwd'], ['/home/']] with no avail. If nothing comes to mind, my other solution was to use a python operator that just calls spython directly (not ideal). Maybe that'll illuminate some issue with my spython install, but I highly doubt that's an issue with pip installs.

[vsoch]

I would next try testing locally with just Singularity Python and see if it works (outside of airflow). If it does, then it's something about the implementation in airflow that is not passing the option forward.

[SiftingSands]

@vsoch I think I found the culprit, or at least a clue.

In the airflow plugin, the client is started outside the creation of the instance: https://github.com/apache/airflow/blob/b0b25910713dd39e0193bdcd95b2cfd9e3fed5e7/airflow/providers/singularity/operators/singularity.py#L150

This appears to not transfer over the options. I tested this outside of airflow by:

from spython.main import Client

options = ['--no-nv']
myinstance = Client.instance(<path to sif>, 
                             options=options, start=True)
contents = Client.execute(myinstance, ["nvidia-smi"])
print(contents)

Produces {'message': 'FATAL: "nvidia-smi": executable file not found in $PATH\n', 'return_code': 255} as expected of --no-nv. However, if I run the following, then the nvidia-smi printout doesn't error out.

from spython.main import Client

options = ['--no-nv']
myinstance = Client.instance(<path to sif>, 
                             options=options, start=False)
myinstance.start()
contents = Client.execute(myinstance, ["nvidia-smi"])
print(contents)

[vsoch]

@SiftingSands I think you are right! The issue is that options and kwargs that are provided to the instance init are only honored given that start is run - since the airflow code doesn't run start at init, we run it later, and those options/args are saved but not used. I think they should be honored (and the bug is here with spython) so I opened a PR to try and fix the problem:

#179

Would you be able to test with airflow to see if the options are carried through?

[SiftingSands]

Confirming that #179 fixes the aforementioned issues with the Singularity plugin for Apache Airflow. Logs from the tasks in airflow indicate that options are now successfully passed over to instances at run-time. I haven't exhaustively tested every option, but I don't think that's necessary given your diagnosis.

Thanks for the rapid response!

[vsoch]

Great! Let's get that PR merged and released.