How to run a instance command in background?

[al3x609]

Expected Behavior

by example, without spython
this executes the %runscript in my container with args

$singularity instance.run image.img --nv test
$singularity run instance://test [args...]

it release the terminal..waiting for more commands
$

Actual Behavior

the terminal in python is not released after running a command on an instance.

Steps to Reproduce

>>>from spython.main import Client as cl
>>>my= cl.instances('test')
>>>cl.run(my,[..args..])

Context

• Operating System: centos 7.5
• singularity version: latest
• spython version: latest git
• python version: 3.7

[vsoch]

This likely is a bug! Could you give me the exact version of singularity (master branch?) along with the recipe for the image you are testing? I can help and debug if I can reproduce this.

[al3x609]

hi,
the error also scales at the time of creating the instance from python.
spython 0.0.43

recipe test:

BootStrap: docker
From: centos:latest
%labels
 test "test-label"

does not release the python's prompt

this happens with ipython and with spython shell too

but the process is created.

[vsoch]

From your recipe, you are doing "run" to an instance that doesn't have a declared runscript (based on your recipe) and so the default would be to call an interactive shell. In the terminal you would go into that shell. In spython, you hang because the shell is another process and it doesn't go in. So - you perhaps want to better identify first what you are trying to do. if you want an interactive shell, doing this from spython doesn't make sense. If you want to truly run a command, then you need to define what it is in the runscript. if you want to quickly "send" a command then do exec. The behavior that I see above is expected given this observation. Let me know what you are trying to do and I can offer suggestions for how to help you achieve that.

[al3x609]

Look what happens is that I need to run several services inside a container, I understand according to your advice in previous conversations, that I must use "instances", now, these services are raised in a script [.sh] within the %runscript section, but I must send some parameters to that script, therefore, as I understood the singularity manual for the instance I must put my script in the %startscript section, but if I do this, I can not pass parameters to it, it does not take them, I have already asked several times how to send parameters to the %startscript section in several places without an answer.

My intention is through python to lift those containers with the spython API, this script will be launched with SLURM.
The error that I mentioned before, also happens if I define a %runscript, it is also hung.

[vsoch]

I'm sorry that the singularity maintainers have not been responsive to you - I no longer wear that hat (I am lead developer for several supporting software but Sylabs is now in charge of Singularity itself) and I don't follow development beyond releases. I will help with the issue of the start script taking arguments, I'm not sure about the hanging but it will take much more work on my part. Let me see if I can help with arguments, since that seems to be the root of many of your troubles.

BootStrap: docker
From: centos:latest
# sudo singularity build container.simg Singularity
# singularity start container.simg myinstance hello moto
%labels
    test "test-label"
%startscript
    echo "The arguments are $@"

And you can see how I built the image,

sudo singularity build container.simg Singularity

Attempt 1: Just start the instance

If you report this not working, I don't expect it to either. But let's try anyway.

singularity start container.simg myinstance hello moto

I don't see any output. But maybe it's an issue that the echo is happening in the container's namespace (not back on the host)?

Attempt 2: Write to file

So let's try doing on that echos to a file that we could see! I'm adjusting the build slightly:

singularity instance.stop -a

BootStrap: docker
From: centos:latest
# sudo singularity build container.simg Singularity
# singularity start container.simg myinstance hello moto
%labels
    test "test-label"
%startscript
    echo "The arguments are $@" >> /tmp/tacomoco.txt

And now build , run again...

sudo singularity build container.simg Singularity
singularity instance.start container.simg myinstance hello moto

Do we have the file? We do! But are there arguments (should be hello moto) NOPE.

$ cat /tmp/tacomoco.txt 
The arguments are 

Attempt 3: Hack with variables passed to container

Trying again, here is the new Singularity recipe

BootStrap: docker
From: centos:latest
# sudo singularity build container.simg Singularity
# singularity start container.simg myinstance hello moto
%labels
    test "test-label"
%startscript
    echo "The arguments are ${ARRGS}" >> /tmp/tacomoco.txt

And we feel like pirates today. SO we have Arrrgs!

singularity instance.stop -a
sudo singularity build container.simg Singularity

but to start, let's instead place the args in front of the process

$ SINGULARITYENV_ARRGS="hello moto" singularity instance.start container.simg myinstance
vanessa@vanessa-ThinkPad-T460s:/tmp$ cat /tmp/tacomoco.txt 
The arguments are 
The arguments are hello moto

There you go! It's a hack, but it's workable. You would want to define the variables in your startscript, and then export the variable in python prefixed with SINGULARITYENV_. If you have issues with os.environ["key"] don't forget there is also os.putenv Want to give that a try?

[al3x609]

it works,
Effectively attempt 1, it did not work.

is something strange but, :) this allows me to use the instances much better, because before, I would only raise an instance without any value in %startscript and then with

singularity run instance://test [args]
would send the arguments to %runscript

Thanks for your patience, I'll check with python

[al3x609]

the only bad thing is that the temporary directory is filled with files, which are not deleted when an instance ends. :| ...

.............................Sep 21 18:23 container.simg.myinstance.singularity-debug.ZjLSpb
-rw------- 1 root    root     0 Sep 21 18:23 container.simg.myinstance.stderr.Inceuu
-rw------- 1 root    root     0 Sep 21 18:23 container.simg.myinstance.stdout.C9l3WP

It would be very useful to improve the instance.stop command to eliminate those residual files

[vsoch]

Woohoo! Just a heads up I'm going to be starting making dinner soon, so likely I'll help you at some ungodly hour in the middle of the night, or sometime tomorrow.

[vsoch]

@al3x609 that's an issue to deal with singularity, not spython. I experienced the same frustration with general control of instances and these temporary files, so I'd suggest any of the following:

Suggestions for Instance Files

These are suggestions for giving the user control for temporary files, which isn't currently done afaikt.

• the user might be able to designate to not create them in the first place

singularity instance.start container.simg web --nostderr --nostdout
singularity instance.start container.simg web --nostd  # implies both
singularity instance.start container.simg web --noall # disable all

• the user might have a command line directive where to write them (and then write to /dev/null)

singularity instance.start container.simg web --stdout=/dev/null

• there might be an equivalent %exitscript that handles cleanup. This could be done automatically, or if done by the user, variables should be provided by Singularity native for an instance that points to its specific filest. Something like:

• SINGULARITY_INSTANCE_STDERR: designates path to the error log

• SINGULARITY_INSTANCE_STDOUT: designates path to the output log

• SINGULARITY_INSTANCE_PIDFILE: designates path to the pid file

• SINGULARITY_INSTANCE_PID: the pid itself

and then the %exitscript

%exitscript
echo "Stopping ${SINGULARITY_INSTANCE_PID}"
cp ${SINGULARITY_INSTANCE_STDERR} /tmp/logs/archive/
rm ${SINGULARITY_INSTANCE_STDERR}

So the above variables would need to be provided for the user.

Instance Status

Another clear need is an ability to get status, or define custom with minimal work. I would want to be able to control a script that a user can get status of my service. For example:

singularity instance.status instance://web

and then without anything, the default would be to echo the content of -

• SINGULARITY_INSTANCE_STATUS

which could do something simple like check if the startscript process is running or not. pseudocode would be:

if startscript is running:
   SINGULARITY_INSTANCE_STATUS=started
else
   SINGULARITY_INSTANCE_STATUS=stopped
export SINGULARITY_INSTANCE_STATUS

But the user could define a custom recipe to also derive this variable (and override the default). The script gets called when the user asks for a status. In my recipe:

%status
if my custom logic passes:
    SINGULARITY_INSTANCE_STATUS=custom
else if something else
    SINGULARITY_INSTANCE_STATUS=pancaketime
else
    SINGULARITY_INSTANCE_STATUS=crapitsmessedup
export SINGULARITY_INSTANCE_STATUS

It's this kind of control / specificity that is needed for services I think! Feel free to open an issue and link this issue comment (and mention me) as I'd be interested to weigh in on the conversation.

[al3x609]

the real test

#!/usr/bin/env python3.7
# -*- coding: utf-8 -*-


import os
from spython.main import Client as cl


def run_cont(app_name, end_time, vnc_display):

    os.environ['SINGUARITYENV_APP'] = app_name
    os.environ['SINGUARITYENV_ENDTIME'] = end_time
    os.environ['SINGUARITYENV_DISP'] = vnc_display

    instance = cl.instance(
        image='/data/singularity/containers/paraview_5.5.2.simg',
        name='test_instance',
        sudo=False,
        options=['--nv']
    )

if __name__ == '__main__':
    run_cont(app_name='paraview', end_time='2018-09-21T21:07:00', vnc_display='1')

[al3x609]

my recipe

BootStrap: localimage

From: /data/singularity/containers/base.1.0.simg

%labels
  name "ParaView-5"

%environment
  export PATH=/opt/ParaView/bin:${PATH}

%files
 entrypoint.sh /opt

%post

  export PARAVIEW_URL="https://www.paraview.org/paraview-downloads/download.php?submit=Download&version=v5.5&type=binary&os=Linux&downloadFile=ParaView-5.5.2-Qt5-MPI-Linux-64bit.tar.gz"
  cd /opt
  chmod +x entrypoint.sh

  # ....  install paraview  dependences .....
  yum --disablerepo=epel install -y \
      qt5-qtbase-common.noarch

  # Paraview installation
  wget --no-check-certificate ${PARAVIEW_URL} -O t.tar.gz
  tar -xvf t.tar.gz
  mv ParaView-5.5.2-Qt5-MPI-Linux-64bit ParaView

  # Clean Section
  rm t.tar.gz
  yum clean all
  rm -rf /var/cache/yum

%startscript
 exec /opt/entrypoint.sh ${SINGUARITYENV_APP} ${SINGUARITYENV_ENDTIME} ${SINGUARITYENV_DISP} > /dev/null 2>&1   &

---

As you can see, the python script works but, it does not release the terminal,

[vsoch]

hey @al3x609 any variables that you want passed into the container MUST start with SINGUARITYENV_ don't see that you did that anywhere here?

[al3x609]

ok , :(
Yes, I'm sorry, I forgot it,
I fixed it, and I create the image again, but the result is the same,

I update the comments with the changes

[vsoch]

Can you please provide a way to get / build the paraview image?