Adding spec version 2.0 #41

vsoch · 2021-02-28T18:42:42Z

This PR will add support for a spec 2.0, which we are version bumping because we have added support for exec and start groups, and moved the previous command (on the level of the instance) to be within start. I've also written up a "ping" example: https://github.com/singularityhub/singularity-compose-examples/pull/4/files#diff-70a0c5dcf3d1984dde44c1e48e3ceee2338fe47624fd784565ece7375adcd21f that can be used to test. There are more extensive changes because I also updated black and the docs.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

This commit also requires singularity python 0.1.0 or greater, which adds support for better instance listing by way of parsing --json. The reason to bump up is that this json flag is only supported for 3.5.2 and up. Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

the reason is that we need to support older version of singularity with instance list --json not having all the fields Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

CHANGELOG.md

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

docs/spec/README.md

briveramelo · 2021-03-11T04:44:50Z

docs/spec/spec-2.0.md

+# Singularity Compose Version 2.0
+
+The second version of the singularity compose spec has added options for
+starting an instance, and exec'ing commands after that. Note that spec v2.0 is


starting, exec'ing, and running

docs/spec/spec-2.0.md

scompose/project/project.py

briveramelo · 2021-03-11T07:29:32Z

scompose/project/project.py

        if circular_dep:
            bot.exit("Unable to create all instances, possible circular dependency.")

    # Build


The 'for loop' above prevents some containers from running. Any container that runs indefinitely will block execution of this loop and prevent all subsequent containers from running. Many web service containers operate this way.

I tried to pass command args like "indefinite.sh arg1 >& stdout.txt &" but I get errors related to invalid command args. Unless there's another existing method for backgrounding an indefinite execution script I'm not aware of, this loop ought to provide a new method to allow for this.

This probably needs to be addressed separately, it's not in scope for the PR here. It totally depends on the command that you provide - for most of the use cases I've seen they just are creating the instance without something that would continue running.

After this PR is merged, we won't release, but you can open another issue with an example to reproduce and we can work on separately. The scope of this PR is already too big.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

briveramelo · 2021-03-12T08:50:57Z

everything i've tested from these comments appears to be working now. the only exception is for the backgrounding issue, which I agree belongs in another issue

vsoch · 2021-03-12T11:20:42Z

Great! Shall we merge here and then work on that?

briveramelo · 2021-03-14T06:42:58Z

@vsoch yes, that would be great!

kt-pham · 2021-03-30T00:36:38Z

Hi, I'm trying to start the following postgres service below, but am running into issues with the port configuration:

cat singularity-compose.yml
version: '2.0'

instances:
postgres:
image: docker://postgres:latest
networks:
- main
ports:
- "5432:5432"
restart: on-failure
volumes:
- /tmp/postgresql-data:/var/lib/postgresql/data

The interactive environment I have is started with fakeroot (and --writable), so I am "root" while trying to run "singularity-compose up":

singularity-compose up -d
Creating postgres
WARNING: Disabling --writable-tmpfs as it can't be used in conjunction with underlay
INFO: Cleaning up image...
ERROR: container cleanup failed: no instance found with name postgres
FATAL: container creation failed: network requires root or --fakeroot, users need to specify --network=none with --net

Is there something you can suggest to get around this?

vsoch · 2021-03-30T00:41:50Z

Your recipe is hard to read because it's not formatted, but I'm wondering what the purpose of the "networks" block is? Same with restart.

But generally if there is a message that there is no instance named postgres, it's likely that there is some issue creating the instance. Try running with --debug so you can see the command, and then run the command verbatim and see if you see the error.

kt-pham · 2021-03-30T00:53:27Z

Hi, sorry about that. The networks and restart were pulled from a docker-compose file I was trying to convert over. Hopefully this is better:

> cat singularity-compose.yml
version: 2.0
instances:
  postgres:
    image: docker://postgres:latest
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "5432:5432"
    volumes:
      - /tmp/postgresql-data:/var/lib/postgresql/data
> singularity-compose up -d
Creating postgres
WARNING: Disabling --writable-tmpfs as it can't be used in conjunction with underlay
INFO:    Cleaning up image...
ERROR:   container cleanup failed: no instance found with name postgres
FATAL:   container creation failed: network requires root or --fakeroot, users need to specify --network=none with --net

The issue I am having is that I am already "root" (started the interactive session with fakeroot), but the error message above is stating that I need root or fakeroot. This only happens when I include the ports section.

vsoch · 2021-03-30T01:00:10Z

As I mentioned, you should try with --debug to see the command and then run it verbatim.

singularity-compose --debug up -d

vsoch · 2021-03-30T01:01:15Z

if you want to use fakeroot, you need to add it as an option -> https://github.com/singularityhub/singularity-compose-examples/blob/ad394dffe92ddf70d792cfdd3f63ad8383a24713/v1.0/jupyter-simple/singularity-compose.yml#L7

vsoch · 2021-03-30T01:02:29Z

I don't think Singularity knows that you are fakeroot, you have to tell it explicitly. If that doesn't work with fakeroot, you might start your interactive session without and use the flag. And if you are running this from inside another container (e.g., Docker) you likely will run into other issues. I'm still not sure singularity runs well (or at all) inside docker for most things.

kt-pham · 2021-03-30T01:07:50Z

The output of --debug after adding the fakeroot:

Singularity> cat singularity-compose.yml 
version: '2.0'
instances:
  postgres:
    image: docker://postgres:latest
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "5432:5432"
    volumes:
      - /tmp/postgresql-data:/var/lib/postgresql/data
    start:
      options:
        - fakeroot
Singularity> singularity-compose --debug up -d
Creating postgres
DEBUG singularity instance start --bind /tmp/postgresql-data:/var/lib/postgresql/data --bind /tmp/resolv.conf:/etc/resolv.conf --bind /tmp/etc.hosts:/etc/hosts --net --network-args "portmap=5432:5432/tcp" --network-args "IP=10.22.0.2" --hostname postgres --writable-tmpfs /tmp/postgres/postgres.sif postgres 
WARNING: Disabling --writable-tmpfs as it can't be used in conjunction with underlay
INFO:    Cleaning up image...
ERROR:   container cleanup failed: no instance found with name postgres
FATAL:   container creation failed: network requires root or --fakeroot, users need to specify --network=none with --net

I suppose this is an issue since I am already "root" via fakeroot (this environment is started with singularity run -f), and adding the fakeroot to the start.options won't really help. I have this encapsulated environment so users have their own space to develop containers in, which enables them as fakeroot in this interactive session.

vsoch · 2021-03-30T01:14:42Z

Okay so this is your command:

singularity instance start --bind /tmp/postgresql-data:/var/lib/postgresql/data --bind /tmp/resolv.conf:/etc/resolv.conf --bind /tmp/etc.hosts:/etc/hosts --net --network-args "portmap=5432:5432/tcp" --network-args "IP=10.22.0.2" --hostname postgres --writable-tmpfs /tmp/postgres/postgres.sif postgres

try that in the terminal directly, if you get the message about fakeroot, first try adding the flag, if that doesn't work, restart your session without fakeroot and then add it.

vsoch · 2021-03-30T01:14:52Z

I don't think you can run singularity inside of singularity.

vsoch · 2021-03-30T01:15:19Z

singularity-compose needs to be run on your host where singularity is installed, and not in a container that happens to have singularity.

kt-pham · 2021-03-30T01:18:57Z

Ok, thanks

vsoch added 4 commits February 28, 2021 11:39

updating the spec to support start and exec args and fakeroot

37c2491

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

updating docs

7c935f4

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

command needs to be moved to start -> command

36ce5e5

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

updating start.command to more properly be start.args

29212ea

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

vsoch mentioned this pull request Feb 28, 2021

sudo used when ports are set, causes failure. Cannot set instance start options to set fakeroot network #38

Closed

vsoch added 7 commits February 28, 2021 15:12

adding support for run group

7e33f77

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

typo

65117a2

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

updating docs

2be379b

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

upload singularity compare to require spython 0.1.1

b0f73f3

the reason is that we need to support older version of singularity with instance list --json not having all the fields Signed-off-by: vsoch <vsoch@users.noreply.github.com>

fix bug with project.ps() listing with an int

87f6288

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

updating run_args to default to None instead of empty string

585b680

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

briveramelo reviewed Mar 11, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

updating changelog

cb277d9

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

briveramelo reviewed Mar 11, 2021

View reviewed changes

checking for already existing containers

efef903

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

vsoch merged commit 570388e into master Mar 14, 2021

vsoch deleted the add/spec-version-2.0 branch March 14, 2021 06:53

Uh oh!

Adding spec version 2.0 #41

Adding spec version 2.0 #41

Uh oh!

Conversation

vsoch commented Feb 28, 2021

Uh oh!

Uh oh!

Uh oh!

briveramelo Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

briveramelo Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

vsoch Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

vsoch Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

briveramelo commented Mar 12, 2021

Uh oh!

vsoch commented Mar 12, 2021

Uh oh!

briveramelo commented Mar 14, 2021

Uh oh!

kt-pham commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

kt-pham commented Mar 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

kt-pham commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

vsoch commented Mar 30, 2021

Uh oh!

kt-pham commented Mar 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kt-pham commented Mar 30, 2021 •

edited

Loading