Prevent newline when on continued lines - conversion issue (RUN)

[kmanalo]

Expected Behavior

Dockerfile (based on an existing one from biocontainers):

# Base image
FROM ubuntu:16.04

# add apt mirror
RUN mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \
    bash -c 'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\n \
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse\n\n" > /etc/apt/sources.list' && \
    cat /etc/apt/sources.list.bkp >> /etc/apt/sources.list && \
    cat /etc/apt/sources.list

Singularity recipe I want to generate from the above is

Bootstrap: docker
From: ubuntu:16.04
%post
# Base image

# add apt mirror
mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \
bash -c 'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\n \
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse\n\n" > /etc/apt/sources.list' && \
cat /etc/apt/sources.list.bkp >> /etc/apt/sources.list && \
cat /etc/apt/sources.list
%runscript
exec /bin/bash "$@"

Actual Behavior

Dockerfile generated is

Bootstrap: docker
From: ubuntu:16.04
%post
# Base image

# add apt mirror
mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \
bash -c 'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\n \
deb
mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\n\
deb
mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\n\
deb
mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse\n\n" > /etc/apt/sources.list' && \
cat /etc/apt/sources.list.bkp >> /etc/apt/sources.list && \
cat /etc/apt/sources.list
%runscript
exec /bin/bash "$@"

Steps to Reproduce

spython recipe Dockerfile # above file

Context

• spython version: 0.0.48
• python version: 3.6.4

Possible Fix

I tried a few ways, but this is the simplest one based on the existing split lines method.

In docker.py :

def _setup(self, action, line):
    ''' replace the command name from the group, alert the user of content,
        and clean up empty spaces
    '''
    bot.debug('[in]  %s' % line)

    # Replace ACTION at beginning
    line = re.sub('^%s' %action, '', line)

    # Handle continuation lines not having ACTION by padding with leading space
    line = " " + line

    # Split into components
    return [x for x in self._split_line(line) if x not in ['', None]]

Also, _split_lines could be completely taken out of docker.py (since it's already in the parent class) unless there's a design reason for that. I would be happy to contribute a PR back or some guidance if you have a better idea for a solution.

BTW, I like the bug report template, thanks!

Kevin

[vsoch]

So it's adding a newline after deb? That's strange, why do you think that's happening?

[vsoch]

okay definitely just reproduced it!

[kmanalo]

Vanessa When I was debugging it I saw lists of the form [‘’, ‘<commands after RUN>’] The first element of the list eventually gets discarded because you handled multiple new lines by consolidation somewhere

…

-Kevin

On Mon, Dec 17, 2018 at 2:21 PM Vanessa Sochat ***@***.***> wrote: okay definitely just reproduced it! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#77 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB6LGOS-KHioMF3jtWYkErpPWJ2T9xPMks5u5-6-gaJpZM4ZW6bL> .

[vsoch]

Yes the correct place to fix it is when we determine if the line is continued and return a parser (or not), not in the function to split lines. You can call me @vsoch, thanks!

[vsoch]

okay I'm going to walk through the bug here so we have it on record! The lines are parsed as expected:

['# Base image\n',
 'FROM ubuntu:16.04\n',
 '\n',
 '# add apt mirror\n',
 'RUN mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \\\n',
 '    bash -c \'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\\n \\\n',
 'deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\\n\\\n',
 'deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\\n\\\n',
 'deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse\\n\\n" > /etc/apt/sources.list\' && \\\n',
 '    cat /etc/apt/sources.list.bkp >> /etc/apt/sources.list && \\\n',
 '    cat /etc/apt/sources.list\n']

We get to line index 4, where the RUN statement starts:

'RUN mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \\\n'

Verify this is going to the _run parser (correct)

Parser is <bound method DockerRecipe._run of [spython-recipe][/tmp/Dockerfile]>

The previous line was a comment, and so far the recipe just has comments and the from header:

previous
 '# add apt mirror\n'

# This is what goes into %post eventually
self.install
['# Base image\n', '\n', '# add apt mirror\n']

When we parse the line with the parser, we should expect to see it added to the self.install

['# Base image\n',
 '\n',
 '# add apt mirror\n',
 'mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \\']

And we do! The next line should be added as a continuation of run, and this also looks okay:

['# Base image\n',
 '\n',
 '# add apt mirror\n',
 'mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \\',
 'bash -c \'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\\n \\']

Now next should be where we hit a bug, the next line is:

'deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\\n\\\n'

And the previous is

'    bash -c \'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\\n \\\n'

I'm going to walk through the function to get the parser, because the bug is in here somewhere...
here is a potential cause when we split the line looking for a command:

['deb',
 'mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\\n\\']

but this isn't technically the bug, because it's expected that lines can start with non commands, and the word deb isn't in the lookup so we return the previous parser (_run). Then _run calls _setup with RUN to remove:

        line = self._setup('RUN', line)
        self.install += line

And again the self._setup is returning the split deb and the rest of the line... and it is because there isn't an action! In which case, I think your proposed fix is reasonable, it fixes to:

['deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\\n\\']

I also get this result:

Bootstrap: docker
From: ubuntu:16.04
%post
# Base image

# add apt mirror
mv /etc/apt/sources.list /etc/apt/sources.list.bkp && \
bash -c 'echo -e "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse\n \
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-backports main restricted universe multiverse\n\
deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse\n\n" > /etc/apt/sources.list' && \
cat /etc/apt/sources.list.bkp >> /etc/apt/sources.list && \
cat /etc/apt/sources.list
%runscript
exec /bin/bash "$@"

Is this the desired result for your use case? If so, I can open a PR for you to test further.

[vsoch]

here is the proposed fix --> #78 would you be able to test this on some other recipes? Preferably with newlines. Thank you!

[vsoch]

oh and @kmanalo thank you for your comment about the issue templates, I like them very much too :O) GItHub is just extremely badass and fun! :)

[kmanalo]

@vsoch I tried #78 with a few Dockerfiles with multi-line RUN and it appears to handle it fine.

[vsoch]

okay, great! I'll merge and release in just a minute then. Thanks for the catch, fix, and test!

[vsoch]

Fix released with https://pypi.org/project/spython/0.0.49/, closing issue! Thanks again!