cmd/initContainer: Try to handle config files that're absolute symlinks by martymichal · Pull Request #460 · containers/toolbox

martymichal · 2020-05-27T15:16:39Z

There has been a long-standing issue[0] when symlinking /etc/resolv.conf
to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were
proposed. Hardcode other paths, where the file could be, completely rely
on tools like flatpak-session-helper that will track such files in a
single directory, or update the symlinking function to follow symlinks
and update the target path.

The first solution is the easiest short-term but not very good
long-term. The second solution is possibly the best long-term but there
is a problem with using flatpak-session-helper. It cannot be used as
root. That leaves the third option, follow symlinks and update the
target path if the target is an invalid symlink.

This commit takes the third approach to solve the issue. Now the target
of the symlinking is first tested if it is a symlink. If it's not then
the symlinking is done right away. If it's a symlink then it is resolved.
When the target is valid, symlinking proceeds normally. If it it's not
then symlinking still proceeds but in two different ways depending on
target being an absolute or a relative symlink:

absolute - target is prepended with /run/host (the target may not be
invalid)
relative - target is not changed (the target will be invalid)

This commit tries to rely on well-made relative symlinks. Those behave
correctly even when they are placed in a different prefix (in Toolbox's
case under /run/host).

Thanks Tudor Roman for raising concern about relative links.

Closes #187

softwarefactory-project-zuul · 2020-05-27T15:20:44Z

Build failed.

shellcheck : SUCCESS in 1m 53s
test-with-podman-stable-f30 : FAILURE in 2m 16s
test-with-podman-stable-f31 : FAILURE in 2m 43s
test-with-podman-rawhide : FAILURE in 2m 20s

softwarefactory-project-zuul · 2020-07-23T20:21:37Z

Build failed.

shellcheck : SUCCESS in 1m 41s
system-test-fedora-31 : SUCCESS in 3m 36s
system-test-fedora-32 : SUCCESS in 3m 48s
system-test-fedora-rawhide : FAILURE in 4m 18s

src/cmd/initContainer.go

martymichal · 2020-07-28T08:38:08Z

I extended the comment in the function to clarify what it does, added a description of the redirectPath function and wrapped returned errors around the original errors.

softwarefactory-project-zuul · 2020-07-28T08:42:38Z

Build failed.

shellcheck : SUCCESS in 1m 57s
system-test-fedora-31 : SUCCESS in 3m 52s
system-test-fedora-32 : SUCCESS in 3m 37s
system-test-fedora-rawhide : FAILURE in 5m 01s

softwarefactory-project-zuul · 2020-08-11T23:00:16Z

Build failed.

shellcheck : SUCCESS in 1m 47s
system-test-fedora-31 : SUCCESS in 4m 16s
system-test-fedora-32 : SUCCESS in 3m 07s
system-test-fedora-rawhide : FAILURE in 3m 32s

martymichal · 2020-08-13T10:40:05Z

@debarshiray when you have time, could you look at this?? 2c4405d and 88a7ae5 made this a bit larger change but I think it's necessary to include it to keep toolboxes work across system updates (or rebases on Silverblue/CoreOS).

softwarefactory-project-zuul · 2020-08-13T10:42:42Z

Build failed.

shellcheck : TIMED_OUT in 5m 33s
system-test-fedora-31 : SUCCESS in 3m 51s
system-test-fedora-32 : SUCCESS in 3m 27s
system-test-fedora-rawhide : FAILURE in 3m 13s

debarshiray

Thanks for working on this!

debarshiray · 2020-08-13T16:27:09Z

src/cmd/initContainer.go

+	// If the target is a link, follow the chain. If needed, repeat the process.
+	for isSymlink {
+		logrus.Debugf("Path %s is a symlink. Resolving.", target)
+		newTarget, err := os.Readlink(target)


Am I wrong in thinking that a lot of this can be simplified by using filepath.EvalSymlinks ?

For example, I have:

$ ls -l /etc/os-release lrwxrwxrwx. 1 root root 21 Apr 28 14:50 /etc/os-release -> ../usr/lib/os-release

Invoking filepath.EvalSymlinks("/etc/os-release") gives me /usr/lib/os-release.

For the sake of sanity, I don't mind requiring that the redirectPath function accepts only absolute paths, because it's an internal function whose callers are under our control. If we go down this path, then I would check the inputs early on in the function and panic out (because it would be a programmer error) if the conditions aren't met.

Ignoring the exact Go APIs that are used, I think that the redirectPath function should roughly do the following:

If target is not a symbolic link, then don't bother checking if it exists or not. Just link containerPath to target.

If target is a symbolic link, then check if it's resolving. If it's resolving, then link containerPath to target.

If target is a symbolic link and if it's not resolving, then either the link is absolute or relative. If it's absolute, then prepend /run/host to the destination of target and link containerPath to it, which may or not lead to a valid file. If it's relative, then link containerPath to target even though we know that it doesn't resolve. In both these cases, we are not trying to guarantee that containerPath points to a valid file. We are merely doing the best we can to dig up the right destination, and giving up if we run out of guesses.

It should not fail the entry-point process even if target doesn't exist or resolve. We can log a debug message or such, but we shouldn't stop the container from starting up just because there's something wrong with a configuration file. In this case, a slightly broken container that starts is vastly better than one doesn't start at all. If nothing else, it makes it easier to debug.

At the end of the day, all sane operating systems should be using relative symbolic links on the host. Fedora does that, and this is exactly the reason why relative symlinks are nice. They work even if you change the root from / to /run/host.

What we are trying to fix, are those few cases where a human created an absolute symbolic link. We should do our best to handle the obvious and simple cases, but if a human has made a mistake, then it's their responsibility to fix that up.

src/cmd/initContainer.go

debarshiray · 2020-08-18T23:06:45Z

src/cmd/initContainer.go

+	// If the target is a link, follow the chain. If needed, repeat the process.
+	for isSymlink {
+		logrus.Debugf("Path %s is a symlink. Resolving.", target)
+		newTarget, err := os.Readlink(target)


Ignoring the exact Go APIs that are used, I think that the redirectPath function should roughly do the following:

If target is not a symbolic link, then don't bother checking if it exists or not. Just link containerPath to target.

If target is a symbolic link, then check if it's resolving. If it's resolving, then link containerPath to target.

If target is a symbolic link and if it's not resolving, then either the link is absolute or relative. If it's absolute, then prepend /run/host to the destination of target and link containerPath to it, which may or not lead to a valid file. If it's relative, then link containerPath to target even though we know that it doesn't resolve. In both these cases, we are not trying to guarantee that containerPath points to a valid file. We are merely doing the best we can to dig up the right destination, and giving up if we run out of guesses.

It should not fail the entry-point process even if target doesn't exist or resolve. We can log a debug message or such, but we shouldn't stop the container from starting up just because there's something wrong with a configuration file. In this case, a slightly broken container that starts is vastly better than one doesn't start at all. If nothing else, it makes it easier to debug.

At the end of the day, all sane operating systems should be using relative symbolic links on the host. Fedora does that, and this is exactly the reason why relative symlinks are nice. They work even if you change the root from / to /run/host.

What we are trying to fix, are those few cases where a human created an absolute symbolic link. We should do our best to handle the obvious and simple cases, but if a human has made a mistake, then it's their responsibility to fix that up.

debarshiray · 2020-08-18T23:21:24Z

Given that operating systems are expected to use relative symbolic links, it seems to me that updates aren't such a pressing problem.

When someone updates from Fedora 32 to 33, then their /etc/resolv.conf will turn into a relative symbolic link. Existing Toolbox containers had their /etc/resolv.conf pointing at /run/host/etc/resolv.conf, which will now be a relative symlink and therefore resolve.

Things can break if a user changes their /etc/resolv.conf into a symbolic link that either:

Points somewhere that's not bind mounted into the container. I won't worry about this unless someone actually comes up with a valid example.
Is absolute, and the change happened after there were already some Toolbox containers with their /etc/resolv.conf configured in a way that didn't expect that. In such a case, it's not that hard for the user to switch to a relative symlink. After all they are messing around with the OS' configuration, and this is a pretty minor tweak.

softwarefactory-project-zuul · 2020-08-19T16:23:00Z

Build failed.

shellcheck : SUCCESS in 2m 31s
system-test-fedora-31 : SUCCESS in 4m 45s
system-test-fedora-32 : SUCCESS in 8m 10s
system-test-fedora-rawhide : FAILURE in 3m 06s

softwarefactory-project-zuul · 2020-08-19T16:30:41Z

Build failed.

shellcheck : SUCCESS in 1m 57s
system-test-fedora-31 : SUCCESS in 4m 12s
system-test-fedora-32 : SUCCESS in 3m 14s
system-test-fedora-rawhide : FAILURE in 3m 23s

containers#460

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-19T16:41:30Z

Build failed.

shellcheck : SUCCESS in 2m 35s
system-test-fedora-31 : SUCCESS in 3m 57s
system-test-fedora-32 : SUCCESS in 3m 13s
system-test-fedora-rawhide : FAILURE in 3m 05s

src/cmd/initContainer.go

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-27T17:52:03Z

Build failed.

shellcheck : SUCCESS in 2m 02s
system-test-fedora-31 : SUCCESS in 3m 05s
system-test-fedora-32 : SUCCESS in 4m 08s
system-test-fedora-rawhide : FAILURE in 3m 04s

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-28T10:39:14Z

Build failed.

shellcheck : SUCCESS in 2m 16s
system-test-fedora-31 : SUCCESS in 3m 40s
system-test-fedora-32 : SUCCESS in 3m 46s
system-test-fedora-rawhide : FAILURE in 3m 12s

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-28T11:07:10Z

Build failed.

shellcheck : SUCCESS in 1m 49s
system-test-fedora-31 : SUCCESS in 3m 35s
system-test-fedora-32 : SUCCESS in 3m 42s
system-test-fedora-rawhide : FAILURE in 3m 18s

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-28T11:15:23Z

Build failed.

shellcheck : SUCCESS in 1m 52s
system-test-fedora-31 : SUCCESS in 3m 50s
system-test-fedora-32 : SUCCESS in 3m 24s
system-test-fedora-rawhide : FAILURE in 2m 55s

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

softwarefactory-project-zuul · 2020-08-28T11:20:41Z

Build failed.

shellcheck : SUCCESS in 1m 58s
system-test-fedora-31 : SUCCESS in 3m 23s
system-test-fedora-32 : SUCCESS in 2m 57s
system-test-fedora-rawhide : FAILURE in 2m 48s

There has been a long-standing issue[0] when symlinking /etc/resolv.conf to /run/host/etc/resolv.conf (host's resolv.conf). Many solutions were proposed. Hardcode other paths, where the file could be, completely rely on tools like flatpak-session-helper that will track such files in a single directory, or update the symlinking function to follow symlinks and update the target path. [0] containers#187 The first solution is the easiest short-term but not very good long-term. The second solution is possibly the best long-term but there is a problem with using flatpak-session-helper. It cannot be used as root. That leaves the third option, follow symlinks and update the target path if the target is an invalid symlink. This commit takes the third approach to solve the issue. Now the target of the symlinking is first tested if it is a symlink. If it's not then the symlinking is done right away. If it's a symlink then it is resolved. When the target is valid, symlinking proceeds normally. If it it's not then symlinking still proceeds but in two different ways depending on target being an absolute or a relative symlink: - absolute - target is prepended with /run/host (the target may not be invalid) - relative - target is not changed (the target will be invalid) This commit tries to rely on well-made relative symlinks. Those behave correctly even when they are placed in a different prefix (in Toolbox's case under /run/host). Thanks Tudor Roman for raising concern about relative links. Based on: containers#380 containers#460

Currently toolbox containers can get misconfigured if some configuration files on the host are absolute symbolic links to some other location. For example, when systemd-resolved is used to manage /etc/resolv.conf on the host, and if the file is an absolute link to /run/systemd/resolve/stub-resolv.conf, then /etc/resolv.conf ends up being invalid inside the container. This happens because the container's /etc/resolv.conf points to /run/host/etc/resolv.conf, which in turn points to /run/systemd/resolve/stub-resolv.conf, but that's absent from the container. This is, of course, not a problem with relative symbolic links. If the host had a relative link to ../run/systemd/resolve/stub-resolv.conf, then it would continue to work inside the container. One solution would have been to use flatpak-session-helper to maintain a copy of the host's /etc/resolv.conf in $XDG_RUNTIME_DIR/.flatpak-helper/monitor. However, that doesn't work when toolbox(1) is run as root. The other option is to prepend the destination of the absolute symbolic link with /run/host to make it resolve inside the container. It might not work with funky links, but it's enough for the common case where a an administrator changed the host's /etc/resolv.conf into a sane, but absolute, symbolic link. Properly configured hosts should anyway use relative symbolic links, because they don't need to be adjusted in such scenarios. That's also what Fedora and Ubuntu do, by default. Thanks to Tudor Roman for raising a concern about relative symbolic links. Based on Martin Pitt's work on the POSIX shell implementation: containers#380 containers#187

softwarefactory-project-zuul · 2020-08-28T11:55:14Z

Build failed.

shellcheck : SUCCESS in 2m 04s
system-test-fedora-31 : SUCCESS in 4m 31s
system-test-fedora-32 : SUCCESS in 3m 24s
system-test-fedora-rawhide : FAILURE in 3m 04s

softwarefactory-project-zuul · 2020-08-28T12:29:18Z

Build failed.

shellcheck : SUCCESS in 1m 58s
system-test-fedora-31 : SUCCESS in 3m 41s
system-test-fedora-32 : SUCCESS in 2m 56s
system-test-fedora-rawhide : FAILURE in 3m 37s

debarshiray · 2020-08-28T12:32:59Z

I took the liberty to address the above issues.

Thanks for your work on this, @HarryMichal !

martymichal added this to the Stable 1.0 milestone Jul 7, 2020

martymichal added 3. Enhancement Improvement to an existing feature 6. Major Change May cause breakage labels Jul 8, 2020

martymichal force-pushed the follow-symlinks-when-linking branch from a0882a1 to 0289674 Compare July 23, 2020 20:13

martymichal added the 3. Bugfix Fixes a bug label Jul 23, 2020

tudurom reviewed Jul 24, 2020

View reviewed changes

src/cmd/initContainer.go Outdated Show resolved Hide resolved

martymichal force-pushed the follow-symlinks-when-linking branch from 0289674 to 24c936a Compare July 28, 2020 08:36

martymichal requested a review from debarshiray July 28, 2020 08:38

martymichal mentioned this pull request Aug 4, 2020

Curl hostname resolve error in toolbox #521

Closed

martymichal force-pushed the follow-symlinks-when-linking branch from 24c936a to ef34d58 Compare August 11, 2020 22:55

martymichal force-pushed the follow-symlinks-when-linking branch 2 times, most recently from b8660cc to 88a7ae5 Compare August 13, 2020 10:36

debarshiray reviewed Aug 13, 2020

View reviewed changes

src/cmd/initContainer.go Show resolved Hide resolved

debarshiray requested changes Aug 18, 2020

View reviewed changes

martymichal force-pushed the follow-symlinks-when-linking branch from 88a7ae5 to 4e0ba3c Compare August 19, 2020 16:13

martymichal force-pushed the follow-symlinks-when-linking branch from 4e0ba3c to 835fc3c Compare August 19, 2020 16:25

martymichal added a commit to martymichal/toolbox that referenced this pull request Aug 19, 2020

cmd/initContainer: Make redirectPath errors non-fatal

dd2b3ec

containers#460

martymichal force-pushed the follow-symlinks-when-linking branch from 835fc3c to dd2b3ec Compare August 19, 2020 16:33

debarshiray reviewed Aug 21, 2020

View reviewed changes

src/cmd/initContainer.go Outdated Show resolved Hide resolved

src/cmd/initContainer.go Outdated Show resolved Hide resolved

src/cmd/initContainer.go Show resolved Hide resolved

debarshiray reviewed Aug 27, 2020

View reviewed changes

src/cmd/initContainer.go Outdated Show resolved Hide resolved

debarshiray reviewed Aug 27, 2020

View reviewed changes

src/cmd/initContainer.go Outdated Show resolved Hide resolved

debarshiray force-pushed the follow-symlinks-when-linking branch from ae0cfed to d755b2a Compare August 27, 2020 17:47

debarshiray force-pushed the follow-symlinks-when-linking branch from d755b2a to 56e86e3 Compare August 28, 2020 10:34

debarshiray force-pushed the follow-symlinks-when-linking branch from 56e86e3 to 2e823c5 Compare August 28, 2020 11:02

debarshiray force-pushed the follow-symlinks-when-linking branch from 2e823c5 to e97fd51 Compare August 28, 2020 11:10

debarshiray force-pushed the follow-symlinks-when-linking branch from e97fd51 to 4c5906e Compare August 28, 2020 11:16

debarshiray force-pushed the follow-symlinks-when-linking branch from 4c5906e to 2b78d78 Compare August 28, 2020 11:49

debarshiray force-pushed the follow-symlinks-when-linking branch from 2b78d78 to 88a95b0 Compare August 28, 2020 12:24

debarshiray merged commit 88a95b0 into containers:master Aug 28, 2020

debarshiray mentioned this pull request Aug 28, 2020

Support /etc/resolv.conf symlinks on the host #380

Closed

debarshiray changed the title ~~cmd/initContainer: Reinforce symlink redirection~~ cmd/initContainer: Try to handle config files that're absolute symlinks Aug 28, 2020

debarshiray mentioned this pull request Nov 16, 2021

Erratic behaviour with toolbox /etc/resolv.conf when changing its symlink target in host #718

Closed

Conversation

martymichal commented May 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented May 27, 2020

Uh oh!

softwarefactory-project-zuul bot commented Jul 23, 2020

Uh oh!

Uh oh!

martymichal commented Jul 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Jul 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 11, 2020

Uh oh!

martymichal commented Aug 13, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 13, 2020

Uh oh!

debarshiray left a comment

Choose a reason for hiding this comment

Uh oh!

debarshiray Aug 13, 2020

Choose a reason for hiding this comment

Uh oh!

debarshiray Aug 18, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

debarshiray Aug 18, 2020

Choose a reason for hiding this comment

Uh oh!

debarshiray commented Aug 18, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 19, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 19, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 19, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Aug 27, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

softwarefactory-project-zuul bot commented Aug 28, 2020

Uh oh!

debarshiray commented Aug 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martymichal commented May 27, 2020 •

edited

Loading