Failed to get output when starting a process

[robvanmieghem]

When creating a zerodb instance in a container as described in https://github.com/threefoldtech/0-stor/blob/development/docs/getting_started.md
I gives

JSX> zdbjob.running
False

JSX> zdbjob.stream()
0

JSX> zos.ping()
'PONG Version: development @Revision: 73b9b038f2c549d478adf7b97a1f3a2662c36f7e'
...

[muhamadazmy]

@robvanmieghem I totally couldn't reproduce this issue on a development build (same revision) using the flist from the documentation. So i will need your help to figure out why this happens on your node.

• Let's start by checking that the zdb binary provided by your flist is the same as the one i have. Mine has md5sum of e14da5aedea47c7bd7184fe1b6868f08. You can easily get the md5sum by logging to the node and then do md5sum /mnt/container/<container-id>/bin/zdb
• Then we can try to execute the binary directly from the flist like chroot /mnt/containers/<container-id>/ /bin/zdb see what u will get.

[zaibon]

I tried to reproduce, and indeed I can't stream any output.
But the 0-db is actually properly started and everything. So this is really just a streaming issue.

[serboctor]

@muhamadazmy I can reproduce it on a master too.

[muhamadazmy]

@serboctor Yes, it's indeed reproduceable on master branch, but not on development! I will look into it now

[muhamadazmy]

@serboctor so weird, now it's also happening on development, i tested it many times on development when this issue was first posted and was not reproducable!

[muhamadazmy]

Well, it seems the issue i had was related to unstable internet so the binaries didn't even start. once the node was able to download and run the binaries stream worked fine, on both master and development.

I will now test this on the affected node itself.

[muhamadazmy]

@robvanmieghem, @serboctor
So there was 2 issues that causes this behavior that affects both master and development branches:

• A client issue where it fails to detect the correct job state (if it's running or not) which causes the stream reading to terminate if the job is not outputting anything in a long time. The fix is here 9121e2a . Note: the fix will have to applied to the client embedded in jumpscale as well.
• The second issue is that zdb actually buffers it's output if it detects a pipe. which means zdb doesn't output the logs directly to stdout/err until it absolutely have to. Which means no output is written to stream on zdb start, and it can take a while before zdb decides to flush the output.

Both issues together causes this behavior. Note that fixing the client, will cause the stream() method to NOT exit unless zdb is actually stopped, but it will still not output any logs until zdb either flush it's output, or exit. (doing operations against zdb will make the output buffer to fill and eventually flush)

[serboctor]

@muhamadazmy added to development_jumpscale threefoldtecharchive/jumpscaleX_core@b4ee26e