Signal handling in main process changed after forking a new worker process

[prajnoha]

Normally, SID daemon reacts to SIGTERM and it shuts itself down gracefully upon reception of this signal. However, if SID daemon forks at least one worker process (and that one can finish completely), the signal handler is not working correctly further - there's no reaction, the signal handler is not called at all.

This seems to be related to the code in worker_control_get_new_worker where a new worker process is forked from daemon. After forking, in the worker process, we separate several resources we also need in the worker and we attach all of these inherited resources to top-level "worker" resource we've just created. Then we destroy the rest of unneeded resources in the worker process, including the "sid" resource which had SIGTERM and SIGINT signals registered (and which was previously the top-level resource).

Above-mentioned sequence is executed in worker-control code right here:

https://github.com/sid-project/sid-mvp/blob/0cd4698aa6d94d4b59cdbb892c689632d8572759/src/resource/worker-control.c#L137-L142

All works as expected (SIGTERM signal handling is OK) after commenting out this line: https://github.com/sid-project/sid-mvp/blob/0cd4698aa6d94d4b59cdbb892c689632d8572759/src/resource/worker-control.c#L142

It looks that registering and then deregistering signal handlers (even though for various event loops) get messed up somehow...

Obviously, this is an issue when we're trying to stop SID daemon (...if that's systemctl stop sid.service, then the systemctl call halts because it is unable to stop the service - that will never stop as the signal is not processed).

[prajnoha]

Also, it's interesting that we register SIGINT exactly the same way as SIGTERM (both in sid resource in parent daemon process and worker resource in the worker process). But it's only SIGTERM handling that has issues.

[prajnoha]

One more important note: the SIGTERM handling is not correct in the main daemon process - that's where the problem is. Thing is, the problem appears even after the worker process is already gone/exited completely and we're left only with the main daemon process and now workers running!

Instantiating the worker process only initiates the problem, but further worker existence is not needed for this problem to occur.

[trgill]

I think a systemd process should be blocking signals and using sd_event_add_signal() to get notifications (?):

https://man7.org/linux/man-pages/man3/sd_event_signal_handler_t.3.html

We should do something like:

sid/src/resource/sid.c

Line 31 in 855fb24

if (sigemptyset(&sig_set) < 0) {

But update this:

sid/src/daemon/sid.c

Line 75 in 855fb24

signal(SIGTERM, &_shutdown_signal_handler);

to use sd_event_add_signal().

I'll make the updates and test it out.

[prajnoha]

I think a systemd process should be blocking signals and using sd_event_add_signal() to get notifications (?):

https://man7.org/linux/man-pages/man3/sd_event_signal_handler_t.3.html

Yes, exactly, systemd requires the signals to be blocked first before registering them with systemd.

But update this:

sid/src/daemon/sid.c

Line 75 in 855fb24

signal(SIGTERM, &_shutdown_signal_handler);

to use sd_event_add_signal().

Ah, sure, this should be cleaned up! However, my intention was to keep all systemd-related code in one place (one of SID's own libraries where it can wrap it). Currently, the calls to systemd lib fns is kept in resource.c (libsidresource) and log-target-journal.c (libsidlog) - just for us to be able to switch into using a different library in the future if there's an alternative for other environments (e.g. #32), making switching easier.

We can possible try to move the _become_daemon functionality inside resource code and provide that as an option somehow when creating resources.

Then, if we called:

sid/src/daemon/sid.c

Line 196 in 855fb24

if (!(sid_res = sid_resource_ref(sid_resource_create(SID_RESOURCE_NO_PARENT,

...we'd passed that within one of sid_resource_create args (maybe with as the sid_resource_flags_t flags one). Then making sure we become daemon only once, so any further resource creations with the "become daemon" request would fail if we're already a daemon. Just an idea, but we can clean this up in a different way... simply, the intention is to keep the systemd-related code in one place and wrapped so we can switch into an alternative if needed.

[trgill]

The forking-and-exiting only makes sense if we start the daemon manually from a terminal and not with systemd, right? I see the sid.service file starts it in the foreground.

Currently the parent hits the select() and gets the SIGTERM from the child and exits without handling the signal. This leaves a defunct process until the child exits.

For the child, we'd like it to field the SIGTERM and exit the event loop, right? I think we need to make some updates for that to work.

This is probably a low priority issue, right? I picked it somewhat randomly. I'll look through the other issues to see where I might be able to help. Please let me know if you have any suggestions.

[prajnoha]

The forking-and-exiting only makes sense if we start the daemon manually from a terminal and not with systemd, right? I see the sid.service file starts it in the foreground.

Yes, indeed, if systemd is used, it's not necessary to daemonize the process, systemd will take care of that for us...

Currently the parent hits the select() and gets the SIGTERM from the child and exits without handling the signal. This leaves a defunct process until the child exits.

For the child, we'd like it to field the SIGTERM and exit the event loop, right? I think we need to make some updates for that to work.

For now, yes, it was supposed to exit the event loop immediately. I'd like to enhance this eventually so that if SID receives SIGTERM, it will stop listening to any new requests and let all existing workers to finish before SID exits completely.

There's one important thing to note when considering signal handlers through systemd lib. If we don't specify the handler for sid_resource_create_signal_event_source (and so the underlying sd_event_add_signal call), the default operation executed by systemd lib is to exit its event loop. If the custom handler is defined, systemd lib lets us handle everything, including possible exit of the event loop. In this case, we need to call sid_resource_exit_event_loop (and so the underlying sd_event_exit) to do so.

This is probably a low priority issue, right? I picked it somewhat randomly. I'll look through the other issues to see where I might be able to help. Please let me know if you have any suggestions.

Relatively low, but it was very annoying so I'm happy it's being resolved now :) Sorry, I promised to open issues here on github for all the things I've hit and have written down in my notes. Somehow I didn't get to that, but will try to do that asap.

[trgill]

Is this ok to close now?

It is fixed by : #50

[prajnoha]

Sure! Closing...

[prajnoha]

Hmm, interesting, after resolving #52, this still seems to be an issue even after we got rid of the direct signal registration in commit a07f331. When I try to send SIGTERM or SIGINT to main SID process after at least one worker is processed, the signal is ignored.

So It looks like this call in the worker still influences the parent process somehow:

sid/src/resource/worker-control.c

Lines 709 to 710 in 6a4dbe2

	/* destroy the unneeded and inherited "sid" resource from parent */
	(void) sid_resource_unref(sid_resource_search(worker_control_res, SID_RESOURCE_SEARCH_TOP, NULL, NULL));

[trgill]

Yeah, after the following in the log:

150701:150792]:resource.c:111:_destroy_event_source    <worker/0bc7df7b-6259-43fa-89e5-cb84043c6185> Event source removed: sigterm.
[150701:150792]:resource.c:111:_destroy_event_source    <worker/0bc7df7b-6259-43fa-89e5-cb84043c6185> Event source removed: sigint.
[150701:150792]:resource.c:111:_destroy_event_source    <worker/0bc7df7b-6259-43fa-89e5-cb84043c6185> Event source removed: main.

The parent process no longer responds to the signals.

I can't find documentation that confirms that is how systemd is designed to work. I'll track down a mailing list or IRC channel to confirm what we are seeing is working as intended.

[prajnoha]

I see that systemd doesn't want users to use the event loop after forking, this is a check they call on each sd-event API call:

https://github.com/systemd/systemd/blob/23afa884d4f3bcd97160a893816f9ba170f62ad4/src/libsystemd/sd-event/sd-event.c#L430-L437

We should be still OK though as we're not using the event loop from the parent in any children anymore (we create a new one in the worker).

That's also the reason I'm using unref in sid_resource_destroy:

sid/src/resource/resource.c

Lines 309 to 310 in 99b185d

	if (res->sd_event_loop)
	res->sd_event_loop = sd_event_unref(res->sd_event_loop);

The unref doesn't do the check. So this one could still be used for doing the cleanup in the child at least. (If we called sd_event_exit instead, that one would still try to finish current event loop iteration I think which would be wrong because this is not the original even loop anymore, it's a memory clone in the child...)

[trgill]

FYI - I added 👍

	if ((r = sd_event_source_set_destroy_callback(sd_es, _on_sid_resource_destroyed)) < 0)
			goto fail;

It does not get called in the parent.

I tried to reproduce this issue with a smaller program:

https://github.com/trgill/signal_test

The does not reproduce the problem we are seeing. I'm in the process of making it closer to what sid does.