gcs: do not trigger container shutdown when signaling init process#2538
gcs: do not trigger container shutdown when signaling init process#2538anmaxvl merged 1 commit intomicrosoft:mainfrom
Conversation
| signal_ok(signals) { | ||
| count(signals) == 0 | ||
| input.isInitProcess | ||
| input.signal in [9, 15] |
There was a problem hiding this comment.
I think this is up for debate.
There was a problem hiding this comment.
of course 😄 it's just when running tests, I realized that we don't have any "default" kill signals, and my change broke stop container/pod.
There was a problem hiding this comment.
We can have a api_version based check - if policy is of an old version, we do this default allow (if it's a sigterm/sigkill - based on old behaviour), otherwise for a new policy we expect the tooling to put in the correct allowed signals list.
| return err | ||
| } | ||
|
|
||
| // Don't allow signalProcessV2 to route around container shutdown policy. |
There was a problem hiding this comment.
Maybe I am missing something, but I don't think this path gates anything. The only difference EnforceShutdownContainerPolicy passing or failing will make is an error message. If it passes it will not do a shutdown and if it fails it will not do a shutdown.
I will claim that if the signal is allowed by the user and that signal is going to cause a shutdown, then they are good with being shutdown. Maybe we could treat SIGKILL as shutdown. TBD with some experiments.
There was a problem hiding this comment.
I agree. the original code didn't quite make sense to me either, since the shutdown policy doesn't enforce much either. it just removes the container from metadata store? Are you suggesting to just not call to shutdown policy here? and simply return on L695?
|
Tested with signals_test.cpp - now a child process of the init process no longer gets a SIGTERM so this fix is good: old behaviour - child gets SIGTERM and is killed: |
a048664 to
8ffbc6f
Compare
| signal_ok(signals) { | ||
| count(signals) == 0 | ||
| input.isInitProcess | ||
| semver.compare(data.api.version, "0.11.0") < 0 |
There was a problem hiding this comment.
data.api.version would always be the version shipped with the GCS I think, the policy api version would be policy_api_version
However, that still doesn't account for fragments having different version. Unfortunately by the time we evaluate the rules we wouldn't know which fragment the container came from (and thus whether it is "old" or not). I think probably the way to do this is to update check_container to, instead of using raw_container.signals straightaway, delegate it to another rule that adds the two "default" signals into the array if the framework_version (instead of api_version) is old (fragments has no api_version, only framework_version, main policy has both).
Also since we're not bumping the version in this PR this probably needs to be a <= instead of <
@matajoh do you agree?
There was a problem hiding this comment.
After discussion with @KenGordon and @matajoh , I've made the suggested implementation:
micromaomao@5309018
Also there is another commit to fix some error reasoning bug:
micromaomao@63f144a
I also tested that the behaviour of signaling an exec process is not changed. This can be triggered with crictl exec -s -t <timeout>:
PS C:\lcow_signals> crictl exec --timeout 1 -s 3873bfc939e24 /bin/sleep infinity
time="2025-10-15T13:52:29Z" level=fatal msg="execing command in container synchronously: rpc error: code = Unknown desc = failed to exec in container: failed to kill exec \"91d6297211fcf000d755a94cda2c7c2e7b86d46ccb0f26cf062038384f902eee\": guest RPC failure: policyDecision< {"decision":"deny","input":{"argList":["/bin/sleep","infinity"],"containerID":"3873bfc939e2415892b5b74a7b1dbade0f7222e266df43df85968ddda59be56e","isInitProcess":false,"rule":"signal_container_process","signal":9},"reason":{"errors":["target isn't allowed to receive the signal"]}} >policyDecision: unknown"
(denial message base64 decoded)
We will need to update the tooling to bump the framework_version and add the default signals to generated policies (only to the container, not for exec process - currently trying to kill an exec process is already denied unless policy allows)
@anmaxvl Please feel free to review and take these 2 changes :)
There was a problem hiding this comment.
they make sense. feel free to author them in a separate PR. I removed the rego changes to implement the immediate fix.
|
With this change I think we still have two (or three?) ways to kill a (specific, individual) container - via signalProcess or killContainerV2 or shutdownContainerV2, the latter two doesn't check the (I could not find a host-side command to send kill/shutdownContainerV2 but in principle the host could send it) |
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
KenGordon
left a comment
There was a problem hiding this comment.
Good now for the non confidential case. Tingmao has a PR for the confidential case coming.
jterry75
left a comment
There was a problem hiding this comment.
We need to remove the "all" functionality from Kill it is not supported in runc and we should not support it either.
When implementing signal container process enforcement policy we
introduced a bug, where instead of signalling just the container
init process we ended up sending signals (SIGTERM or SIGKILL) to
all processes running inside a container (by invoking `runc kill --all`).
`container.Kill` no longer sends signals to all container processes.
This results in an unpleasant behavior, where the init process
could be handling (e.g. ignoring) SIGTERM, where as other processes
inside container don't.
This PR makes a change to the order in which the signal container
policy is enforced:
- always call `EnforceSignalContainerProcessPolicy` before sending
any signals. Otherwise, this looks like a bug, since we would
never call `EnforceSignalContainerProcessPolicy` with
`signalingInitProcess == true` for `SIGTERM` and `SIGKILL` and
potentially bypassing policies, which do not allow `SIGTERM` or
`SIGKILL` to be sent to the init process.
- no longer call `ShutdownContainer` and instead revert back to
calling `process.Kill`.
Signed-off-by: Maksim An <maksiman@microsoft.com>
jterry75
left a comment
There was a problem hiding this comment.
We spoke offline. This change is ok for now. We should do a follow up change where we modify the Kill opts on the Task/Cow interfaces to forward the all intent to the guest over the bridge and use that to determine setting the --all flag in runc case.
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
…2538) When implementing signal container process enforcement policy we introduced a bug, where instead of signalling just the container init process we ended up sending signals (SIGTERM or SIGKILL) to all processes running inside a container (by invoking `runc kill --all`). `container.Kill` no longer sends signals to all container processes. This results in an unpleasant behavior, where the init process could be handling (e.g. ignoring) SIGTERM, where as other processes inside container don't. This PR makes a change to the order in which the signal container policy is enforced: - always call `EnforceSignalContainerProcessPolicy` before sending any signals. Otherwise, this looks like a bug, since we would never call `EnforceSignalContainerProcessPolicy` with `signalingInitProcess == true` for `SIGTERM` and `SIGKILL` and potentially bypassing policies, which do not allow `SIGTERM` or `SIGKILL` to be sent to the init process. - no longer call `ShutdownContainer` and instead revert back to calling `process.Kill`. Signed-off-by: Maksim An <maksiman@microsoft.com> (cherry picked from commit 04735e0)
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
…icrosoft#2538) When implementing signal container process enforcement policy we introduced a bug, where instead of signalling just the container init process we ended up sending signals (SIGTERM or SIGKILL) to all processes running inside a container (by invoking `runc kill --all`). `container.Kill` no longer sends signals to all container processes. This results in an unpleasant behavior, where the init process could be handling (e.g. ignoring) SIGTERM, where as other processes inside container don't. This PR makes a change to the order in which the signal container policy is enforced: - always call `EnforceSignalContainerProcessPolicy` before sending any signals. Otherwise, this looks like a bug, since we would never call `EnforceSignalContainerProcessPolicy` with `signalingInitProcess == true` for `SIGTERM` and `SIGKILL` and potentially bypassing policies, which do not allow `SIGTERM` or `SIGKILL` to be sent to the init process. - no longer call `ShutdownContainer` and instead revert back to calling `process.Kill`. Signed-off-by: Maksim An <maksiman@microsoft.com> (cherry picked from commit 04735e0)
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
… in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
* Restore runc kill all behavior for init processes PR #2538 removed the `runc kill --all` flag, when signaling containers. However, when cleaning up after the init process exists, the `--all` flag is still needed to remove any potentially orphaned processes when using runc before v1.2. See: opencontainers/runc@f8ad20f#diff-ade6035c3e554d7627cdc368b27f475fc0dad83e02382a1dea9cae9b75871087 Additionally, switch to using error strings directly from runc code in `internal\guest\runtime\runc\utils.go`: they have been available since runc v1.1.0-rc.1. See: opencontainers/runc#3033 Also, add logic to match on container not/still running error strings and return them for `Kill`, since returning `ERROR_VMCOMPUTE_SYSTEM_ALREADY_STOPPED` (`0xc0370110`) when killing a stopped container is expected behavior and handled appropriately in `"cmd/containerd-shim-runhcs-v1".(*hcsExec).Kill()`. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * go mod tidy and vendor; fix lint issues Fix lint errors (introduced by go1.24): ``` printf: non-constant format string in call to github.com/pkg/errors.Wrapf (govet) ``` Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
… in old policies (#2540) * rego: Allow sending SIGTERM and SIGKILL to the container init process in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in #2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com> * Fix missing denial reason when a signal request to a non-init process is denied This happens if the container.signals list contains relevant signals, but the process's signals list does not allow the signal. Old: {"decision":"deny","input":{"argList":["/bin/sleep","infinity"],"containerID":"0971693a04cdd4f2eeefc569754b5cd8046ec0b7c7ed6899bb3dec0dd45ba735","isInitProcess":false,"rule":"signal_container_process","signal":9},"reason":{"errors":[]}} Now: {"decision":"deny","input":{"argList":["/bin/sleep","infinity"],"containerID":"3873bfc939e2415892b5b74a7b1dbade0f7222e266df43df85968ddda59be56e","isInitProcess":false,"rule":"signal_container_process","signal":9},"reason":{"errors":["target isn't allowed to receive the signal"]}} Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> --------- Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
When implementing signal container process enforcement policy we
introduced a bug, where instead of signalling just the container
init process we ended up sending signals (SIGTERM or SIGKILL) to
all processes running inside a container (by invoking
runc kill --all).This results in an unpleasant behavior, where the init process
could be handling (e.g. ignoring) SIGTERM, where as other processes
inside container don't.
This PR makes a change to the order in which the signal container
policy is enforced:
EnforceSignalContainerProcessPolicybefore sendingany signals. Otherwise, this looks like a bug, since we would
never call
EnforceSignalContainerProcessPolicywithsignalingInitProcess == trueforSIGTERMandSIGKILLandpotentially bypassing policies, which do not allow
SIGTERMorSIGKILLto be sent to the init process.ShutdownContainerand instead revert back tocalling
process.Kill.