Possible bug - coredump hangs

[timbuchwaldt]

We had a production outage some time ago but reporting this here got lost during other tasks.
I pinpointed the issue to a big number of running coredump-composer processes that never finished processing the coredump. This in turn lead to the crashed process being kept alive (they can't be killed until the coredump handling finished) and in turn containerd becoming very unhappy about pods not terminating.

During the outage I unfortunately didn't debug further where the composer was stuck, but I could see the processes clearly running for a long time not doing active work.

My suggestion would be extending the coredump composer to have an upper bound of processing time after which it terminates itself to prevent those stuck processes staying around and potentially guarding the different parts of the file creation process with timeouts (I'm imagining crictl being stuck, but we have already written out the dump itself and I'd rather have it without lots of extra info compared to not having it at all).

[No9]

Thanks for the report @timbuchwaldt
Is this something you plan to PR?

[No9]

Thinking about this a little further: putting a timer around this is fairly involved as it will need to setup a separate thread to execute the tasks on and if they don't complete then time out.

[timbuchwaldt]

Hey @No9 - I'd look into working on that change, yep.

I see the complexity of this change, but from what I gathered during our outage I would say this is warranted. A system meant to help debugging problems should not cause/amplify them in any way.

[No9]

Totally agree it's required - Just wanted to let you know what you're taking on.
I'll commit to providing a review and getting it published ASAP.

[timbuchwaldt]

I have looked a bit into this, not super happy with what I found so far. Rust discourages timeouting on threads and I don't see another nice super generic way. GNU Coretuils might come in handy though: The timeout command can wrap any binary and implement timeouting on it. I think technically it should work to switch the core_pattern to |timeout 5 $cdc-stuff in order t ogive it a 5s timeout, what do you think?

[No9]

Not 100% on using timeout as it may not be available on the all the xKS platforms which means we will end up distributing it similar to the crio exe.
Do you have any links to the conversations on timing out threads?

[timbuchwaldt]

Ah mhm good point, yeah.

https://stackoverflow.com/questions/36181719/what-is-the-correct-way-in-rust-to-create-a-timeout-for-a-thread-or-a-function
https://www.reddit.com/r/rust/comments/eit5oo/any_way_to_set_a_timeout_on_a_background_thread/

[No9]

OK - the reddit thread is talking about managing Rhai which is a script execution engine and much bigger beast than what we have.
While I agree we don't want to hand craft this the first link mentions recv_timeout
Have you looked at that as an option?

The other option is to provide what would really be our own timeout implementation the calls the cdc but which we can distribute consistently. Bit of a lift with this as well but managing at the process level is a clean approach.

[timbuchwaldt]

RE Reddit: True.

I implemented the recv_timeout based function. I think this might benefit from a test - but that's a task for ... not tonight.

[timbuchwaldt]

Hey @No9 - can I poke you on this? :D

[No9]

Yup - thanks for the reminder. Been at some confs but should be able to look at this tomorrow.