Node crashes/exits ungracefully on uncaught exceptions/events relating to QUIC connections or QUIC streams

[tegefaulkes]

Describe the bug

The node seems to be randomly crashing with ErrorQUICConnectionIdleTimeout.

INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 92:Destroy QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 92:Destroyed QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 96:Create QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 96:Created QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.RPCServer:Handling stream with method (nodesConnectionSignalInitial)
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05.QUICStream 144:Create QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05.QUICStream 144:Created QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.RPCServer:Handled stream with method (nodesConnectionSignalInitial)
INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 96:Destroy QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.QUICServer.QUICConnection f4ba6d8b49c528090fee5cab01c73326e6b46e58.QUICStream 96:Destroyed QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient:ErrorQUICConnectionIdleTimeout: QUIC Connection max idle timeout exhausted - 
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05:ErrorQUICConnectionIdleTimeout: QUIC Connection max idle timeout exhausted - 
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05:Stop QUICConnection
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnection [v2kgfdi6dutt0mmv78q58la30mafige86igvpch0dbtnfmb8l70sg@::ffff:159.196.126.185:51410]:Destroying NodeConnection
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient:Destroy QUICClient to [::ffff:159.196.126.185]:51410
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05.QUICStream 144:Destroy QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05:ErrorQUICConnectionIdleTimeout: QUIC Connection max idle timeout exhausted - 
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05:ErrorQUICConnectionIdleTimeout: QUIC Connection max idle timeout exhausted - 
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05.QUICStream 144:Destroyed QUICStream
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient.QUICConnection 0bb985a87456834e1683140788ee89bfc5ef4f05:Stopped QUICConnection
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [159.196.126.185:51410].QUICClient:Destroyed QUICClient to [::ffff:159.196.126.185]:51410
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnection [v2kgfdi6dutt0mmv78q58la30mafige86igvpch0dbtnfmb8l70sg@::ffff:159.196.126.185:51410]:Destroyed NodeConnection
ErrorQUICConnectionIdleTimeout

cmcdragonkai ➜ matrix-framework-13-ryzen-7040  ➜ ~/Projects/Polykey-CLI
 $ 

It also can produce - any of these things are being leaked up to the process for it to do an ungraceful exit/crash:

• ErrorQUICConnectionIdleTimeout
• ErrorNodeConnectionTransportGenericError
• TypeError: Invalid state: WritableStream is closed

To Reproduce

1. Run a Polykey agent.
2. It will crash on its own after a while.

Expected behavior

Should not crash with ErrorQUICConnectionIdleTimeout

Screenshots

Platform (please complete the following information)

• Version: Polykey-CLI 0.4.1.

Additional context

• This is related to Polykey-CLI#185 because it appears that a node that is leaking a connection will fail to shutdown gracefully, and can then propagate a problem to a remote node too, causing it to shutdown ungracefully.
• Sometimes, the node will also crash with ErrorNodeConnectionTransportGenericError.
• The agent will sometimes also fail with ErrorQUICStreamInternal: Failed to prime local stream state with a 0-length message. (source: Polykey-CLI#198 (comment))
• Create tests for PolykeyAgent stability during random agent-agent interactions #8

Notify maintainers

@tegefaulkes

[linear]

ENG-334 Node crashing with `ErrorQUICConnectionIdleTimeout`

[CMCDragonkai]

I've seen this twice now.

It particularly happens when there's a failure on a QUIC connection from the other remote side node. Like if they force close their node, our node then fails.

INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnectionForward [79.103.253.41:44353].QUICClient:Destroyed QUICClient to [::ffff:79.103.253.41]:44353
INFO:polykey.PolykeyAgent.NodeConnectionManager.NodeConnection [veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g@::ffff:79.103.253.41:44353]:Destroyed NodeConnection
ErrorQUICConnectionIdleTimeout

Twice when Alexi who is in Greece from the IP:

79.103.253.41 	N/A     	veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g	58143	55780  	0

Who tried to clone my vault with polykey vaults clone resulted in ErrorRPCTimeout. This is probably due to a slow clone or something.

Then he had to force shutdown his agent, because graceful agent stop was not possible. This is likely due to QUIC again as there are possibly open fds or open connections that hold the process open.

Finally after force closing their agent, and waiting for some time… my agent ended up failing.

You can see the prior connection is exactly the NodeID of the node that was force closed from Greece.

So this error is connected to:

1. Possibly a slow clone causing an ErrorRPCTimeout, which left a connection IO continue to run without being cancelled.
2. This "leaked" connection IO is probably causing the agent to not shutdown gracefully. We know this because nodejs will not shutdown when there's IO/fds that are kept open. Otherwise nodejs always shutsdown when there is not an infinite loop.
3. Then a leaked connection remotely seems to create a problem on the other side, thus causing a ErrorQUICConnectionIdleTimeout. This appears to be an uncaught exception, and results in the entire agent to ungracefully shutdown.

[CMCDragonkai]

On the prior attempt:

Cloning on Windows results in ErrorRPCTimeout, everything else worked

Windows agent had to be force closed, it looks like IO is keeping the process open.

The nodes connections:

79.103.253.41 N/A veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g 33226 52481 0

->

79.103.253.41 N/A veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g 33226 56108 0
79.103.253.41 N/A veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g 33226 55531 0
79.103.253.41 N/A veu6q4pppbeoumv9n3uj819t6c5thr8kb9jshpfco85l4cvst4c6g 33226 55695 0

Then upon restarting the agent, and reconnecting to my Linux agent, the vaults clone still failed with ErrorRPCTimeout.

And then, a few minutes later, I get this error on my Linux the agent is now crashed and no longer running:

ErrorNodeConnectionTransportGenericError: Transport received a generic error

What's interesting, is that I find that sometimes… even this is also an uncaught exception:

ErrorNodeConnectionTransportGenericError: Transport received a generic error

This also causes the agent to ungracefully exit. So the first time what I described happen, this was the exception. But the second time, was the ErrorQUICConnectionIdleTimeout.

[CMCDragonkai]

I'm going to rename this issue, given that there is now evidence of at least 2 different errors that is being uncaught and causing ungraceful exit.

[CMCDragonkai]

This is connected to #185, because that is likely caused by the one side leaking connections. When this occurs, it's a resource/IO/fd leak. And that will cause the graceful shutdown to halt forever.

[tegefaulkes]

ErrorQUICConnectionIdleTimeout is the expected error here and should be handled just fine. When an connection is orphaned by one side of the connection then it is cleaned up by the idle timeout.

The ErrorNodeConnectionTransportGenericError error is very likely the problem here. So we'll need to trace why that's thrown and what's causing it exactly.

[CMCDragonkai]

Pull error handling always uses standard exception handlers to deal with the problem. This we understand.

Push error handling looks underimplemented. It relies on object-system context error handlers. We need to make sure that QUIC errors are being "captured" by the nodes domain fully so that it doesn't just bubble to the process context error handlers which is likely being defined in the ExitHandler.

This is because we're still working out the architecture of push-flow in MatrixAI/Polykey#444.

The fact that these exceptions are all bubbling up to the process means that errors are not properly handled within NodeConnectionManager.

[aryanjassal]

I had left my PC running for over 2 days with Polykey agent running in the background, and I got this error after about 5 hours of running.

ErrorQUICStreamInternal: Failed to prime local stream state with a 0-length message

[aryanjassal]

I just observed this exact error when I ran this command.

$ npm run polykey -- secrets mkdir vault:aslkfj/a

> polykey-cli@0.15.0 polykey
> ts-node src/polykey.ts secrets mkdir vault:aslkfj/a

mkdir: cannot create directory aslkfj/a: No such file or directory
ErrorPolykeyCLIMakeDirectory: Failed to create one or more directories - Failed to create one or more directories

This command took about 10 seconds, which is way too long for this, and then the agent crashed with the 0-length message error. I tried this and got a RPC timeout error.

$ npm run polykey -- secrets mkdir vault:aslkfj/asf/asd/fa

> polykey-cli@0.15.0 polykey
> ts-node src/polykey.ts secrets mkdir vault:aslkfj/asf/asd/fa

ErrorPolykeyCLIUnexpectedError: An unexpected error occured - Thrown 'ErrorRPCTimedOut'
  cause: ErrorRPCTimedOut: RPC has timed out

However, this also seems inconsistent, as trying to replicate this issue again yielded the command taking a while, but succeeding in the end without the agent crashing with QUIC.