Add timeout support to the Agnostic RPC system

[tegefaulkes]

Specification

We need to support default and client supplied timeouts for the RPCServer handlers.

We need to support some features with this.

• User can provide a timeout when making a client call
• Client calls have default timeouts,
• When a client times out the connection is immediately ended with a timeout code.
• Handlers are provided with a timeout timer as a TimedCancellable context
• Handlers have a default timeout provided
• Handler timeout can be overridden by the client supplied value but only if it's less than the default.
• When a handler times out is the abort is advisory and then the connection is forced closed after a small delay.

I need to do some prototyping with how I want to implement this. We have two options.

1. The RPC system supports timeouts directly.
2. RPC system allows timeouts to be supplied via the middleware.

Note that the raw handlers don't support the middleware, It's pretty low level in that respect. So raw handlers wouldn't support timeouts with option 2. Conversely Option 1 means adding a timeout field to the RPC messages since by design the RPC system doesn't interpret or apply constraints to the params.

Given some constraints the first option would mean the timeout feature is always available and enforced. The second option means it's up to the implementation of the middleware and handlers to enforce it. I need to weigh both options after some prototyping. I'm leaning towards option 1, but It may require appending a field to the JSONRPC request message format.

RPCServer changes

1. Two parameters have been added to the RPCServer when creating it. These are defaultTimeout and forceEndDelay. defaultTimeout sets the default amount of time before a handler times out and an abort signal is sent. Since this signal is advisory the forceEndDelay (name subject to change) sets the time between the signal and the streamPair being directly aborted.
2. When handling a stream a timeout timer is created sends a timeout abort to the abort controller. When created it is started with a delay of defaultTimeout. When a handler is selected the timer is reset with the provided timeout for the handler if is is less than the default. If none is set then the timer is just refresh-ed.
3. The duplex handler and by extension the other 3 basic handler types will refresh the timeout timer every time a message is sent or received. This is not done by the raw handler for now. A raw handler can directly affect the timeout timer by refreshing or resetting it.
4. The handlers have a timeout parameter that can be set when implementing the handler. This will be used as the timeout if it defined and less than the RPCServer defaultTimeout that was provided.
5. The signal and timer are provided to the handlers as a ctx: ContextTimed. The timer can be refresh-ed or reset(delay) or even cancel(reason). This gives us the freedom to override the timeout behaviour from the handlers if we want that. If that is not desired, we can reduce the functionality by casting it with a more restrictive interface.

Additional context

• Related Update GRPC client calls and service handlers to support cancellations and deadlines #466
• Related Apply timedCancellable across the board to control how long side-effects are allowed to complete #450
• Related Apply timedCancellable to Sigchain such as the deadline in nodes claim process #243

Tasks

1. RPCServer handle timeouts and provide them to the handlers
2. RPCClient takes a timeout value or default and force closes the connection when exceeded.
3. RPCClient can communicate it's timeout to the RPCServer.

[tegefaulkes]

This requires some prototyping and expansion of the spec. I think the cleanest way to implement this is option 1. But that means modifying the JSON RPC messages structure which we are trying to avoid. Mind that if the timeout parameter is optional it shouldn't conflict with the normal structure. Think of it as extending the message structure.
RPCMessageRequest & {timeout? number};

[CMCDragonkai]

On the receiving side we should default to a value even if not set.

Absolute timeout vs keep alive timeout. For streams it could default to null for absolute timeout.

Keep alive timeout should be something that each rpc params sets up themselves.

I'm just wondering how we would prevent any DoS and we should avoid providing infinite streams for agent service so that we don't allow any DoS between agents.

[tegefaulkes]

This is comprised of a two stages.

1. Updating the RPC code to support timeouts and provide them to handlers via the ctx: ContextTimed parameter.
2. updating all handlers to make use of the timeouts as well as set default timeouts.

The first taking maybe 1-2 days, while the 2nd is tedious given the amount of handlers so It could take 2-3 days. This could be done before or after the agent handler migration.

[CMCDragonkai]

I think this or QUIC should be tackled first. The agent rpc migration has to wait until quic is ready and I'd like to have this ready for testing. So this and quic then the agent handler migration.

[tegefaulkes]

I'll start with the server side default timeouts. For this the following changes need to be made

1. Timeout parameter added to the handler classes so they can be set when defining the handler.
2. Handlers now take a TimedCancellable context when they are called. Can we use the timed cancellable decorator here? How does that work with extended methods?
3. The RPCServer will take a default timeout parameter. This should be a relatively large value to accommodate long running RPC handlers.
4. When handling a stream we create the timeout and abort signal and pass it along to the handler. The value of the timeout should be the minimum out of the global default, handler default and the client provided value.

[CMCDragonkai]

For streams it must be possible to have a timeout as infinite or the timeout would be a keep alive applied to per-message. 13 Mar 2023 16:35:55 Brian Botha ***@***.***>:

I'll start with the server side default timeouts. For this the following changes need to be made

1. > Timeout parameter added to the handler classes so they can be set when defining the handler. 2. > Handlers now take a *TimedCancellable* context when they are called. Can we use the timed cancellable decorator here? How does that work with extended methods? 3. > The RPCServer will take a default timeout parameter. This should be a relatively large value to accommodate long running RPC handlers. 4. > When handling a stream we create the timeout and abort signal and pass it along to the handler. The value of the timeout should be the minimum out of the global default, handler default and the client provided value.

…

— Reply to this email directly, view it on GitHub[#510 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAE4OHPJ77AA3YD3LBWDFPDW36VNRANCNFSM6AAAAAAVOEGQM4]. You are receiving this because you commented.[Tracking image][https://github.com/notifications/beacon/AAE4OHPOP2LVZFGUYF5H2Z3W36VNRA5CNFSM6AAAAAAVOEGQM6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSXOKAUE.gif]

[CMCDragonkai]

Now that I think about it. It makes sense for streams (non-unary rpc) that would use the timeout to mean per-message timeout. 13 Mar 2023 16:38:23 Roger Qiu ***@***.***>:

For streams it must be possible to have a timeout as infinite or the timeout would be a keep alive applied to per-message. 13 Mar 2023 16:35:55 Brian Botha ***@***.***>: > > I'll start with the server side default timeouts. For this the following changes need to be made >

1. >> Timeout parameter added to the handler classes so they can be set when defining the handler. 2. >> Handlers now take a *TimedCancellable* context when they are called. Can we use the timed cancellable decorator here? How does that work with extended methods? 3. >> The RPCServer will take a default timeout parameter. This should be a relatively large value to accommodate long running RPC handlers. 4. >> When handling a stream we create the timeout and abort signal and pass it along to the handler. The value of the timeout should be the minimum out of the global default, handler default and the client provided value.

…

> > — > Reply to this email directly, view it on GitHub[#510 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAE4OHPJ77AA3YD3LBWDFPDW36VNRANCNFSM6AAAAAAVOEGQM4]. > You are receiving this because you commented.[Tracking image][https://github.com/notifications/beacon/AAE4OHPOP2LVZFGUYF5H2Z3W36VNRA5CNFSM6AAAAAAVOEGQM6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSXOKAUE.gif] >

[tegefaulkes]

That's possible but a little more tricky. It requires some feed back compared to the easier method of creating the timeout and forgetting about it.

[CMCDragonkai]

The reason is because a timeout for the entire duration of a stream is kind of useless, we don't really have a situation where that would be useful.

However a timeout for unary is ultimately intended to prevent DOS and agents that are just stuck.

Therefore a similar reason should apply for streams, it should be a per-message timer.

Furthermore:

1. null - explicitly infinite
2. undefined - should mean I don't care, set the default
3. number - set a specific number

On the client side, during any given call, the client can set one of the 3 values above.

On the server side, it can do one of several things:

1. Accept number
2. Consider that number is too high, and set to the max
3. Accept undefined and use the default
4. Accept null and use the default
5. Consider that null is too high, and set to the max
6. Consider that number is too low, and set to the min

So therefore on the server side the number of configurations is a bit more sophisticated.

You could say:

1. This handler has max timeout of 3000 (therefore anything higher than 3000 or null is set to 3000)
2. This handler has max timeout of null - meaning infinity
3. This handler has a min timeout of 100
4. This handler has a min timeout of 0

So that could mean a handler should be able to set a min and max timeout to provide a range.

I'm actually not sure whether we even should have min/max, or just have a default.

Furthermore, I'm also not sure if we even allow the client to "request" a timeout. Perhaps it makes more sense that timeouts can be set on both sides, but there's no actual negotiation.

Here's an idea. Timeouts can be set on both sides, but they are only advisory to the other side.

1. If a client sets a timeout (and the server doesn't set one), it's a deadline for the response. If a response does not arrive. Then the call is timed out with an abortion.
2. If a server sets a timeout (and the client doesn't set one), it's a deadline for responding - as in the default timer to pass down to its operations.
3. If both the client and server sets a timeout, then the resulting timeout will be the lower of the 2.
4. If both the client and server does not set a timeout, then the resulting timeout is infinity.

[CMCDragonkai]

I think the last idea is clearer... to do. And that is also applied per-message when in CS, DS, SS.

As for timeout applied to replies from the client, let's address this later, it's not that important.

[CMCDragonkai]

@tegefaulkes the timed cancellable functions works by wrapping functions. It's a higher order function that wraps a lower order function.

There are actually 2 ways:

1. Using the decorator applies to only class methods.
2. Use the HOF wrapper

The first way can only work if handlers are defined as methods in a class. Arrow functions or just plain exported functions can only use the HOF wrapper. See the tests for how to use it.

[CMCDragonkai]

BTW, there are 2 possible abortions.

1. Timer expires, that aborts all subsequent operations - remember when we had the agent still doing work even when the client already died or disappeared. We want to prevent this!
2. The client just cancelling the stream, if that occurs, we should also abort.

TimedCancellable supports both.

You may want to create symbols specific to the 2nd case though. See how we did something similar in the tasks domain.