docs: add human-in-the-loop page for AI Transport

[GregHolmes]

Description

Add documentation for human-in-the-loop (HITL) workflows in AI Transport, covering how to implement human oversight of AI agent actions.

Topics covered:

• Why human-in-the-loop matters (safety, compliance, trust, accountability)
• Request-approval pattern over Ably channels
• Using JWT claims and capabilities for authorization
• Handling approval timeouts and rejections

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mschristensen]

Thanks Greg, needs a rebase but commented on the HITL doc page. Some good ideas in there but have flagged some bits for discussion, let me know if you want to catch up on the approach.

[mschristensen]

Suggested change

	Human-in-the-loop (HITL) enables human oversight of AI agent actions. When an agent needs to perform sensitive operations, such as modifying data, making purchases, or accessing restricted resources, the action is paused and routed to an authorized human for approval before execution.
	Human-in-the-loop (HITL) enables human oversight of AI agent actions. When an agent needs to perform sensitive operations, such as modifying data, performing sensitive actions, or accessing restricted resources, the action is paused and routed to an authorized human for approval before execution.

[mschristensen]

I know we need to update the other pages, but I think you (correctly) argued in the past that we should not use bold prefixes in bullets, which smells like an LLM

[GregHolmes]

You're right! I asked Claude to review my work and improve where needed. I may have missed this! Updated.

[mschristensen]

This feels like typical AI word salad, how about:

Suggested change

	HITL doesn't slow down AI — it ensures the right actions happen at the right time with appropriate oversight.
	HITL puts a human approval step in front of agent actions that carry risk or uncertainty.

[mschristensen]

An additional point thats not captured here - sometimes HITL is useful when the agent needs to more clarification or human input before taking further action.

[mschristensen]

Hmm, I'm not sure we need to use a dedicated channel (with capabilities on hitl:* etc). Why not just sent the HITL request from the agent on the same channel? The agent is going to do a lookup when it gets the response from the user anyway to determine if it is allowed to make that decision, so I don't think these needs to be enforced at the Ably channel level?

[GregHolmes]

I had to make a few changes elsewhere as the page was more focused on dedicated channel capabilities etc. So I've stuck it (and the couple changes recommended above) into a single fixup: d2c24d0

This may change depending on your feedback below though.

[mschristensen]

I think this is a bit overly opinionated - I don't think all HITL loops need to carry a risk level or a reason?

[GregHolmes]

I've removed the opinionated bits from this section including updating the code:

8dd7f22

[mschristensen]

Again I think this is getting too opinionated about how to implement HITL flows. Instead, this doc should focus on the core Ably primitives you would rely on to implement HITL.

[GregHolmes]

I think this was covered in the comment above.

[mschristensen]

Same here. I think the doc should just cover an agent sending a HITL request to a user, and obtaining the response from the user (with verified identity/role etc) and taking an action accordingly.

[GregHolmes]

I've removed all the opinionated work from this section: 901564a

[mschristensen]

Application logging is a customer's concern, unrelated to Ably, and doesn't need to be documented here.

However, an Ably primitive that might be relevant to audit trails is integration rules, which I think would be valid to call out here (although I think a brief mention and link to integration rule docs is sufficient, rather than a full fledged example).

[GregHolmes]

Updated in the last 3 fixups.

[mschristensen]

Again I think this is an implementation question, possible suitable for a guide, but not for feature documentation.

[GregHolmes]

Agreed, I've removed this section in 0a3c7a1

[paddybyers]

This looks like it needs the base branch changing

[GregHolmes]

Thank you @mschristensen I've updated from your feedback.

@paddybyers I think Mike rebased the release branch with main, which then left my PR here outdated. But I didn't see as was off yesterday. All rebased now. Thanks for flagging.

[mschristensen]

Thanks Greg, looks good, a few last comments

[mschristensen]

I think we should remove this ASCII diagram, until we have nicely designed ones. I created a ticket: https://ably.atlassian.net/browse/AIT-257

[GregHolmes]

Thank you. I didn't think the ASCII would be a releaseable piece to show but more something that could assist design on any images we'd need to create.

c8d3d5e

[mschristensen]

This section appears twice. This one should be removed.

[GregHolmes]

Yep! I think I forgot to remove when I moved things around: 2abee1a

[mschristensen]

We shouldn't use an em-dash, which is a tell-tale sign of AI generated content

Suggested change

	Use the trusted `userClaim` from the JWT to verify the approver has sufficient permissions for the specific action. Different actions may require different authorization levels — a moderator might approve content-related actions while financial operations require an admin.
	Use the trusted `userClaim` from the JWT to verify the approver has sufficient permissions for the specific action. Different actions may require different authorization levels - for example, a moderator might approve content-related actions while financial operations require an admin.

[GregHolmes]

cb77913

Updated!

[mschristensen]

We havent shown how to use identified clients or, as used later in this doc, roles. We should document a minimal example of the JWT claims, like we do in the accepting user input docs, and include relevant links to the sessions & identity docs

[GregHolmes]

I've added a new h3 within Process the decision called Verify approver identity. Let me know what you think: 3a0588e

[mschristensen]

We haven't included this in other docs pages, and I'm not convinced it's necessary; the docs up to this point should be easy enough to follow that it's not required. A complete example imposes certain decisions on the implementation which prior docs aimed to discuss in more nuanced detail, allowing the reader to decide what is appropriate for them. This seems more suitable for a guide.

However, let me know if you disagree (and if you do, we should consider whether other docs pages need an equivalent section, and we should ticket that).

[GregHolmes]

You're right, it shouldn't be here. I've now removed: d1bfedb

We're going to cover this type of thing in the guides so there's no need in feature pages.

[mschristensen]

The mechanism could be either role (via user claims) or just the verified identity (via identified clients), so I think it's ok to just state:

Suggested change

	5. The agent receives the decision, verifies the responder's role via [user claims](/docs/ai-transport/features/sessions-identity/identifying-users-and-agents#user-claims), and proceeds accordingly.
	5. The agent receives the decision, verifies the responder's identity or role and proceeds accordingly.

...since the detail is covered later in the document.

[mschristensen]

Suggested change

	const channel = ably.channels.get('conversation:session-123');
	const channel = ably.channels.get('{{RANDOM_CHANNEL_NAME}}');

[mschristensen]

Everywhere we say "conversation channel", I think we should just say "channel"

Suggested change

	2. The agent publishes an authorization request to the conversation channel.
	2. The agent publishes an authorization request to the channel.

[mschristensen]

Suggested change

	When an agent identifies an action requiring human oversight, it publishes a request to the conversation channel. The request should include sufficient context for the approver to make an informed decision. The `requestId` enables correlation between requests and responses when handling multiple concurrent approval flows.
	When an agent identifies an action requiring human oversight, it publishes a request to the channel. The request should include sufficient context for the approver to make an informed decision. The `requestId` enables correlation between requests and responses when handling multiple concurrent approval flows.