[AIT-251] Add an overview page for AI Transport #3094
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
||
| AI Transport runs on Ably's fault-tolerant and highly-available platform. The platform enables data to be streamed between all internet-connected devices at low latencies across the globe. Its elastic global infrastructure delivers enterprise-scale messaging that effortlessly scales to meet demand. | ||
|
|
||
| Drop AI Transport into your applications to transform unreliable HTTP connections into reliable, stateful AI experiences that keep users engaged. |
There was a problem hiding this comment.
I personally don't think we should be making our argument based on HTTP streams being unreliable. Generally they're not. The key benefits of AIT vs streamed HTTP are:
- they're stateful;
- they're bidirectional.
We should of course additionally talk about resumability etc, but if HTTP reliability was the only concern, there wouldn't be that much incentive to use AIT.
There was a problem hiding this comment.
I've reworded it away from HTTP streams entirely; e1e19ed
|
|
||
| Token streaming is the core of how LLMs deliver their responses to users. Tokens are progressively streamed to users from your LLM so that users don't need to wait for a complete response before seeing any output. | ||
|
|
||
| If you stream tokens over brittle HTTP then any interruption to the connection means that all tokens transmitted during the interruption are lost. That might be if a user switches tabs, temporarily loses network connectivity or if their browser crashes - all common scenarios that should be handled gracefully for industry-leading user experiences. |
|
|
||
| ### Enterprise controls <a id="enterprise"/> | ||
|
|
||
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [RBAC](/docs/auth/capabilities). |
There was a problem hiding this comment.
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [RBAC](/docs/auth/capabilities). | |
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [fine-grained authorization](/docs/auth/capabilities). |
I'm not that comfortable describing it as RBAC, because it's not role-based
There was a problem hiding this comment.
Changed to 'granular access controls' 5ae55a7
There was a problem hiding this comment.
Personally I think "granular" is ambiguous - in some contexts it means coarse-grained, and fine-grained in others, so I try to avoid it
|
|
||
| Regardless of whether you're building with OpenAI, Anthropic or Vercel, AI Transport enables you to add a realtime delivery layer to your application, providing the infrastructure required to deliver modern, stateful AI experiences to users. | ||
|
|
||
| AI Transport runs on Ably's fault-tolerant and highly-available platform. The platform enables data to be streamed between all internet-connected devices at low latencies across the globe. Its elastic global infrastructure delivers enterprise-scale messaging that effortlessly scales to meet demand. |
There was a problem hiding this comment.
Perhaps keywords here can include links to platform docs, e.g.:
- /docs/platform/architecture/fault-tolerance
- /docs/platform/architecture/latency
- /docs/platform/architecture/platform-scalability
|
|
||
|  | ||
|
|
||
| ## Get started |
There was a problem hiding this comment.
We will have JS guides for OpenAI and Anthropic for the release at the end of this week, which we can link to.
Python translations will follow shortly, along with Vercel and LangChain guides.
There was a problem hiding this comment.
I've put these in as placeholders, as wasn't sure of the final list before release.
|
|
||
| ## Features | ||
|
|
||
| AI Transport provides a range of features that extend the existing functionality of Ably's Pub/Sub product to enable you to deliver reliable, stateful AI experiences, that provide the first class UX that is expected from modern applications. |
There was a problem hiding this comment.
AIT is not just Pub/Sub, as LiveObjects will be used for some state sync requirements, and (eventually) we expect to have support in Chat for conversational agents (although this is a bit further out)
There was a problem hiding this comment.
Yes, but I couldn't find anything using LO yet in the docs I was working from. Happy to add if there will be some docs on this before release.
|
|
||
| Token streaming is the core of how LLMs deliver their responses to users. Tokens are progressively streamed to users from your LLM so that users don't need to wait for a complete response before seeing any output. | ||
|
|
||
| If you stream tokens over brittle HTTP then any interruption to the connection means that all tokens transmitted during the interruption are lost. That might be if a user switches tabs, temporarily loses network connectivity or if their browser crashes - all common scenarios that should be handled gracefully for industry-leading user experiences. |
There was a problem hiding this comment.
Or refreshes the page
|
|
||
| ### State hydration <a id="hydration"/> | ||
|
|
||
| AI Transport benefits from all the same functionality of Ably's platform. One of the key features for AI Transport is message history; enabling users and agents to retrieve the state of their conversation at any point in time. |
There was a problem hiding this comment.
In addition to message history, synced state via LiveObjects will be a key part of the story (although the state sync stuff isnt yet documented, but is high priority for us)
There was a problem hiding this comment.
Will this exist before release?
|
|
||
| [Read more about state hydration](/docs/ai-transport/features/token-streaming/message-per-response#hydration). | ||
|
|
||
| ### Status-aware cost controls <a id="cost"/> |
There was a problem hiding this comment.
I think the value prop is more than just cost control - it more generally enables async workloads, i.e. a user can go away, the agent continues work, can notify the user via push notifications when done, and the user can return at any point to hydrate and resume the latest session state
There was a problem hiding this comment.
Updated that section, but not sure if it has too much overlap with state hydration or not now: 77d9ca6
|
|
||
| Take a look at some example code running in-browser of the sorts of features you can build with AI Transport underpinning your applications: | ||
|
|
||
| <Tiles> |
There was a problem hiding this comment.
We have an example for message per token (in PR) and message per response (coming soon), but these examples are not model-specific (they use a mock LLM)
There was a problem hiding this comment.
I put this in as a placeholder too (same as the getting started guides). Will they both be ready for release?
Description
This PR adds an overview page for AI Transport.
Checklist