diff --git a/api-reference/inference-api/gateway-for-other-apis.mdx b/api-reference/inference-api/gateway-for-other-apis.mdx index 6f1b6182..83e70917 100644 --- a/api-reference/inference-api/gateway-for-other-apis.mdx +++ b/api-reference/inference-api/gateway-for-other-apis.mdx @@ -1,84 +1,144 @@ --- title: "Gateway to Other APIs" -description: Access any custom provider endpoint through Portkey API +description: "Forward requests to provider paths outside unified routes (rerank, video, listen, and more) through https://api.portkey.ai/v1/.... Same auth, configs, retries, logging, and analytics as first-class gateway traffic." icon: "chart-network" --- + -This feature is available on all Portkey plans. +Available on all Portkey [plans](https://portkey.ai/pricing). -Portkey API has first-class support for monitoring and routing your requests to 10+ provider endpoints, like `/chat/completions`, `/audio`, `/embeddings`, etc. We also make these endpoints work across 250+ different LLMs. +Prefer **[unified routes](/product/ai-gateway/universal-api)** for chat, images, embeddings, messages, responses, and audio—they stay consistent across 250+ models. + +Create an API key from [**API Keys**](https://app.portkey.ai/api-keys) when onboarding. + +## Quick start + +```sh +curl https://api.portkey.ai/v1/ \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: openai" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ ... }' +``` + +Replace `` with the suffix the provider expects (`rerank`, `videos`, `listen`, etc.). On a **self-hosted** gateway, use that deployment’s host; keep the `/v1/` pattern. + +## Unified routes vs proxy + +Dedicated routes register **before** the catch-all. They use unified handlers—not the proxy. + +| Path | Handler | +|------|---------| +| `POST /v1/chat/completions` | [Chat completions](/product/ai-gateway/chat-completions) | +| `POST /v1/images/generations` | Unified image handler | +| `POST /v1/embeddings` | Unified embeddings handler | +| `POST /v1/messages` | [Messages API](/product/ai-gateway/messages-api) | +| `POST /v1/responses` | [Responses API](/product/ai-gateway/responses-api) | +| `POST /v1/audio/...` | Unified audio handler | +| `/v1/files`, `/v1/batches`, … | Dedicated handlers | +| **Anything else under `/v1/*`** | **Proxy handler** | -**However**, there are still many endpoints like Cohere's `/rerank` or Deepgram's `/listen` that are uncommon or have niche use cases. +### Pick a URL -With the **Gateway to Other APIs** feature, you can route to any custom provider endpoint using Portkey (including the ones hosted on your private setups) and get **complete logging & monitoring** for all your requests. +| Goal | URL | +|------|-----| +| Chat (configs, fallbacks, retries) | `POST https://api.portkey.ai/v1/chat/completions` (unified) | +| Image generation | `POST https://api.portkey.ai/v1/images/generations` (unified) | +| Provider path **without** a unified route | `POST https://api.portkey.ai/v1/` (proxy) | +| Force proxy on a path that would hit a unified route | `POST https://api.portkey.ai/v1/proxy/` — see [Legacy `/v1/proxy`](#legacy-v1proxy-prefix) | -## Supported HTTP Methods + +`POST https://api.portkey.ai/v1/chat/completions` does **not** use the proxy. A dedicated chat handler runs first. + - - - - - - +## How the proxy works -Both the REST API and Portkey SDKs (Python, NodeJS) support all of these HTTP methods. +For paths that reach the proxy: -# How to Integrate +1. Send **method**, **path**, **query string**, and **body** (JSON, multipart, or binary) to the gateway. +2. Portkey applies **auth**, **config**, **credentials**, **routing**, **retries**, **caching**, **guardrails**, and the rest of the gateway pipeline. +3. The request is sent to the **provider base URL** (or `x-portkey-custom-host`) with the **remaining path** the provider expects. +4. The response is essentially the **provider response** (pass-through), plus standard Portkey response headers. -1. Get your Portkey API key -2. Add your provider details to Portkey -3. Make your request using Portkey's API or SDK +[Fallback](/product/ai-gateway/fallbacks), [load balancing](/product/ai-gateway/load-balancing), and [conditional routing](/product/ai-gateway/conditional-routing) use the same config engine as every other gateway call. +## Supported HTTP methods -## 1. Get Portkey API Key -Create or log in to your Portkey account. Grab your account's API key from the ["API Keys" page](https://app.portkey.ai/api-keys). +| Method | Supported | +|--------|-----------| +| `POST` | Yes | +| `GET` | Yes (realtime path excluded on Workers) | +| `PUT` | Yes | +| `DELETE` | Yes | +| `PATCH` | No — not registered on the `/v1/*` catch-all | -## 2. Add Provider Details +## Required and optional headers -Choose one of these authentication methods: +### Required + +| Header | Purpose | +|--------|---------| +| `x-portkey-api-key` | Portkey API key | +| `x-portkey-provider` *or* `x-portkey-config` | Routing: provider slug, Model Catalog `@slug`, or full config JSON | +| `Authorization` | Provider secret when credentials are not supplied only via Model Catalog | +| `Content-Type` | Match the provider (`application/json` for JSON bodies) | + +### Optional + +| Header | Purpose | +|--------|---------| +| `x-portkey-metadata` | JSON tags for [analytics](/product/observability/metadata) and [conditional routing](/product/ai-gateway/conditional-routing) | +| `x-portkey-custom-host` | Override provider base URL — [Custom hosts](/product/ai-gateway/custom-hosts) | +| `x-portkey-forward-headers` | Comma-separated client header names to forward upstream | + + +`x-portkey-*` headers are consumed by the gateway and **not** forwarded as arbitrary upstream passthrough. Store secrets in `Authorization` (or the header the provider expects), or in [Model Catalog](https://app.portkey.ai/model-catalog). + + +Full header list: [Inference API headers](/api-reference/inference-api/headers). + +## Authenticate - -Add your provider credentials to [Model Catalog](https://app.portkey.ai/model-catalog) and use the AI Provider slug to authenticate your requests. + +Add credentials in [Model Catalog](https://app.portkey.ai/model-catalog). Use `x-portkey-provider: @` and omit `Authorization` when the catalog supplies auth. - ```sh cURL curl https://api.portkey.ai/v1/rerank \ -H "Content-Type: application/json" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ - -H "x-portkey-provider: @cohere-prod" \ + -H "x-portkey-provider: @cohere-prod" ``` -```py Python +```python Python from portkey_ai import Portkey portkey = Portkey( - api_key = "PORTKEY_API_KEY", - provider = "@cohere-prod" + api_key="PORTKEY_API_KEY", + provider="@cohere-prod", ) ``` -```ts JavaScript -import Portkey from 'portkey-ai'; +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: 'PORTKEY_API_KEY', - provider: '@cohere-prod' + apiKey: "PORTKEY_API_KEY", + provider: "@cohere-prod", }); ``` + -Model Catalog lets you: -- Manage all provider credentials in one place -- Set budget limits & rate limits per provider -- Rotate between different provider keys +Model Catalog centralizes keys, [budget and rate limits](/product/model-catalog/integrations#3-budget-%26-rate-limits), and rotation. - -Set the provider name from one of Portkey's 40+ supported providers list and use your provider credentials directly with each request. + +Use a supported provider slug and pass the provider secret on each request. ```sh cURL @@ -86,268 +146,345 @@ curl https://api.portkey.ai/v1/rerank \ -H "Content-Type: application/json" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ -H "x-portkey-provider: cohere" \ - -H "Authorization: Bearer $COHERE_API_KEY" \ + -H "Authorization: Bearer $COHERE_API_KEY" ``` -```py Python + +```python Python from portkey_ai import Portkey portkey = Portkey( - api_key = "PORTKEY_API_KEY", - provider = "cohere", - Authorization = "Bearer COHERE_API_KEY" + api_key="PORTKEY_API_KEY", + provider="cohere", + Authorization="Bearer COHERE_API_KEY", ) ``` -```ts JavaScript -import Portkey from 'portkey-ai'; + +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: 'PORTKEY_API_KEY', + apiKey: "PORTKEY_API_KEY", provider: "cohere", - Authorization: "Bearer COHERE_API_KEY" + Authorization: "Bearer COHERE_API_KEY", }); ``` - -Route to your privately hosted model endpoints. -- Choose a compatible provider type (e.g., `openai`, `cohere`) -- Provide your endpoint URL with `customHost` -- Include `Authentication` if needed + +Choose a compatible `x-portkey-provider` (`openai`, `cohere`, etc.). Set the URL with `x-portkey-custom-host` (header) or `custom_host` / `customHost` in the SDK. - ```sh cURL curl https://api.portkey.ai/v1/rerank \ -H "Content-Type: application/json" \ -H "x-portkey-api-key: $PORTKEY_API_KEY" \ -H "x-portkey-provider: cohere" \ - -H "x-portkey-custom-host: https://182.145.24.5:8080/v1" \ - -H "Authorization: Bearer $COHERE_API_KEY" \ + -H "x-portkey-custom-host: https://your-public-host.example.com/v1" \ + -H "Authorization: Bearer $COHERE_API_KEY" ``` -```py Python +```python Python from portkey_ai import Portkey portkey = Portkey( - api_key = "PORTKEY_API_KEY", - provider = "cohere", - custom_host = "https://182.145.24.5:8080/v1", - Authorization = "Bearer COHERE_API_KEY" + api_key="PORTKEY_API_KEY", + provider="cohere", + custom_host="https://your-public-host.example.com/v1", + Authorization="Bearer COHERE_API_KEY", ) ``` -```ts JavaScript -import Portkey from 'portkey-ai'; +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: 'PORTKEY_API_KEY', + apiKey: "PORTKEY_API_KEY", provider: "cohere", - customHost: "https://182.145.24.5:8080/v1", - Authorization: "Bearer COHERE_API_KEY" + customHost: "https://your-public-host.example.com/v1", + Authorization: "Bearer COHERE_API_KEY", }); ``` + + +On Portkey **SaaS**, custom hosts must be **publicly reachable**. Private IPs apply only to **self-hosted / hybrid enterprise** setups. See [Custom hosts](/product/ai-gateway/custom-hosts). + -## 3. Make Requests - - - -Construct your request URL: -1. Portkey Gateway base URL remains same: `https://api.portkey.ai/v1` -2. Append your custom endpoint at the end of the URL: `https://api.portkey.ai/v1/{provider-endpoint}` - - -```bash POST -curl --request POST \ - --url https://api.portkey.ai/v1/rerank \ - --header 'Content-Type: application/json' \ - --header 'x-portkey-api-key: $PORTKEY_API_KEY' \ - --header 'x-portkey-provider: @cohere-prod' \ - --data '{ - "model": "rerank-english-v2.0", - "query": "What is machine learning?", - "documents": [ - "Machine learning is a branch of AI focused on building systems that learn from data.", - "Data science involves analyzing and interpreting complex data sets." - ] - }' -``` +Base URL for all proxy paths: `https://api.portkey.ai/v1` plus the provider suffix (e.g. `https://api.portkey.ai/v1/rerank`). -```bash GET -curl --request GET \ - --url https://api.portkey.ai/v1/collections \ - --header 'Content-Type: application/json' \ - --header 'x-portkey-api-key: $PORTKEY_API_KEY' \ - --header 'x-portkey-provider: @provider-prod' -``` +## Examples -```bash PUT -curl --request PUT \ - --url https://api.portkey.ai/v1/collections/my-collection \ - --header 'Content-Type: application/json' \ - --header 'x-portkey-api-key: $PORTKEY_API_KEY' \ - --header 'x-portkey-provider: @provider-prod' \ - --data '{ - "metadata": { - "description": "Updated collection description" - } - }' -``` +### POST `/rerank` (Model Catalog) -```bash DELETE -curl --request DELETE \ - --url https://api.portkey.ai/v1/collections/my-collection \ - --header 'Content-Type: application/json' \ - --header 'x-portkey-api-key: $PORTKEY_API_KEY' \ - --header 'x-portkey-provider: @provider-prod' + +```sh cURL +curl https://api.portkey.ai/v1/rerank \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @cohere-prod" \ + -d '{ + "model": "rerank-english-v2.0", + "query": "What is machine learning?", + "documents": [ + "Machine learning is a branch of AI focused on building systems that learn from data.", + "Data science involves analyzing and interpreting complex data sets." + ] + }' ``` - - - -The SDK fully supports all HTTP methods: `POST`, `GET`, `PUT`, and `DELETE`. - - -```python POST +```python Python from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", - provider="@PROVIDER" + provider="@cohere-prod", ) response = portkey.post( - '/rerank', + "/rerank", model="rerank-english-v2.0", query="What is machine learning?", documents=[ "Machine learning is a branch of AI focused on building systems that learn from data.", - "Data science involves analyzing and interpreting complex data sets." - ] + "Data science involves analyzing and interpreting complex data sets.", + ], ) +print(response) ``` -```python GET -from portkey_ai import Portkey +```javascript JavaScript +import Portkey from "portkey-ai"; -portkey = Portkey( - api_key="PORTKEY_API_KEY", - provider="@PROVIDER" -) +const portkey = new Portkey({ + apiKey: "PORTKEY_API_KEY", + provider: "@cohere-prod", +}); -response = portkey.get('/collections') +const response = await portkey.post("/rerank", { + model: "rerank-english-v2.0", + query: "What is machine learning?", + documents: [ + "Machine learning is a branch of AI focused on building systems that learn from data.", + "Data science involves analyzing and interpreting complex data sets.", + ], +}); + +console.log(response); ``` + + +### POST `/videos` (direct provider auth) + +Replace the body with the payload the provider’s video API expects. -```python PUT + +```sh cURL +curl https://api.portkey.ai/v1/videos \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: openai" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "sora-1", + "prompt": "A red balloon floating over a city skyline at sunset" + }' +``` + +```python Python from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", - provider="@PROVIDER" + provider="openai", + Authorization="Bearer OPENAI_API_KEY", ) -response = portkey.put( - '/collections/my-collection', - metadata={ - "description": "Updated collection description" - } +response = portkey.post( + "/videos", + model="sora-1", + prompt="A red balloon floating over a city skyline at sunset", ) +print(response) +``` + +```javascript JavaScript +import Portkey from "portkey-ai"; + +const portkey = new Portkey({ + apiKey: "PORTKEY_API_KEY", + provider: "openai", + Authorization: "Bearer OPENAI_API_KEY", +}); + +const response = await portkey.post("/videos", { + model: "sora-1", + prompt: "A red balloon floating over a city skyline at sunset", +}); + +console.log(response); +``` + + +### GET, PUT, DELETE + +The SDK exposes `.get()`, `.post()`, `.put()`, and `.delete()` for proxy paths. + + +```sh cURL +curl https://api.portkey.ai/v1/collections \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @provider-prod" + +curl -X PUT https://api.portkey.ai/v1/collections/my-collection \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @provider-prod" \ + -d '{"metadata": {"description": "Updated description"}}' + +curl -X DELETE https://api.portkey.ai/v1/collections/my-collection \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @provider-prod" ``` -```python DELETE +```python Python from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", - provider="@PROVIDER" + provider="@provider-prod", ) -response = portkey.delete('/collections/my-collection') -``` - - +portkey.get("/collections") - -The SDK fully supports all HTTP methods: `POST`, `GET`, `PUT`, and `DELETE`. +portkey.put( + "/collections/my-collection", + metadata={"description": "Updated description"}, +) - -```javascript POST -import Portkey from 'portkey-ai'; +portkey.delete("/collections/my-collection") +``` + +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: "PORTKEY_API_KEY", - provider: "@cohere-prod" + apiKey: "PORTKEY_API_KEY", + provider: "@provider-prod", }); -const response = await portkey.post('/rerank', { - model: "rerank-english-v2.0", - query: "What is machine learning?", - documents: [ - "Machine learning is a branch of AI focused on building systems that learn from data.", - "Data science involves analyzing and interpreting complex data sets." - ] +await portkey.get("/collections"); + +await portkey.put("/collections/my-collection", { + metadata: { description: "Updated description" }, }); + +await portkey.delete("/collections/my-collection"); ``` + -```javascript GET -import Portkey from 'portkey-ai'; +### Custom host (full `POST /rerank`) -const portkey = new Portkey({ - apiKey: "PORTKEY_API_KEY", - provider: "@cohere-prod" -}); + +```sh cURL +curl https://api.portkey.ai/v1/rerank \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: cohere" \ + -H "x-portkey-custom-host: https://your-public-host.example.com/v1" \ + -H "Authorization: Bearer $COHERE_API_KEY" \ + -d '{"model":"rerank-english-v2.0","query":"What is machine learning?","documents":["Document one.","Document two."]}' +``` -const response = await portkey.get('/collections'); +```python Python +from portkey_ai import Portkey + +portkey = Portkey( + api_key="PORTKEY_API_KEY", + provider="cohere", + custom_host="https://your-public-host.example.com/v1", + Authorization="Bearer COHERE_API_KEY", +) + +response = portkey.post( + "/rerank", + model="rerank-english-v2.0", + query="What is machine learning?", + documents=["Document one.", "Document two."], +) +print(response) ``` -```javascript PUT -import Portkey from 'portkey-ai'; +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: "PORTKEY_API_KEY", - provider: "@cohere-prod" + apiKey: "PORTKEY_API_KEY", + provider: "cohere", + customHost: "https://your-public-host.example.com/v1", + Authorization: "Bearer COHERE_API_KEY", }); -const response = await portkey.put('/collections/my-collection', { - metadata: { - description: "Updated collection description" - } +const response = await portkey.post("/rerank", { + model: "rerank-english-v2.0", + query: "What is machine learning?", + documents: ["Document one.", "Document two."], }); + +console.log(response); ``` + -```javascript DELETE -import Portkey from 'portkey-ai'; +### Fallback with `x-portkey-config` -const portkey = new Portkey({ - apiKey: "PORTKEY_API_KEY", - provider: "@cohere-prod" -}); +Same JSON shape as [Configs](/product/ai-gateway/configs): strategies, targets, retries, hooks. -const response = await portkey.delete('/collections/my-collection'); +```sh +curl https://api.portkey.ai/v1/rerank \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-config: {\"strategy\":{\"mode\":\"fallback\"},\"targets\":[{\"provider\":\"@cohere-prod\"},{\"provider\":\"@cohere-backup\"}]}" \ + -d '{"model":"rerank-english-v2.0","query":"...","documents":["..."]}' ``` - - - -## End-to-end Example +Minify JSON for the header (e.g. `jq -c . config.json`) or set the header in application code. + + +Root-level **`options`** in `x-portkey-config` is rejected. Use the current config shape (`strategy`, `targets`, per-target fields). + - +## Legacy `/v1/proxy` prefix -A complete example showing document reranking with Cohere: +Prefix with **`/v1/proxy`** so the gateway strips that segment and forwards the remainder—forcing **proxy** semantics on a path that would otherwise hit a **unified** handler (raw passthrough on `chat/completions`, for example). + +```sh +curl https://api.portkey.ai/v1/proxy/chat/completions \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: openai" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Hello"}]}' +``` + +For new work, prefer a **unified** route or a **distinct** proxy path. Use `/v1/proxy/...` only when passthrough on a unified path is required. + +## Advanced examples + + + ```python from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", - provider="@COHERE_VIRTUAL_KEY" + provider="@COHERE_VIRTUAL_KEY", ) response = portkey.post( - '/rerank', + "/rerank", return_documents=False, max_chunks_per_doc=10, model="rerank-english-v2.0", @@ -355,38 +492,36 @@ response = portkey.post( documents=[ "Carson City is the capital city of the American state of Nevada.", "Washington, D.C. is the capital of the United States.", - "Capital punishment has existed in the United States since before its founding." - ] + "Capital punishment has existed in the United States since before its founding.", + ], ) ``` - - -A complete example showing document reranking with Cohere models on AWS Bedrock: + -For Bedrock providers, pass the complete ARN along with the model name in the `model` field to make a successful request. +For Bedrock, pass the full ARN with the model name in the `model` field. ```sh cURL -curl --location 'https://api.portkey.ai/v1/rerank' \ ---header 'x-portkey-api-key: $PORTKEY_API_KEY' \ ---header 'x-portkey-provider: @BEDROCK_PROVIDER_SLUG' \ ---header 'Content-Type: application/json' \ ---data '{ +curl https://api.portkey.ai/v1/rerank \ + -H "Content-Type: application/json" \ + -H "x-portkey-api-key: $PORTKEY_API_KEY" \ + -H "x-portkey-provider: @BEDROCK_PROVIDER_SLUG" \ + -d '{ "model": "arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0", "query": "What is the capital of the United States?", "top_n": 3, "documents": [ - "Carson City is the capital city of the American state of Nevada.", - "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", - "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", - "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", - "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." + "Carson City is the capital city of the American state of Nevada.", + "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", + "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", + "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", + "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." ] -}' + }' ``` ```python Python @@ -394,11 +529,11 @@ from portkey_ai import Portkey portkey = Portkey( api_key="PORTKEY_API_KEY", - provider="@BEDROCK_PROVIDER_SLUG" + provider="@BEDROCK_PROVIDER_SLUG", ) response = portkey.post( - '/rerank', + "/rerank", model="arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0", query="What is the capital of the United States?", top_n=3, @@ -407,43 +542,66 @@ response = portkey.post( "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", - "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." - ] + "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states.", + ], ) ``` -```javascript NodeJS -import Portkey from 'portkey-ai'; +```javascript JavaScript +import Portkey from "portkey-ai"; const portkey = new Portkey({ - apiKey: "PORTKEY_API_KEY", - provider: "@BEDROCK_PROVIDER_SLUG" + apiKey: "PORTKEY_API_KEY", + provider: "@BEDROCK_PROVIDER_SLUG", }); -const response = await portkey.post('/rerank', { - model: "arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0", - query: "What is the capital of the United States?", - top_n: 3, - documents: [ - "Carson City is the capital city of the American state of Nevada.", - "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", - "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", - "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", - "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." - ] +const response = await portkey.post("/rerank", { + model: "arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0", + query: "What is the capital of the United States?", + top_n: 3, + documents: [ + "Carson City is the capital city of the American state of Nevada.", + "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", + "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.", + "Capitalization or capitalisation in English grammar is the use of a capital letter at the start of a word. English usage varies from capitalization in other languages.", + "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states.", + ], }); ``` + + +## Tokens, pricing, and analytics + +- **Token / usage fields** in the body come from the **provider**, same as calling them without Portkey. +- **Portkey billing and analytics** run on proxy traffic. Pricing accuracy depends on endpoint and model recognition; unusual paths may need **[cost / pricing configuration](/product/observability/cost-management)**—validate in the dashboard or contact support if numbers look off. +- Proxy requests appear in **logs** like any other gateway call. + +## Limitations and gotchas + +- Response bodies are **pass-through** from the provider—no Portkey JSON reshape on proxy routes. +- **`PATCH`** is not supported on the proxy catch-all. +- **Streaming, multipart, and binary** use different internal paths—validate with the exact content type and provider before production. +- **End-to-end success** still requires gateway support for that provider (auth, signing, allowed paths) and a real provider API behind the path. +- SDKs typed for one provider’s contract can break when the **upstream** changes—same as calling that provider directly. + +## Checklist + +- Prefer a **[unified route](/product/ai-gateway/universal-api)** when one exists. +- Send **`x-portkey-api-key`**. +- Send **`x-portkey-provider`** or **`x-portkey-config`**. +- Keep provider secrets in **`Authorization`** or Model Catalog—not in `x-portkey-*` headers as fake “secrets.” +- Match the **provider’s** request body. +- For passthrough on a unified path, use **`/v1/proxy/...`**. -# Caveats & Considerations +## Related -- Response objects are returned exactly as received from the provider, without Portkey transformations -- REST API supports all HTTP methods: `POST`, `GET`, `PUT`, and `DELETE` -- SDK supports all HTTP methods: `POST`, `GET`, `PUT`, and `DELETE` -- There are no limitations on which provider endpoints can be proxied -- All requests are logged and monitored through your Portkey dashboard +- [Universal API](/product/ai-gateway/universal-api) +- [Configs](/product/ai-gateway/configs) +- [Custom hosts](/product/ai-gateway/custom-hosts) +- [Inference API headers](/api-reference/inference-api/headers) -# Support +## Support -Need help? Join our [Developer Forum](https://portkey.wiki/community) for support and discussions. +Join the [Developer Forum](https://portkey.wiki/community) for help and discussions. diff --git a/product/ai-gateway.mdx b/product/ai-gateway.mdx index c213b449..dcbe8c6a 100644 --- a/product/ai-gateway.mdx +++ b/product/ai-gateway.mdx @@ -67,6 +67,10 @@ description: The world's fastest AI Gateway with advanced routing & integrated G Route requests to privately hosted or local models using custom host URLs + + + Call rerank, video, listen, and other provider paths through https://api.portkey.ai/v1/... with the same configs and logging as unified routes +