> ## Documentation Index
> Fetch the complete documentation index at: https://docs.langchain.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom model providers

> Route requests through the LLM Gateway to any OpenAI-compatible endpoint, such as a self-hosted open-source model served through an inference server.

<Note>
  **Private beta:** The LLM Gateway is in private [beta](/langsmith/release-stages). Sign up for [the waitlist](https://www.langchain.com/langsmith-llm-gateway-waitlist) to get access.
</Note>

In addition to the [built-in providers](/langsmith/llm-gateway#supported-providers), the LLM Gateway can proxy requests to **any OpenAI-compatible endpoint**, such as a self-hosted open-source model served through an inference server (vLLM, Ollama, and similar).

## How it works

A custom provider is defined by an **OpenAI Compatible Endpoint** [model configuration](/langsmith/model-configurations) that you save under **Settings → Model configurations** in the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-llm-gateway-custom-providers). The gateway uses the following options from the configuration:

* A **base URL**: the upstream endpoint the gateway forwards requests to.
* A **model name**: the model identifier your upstream expects.
* An **API key**: stored as a [workspace secret](/langsmith/llm-gateway-admin-setup#2-add-provider-secrets), never sent by the client.

You address the saved configuration by name through one of two routes, depending on whether you want callers to choose the model or want to enforce the configured one:

| Route                                                        | Model name in the request body                                                |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| `https://gateway.smith.langchain.com/providers/{configName}` | Forwarded to the upstream as-is—the client picks the model.                   |
| `https://gateway.smith.langchain.com/models/{configName}`    | Overridden with the configuration's model name—the client's value is ignored. |

Both routes look up the same configuration, resolve the same secret, and proxy to the same upstream URL; they only differ in whether the model name is enforced.

<Note>
  `{configName}` is the configuration name from your workspace [model configuration](/langsmith/model-configurations). If the name contains characters that aren't URL-safe (such as `/` or spaces), URL-encode them in the path. For example, a configuration named `meta-llama/Llama-3.1-8B-Instruct` becomes `https://gateway.smith.langchain.com/providers/meta-llama%2FLlama-3.1-8B-Instruct/v1/chat/completions`.
</Note>

## 1. Create a custom provider configuration

1. Add the upstream endpoint's API key as a workspace secret under **Settings → Integrations → Provider Secrets**. Give it a descriptive name (for example, `MY_PROVIDER_API_KEY`).
2. Go to **Settings → Model configurations** and create a configuration with **OpenAI Compatible Endpoint** as the provider.
3. Set the **Base URL** to your upstream endpoint (for example, `https://my-inference-server.example.com/v1`) and the **Model Name** to a model identifier the endpoint expects.
4. Set the **API Key Name** to the secret you created.
5. Save the configuration with a **name**. This name is what you'll use in the gateway route.

<Note>
  The **Model Name** you save only matters if you call the configuration through `/models/{configName}`. Through `/providers/{configName}`, the client's `model` field is sent through unchanged, so a single configuration can serve any model your upstream supports (for example, any model pulled into a shared Ollama instance).
</Note>

## 2. Make a call

Call the saved configuration by name (`my-custom-endpoint` in the following examples).

### Any model: `/providers/{configName}`

Use this route when the upstream serves multiple models and you want callers to pick which one:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
curl https://gateway.smith.langchain.com/providers/my-custom-endpoint/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"ping"}]}'
```

The gateway forwards the request body's `model` field to the upstream as-is.

### One model: `/models/{configName}`

Use this route to pin every call through this configuration to a single model, regardless of what the client requests—useful for enforcing model behavior for a team or application:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"catppuccin-mocha"}}
curl https://gateway.smith.langchain.com/models/my-custom-endpoint/v1/chat/completions \
    -H "Authorization: Bearer $LANGSMITH_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"ping"}]}'
```

The gateway overrides the request body's `model` field with the model name from the saved configuration, so any value the client passes (or omitting it, as per the example) is ignored. To serve multiple pinned models from the same upstream, create one configuration per model (each with its own name and `/models/{configName}` route).

## Supported endpoints

Both `/providers/{configName}` and `/models/{configName}` use the same allowlist as the built-in OpenAI provider: `POST /v1/chat/completions` (including streaming), `POST /v1/responses`, and the `GET /v1/models` listing endpoints. Any other path returns `501 Not Implemented`.

## Next steps

* [Spend policies](/langsmith/llm-gateway-spend-policies): apply cost limits to custom providers.
* [PII and secrets redaction](/langsmith/llm-gateway-redaction): redact sensitive data before it reaches your endpoint.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/llm-gateway-custom-providers.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>