LLM Gateway

Use the LLM Gateway to proxy LLM calls through LangSmith, enforce spend limits, redact sensitive data, and centrally manage provider credentials.

Private beta: The LLM Gateway is in private beta. APIs and features may change as we iterate. Sign up for the waitlist to get access.

The LLM Gateway is a proxy that sits between your agents (or any LLM client) and the LLM providers they call. Instead of each client storing provider API keys locally, keys are stored once in LangSmith as Provider Secrets. Clients authenticate with a LangSmith API key.

When a request passes through the gateway, it:

1. Authenticates the caller using their LangSmith API key.
2. Resolves the actual provider API key from your workspace’s Provider Secrets.
3. Evaluates governance policies (spend limits, PII redaction, secrets redaction).
4. Proxies the request to the upstream provider.
5. Traces the call to LangSmith.

Every gateway-proxied call appears as a trace in LangSmith. When a policy fires, the event flows into LangSmith Engine for triage—you can go from a blocked request to the trace that triggered it to the fix, all in one product.

Feature availability

Capability	Description
Spend limits	Hard-block enforcement at the organization, workspace, API key, or user level. When a cap is hit, the caller receives a 402 response with an actionable error message.
Spend visibility	Real-time cost rollups by workspace, user, API key. Per-model visibility can be had via Custom Charts.
PII redaction	Detects and redacts names, places, nationality, religion, political affiliation, and ages from requests before they reach the model.
Secrets redaction	Detects and redacts US phone numbers, US SSNs, API keys, tokens, and credentials from requests. Covers AWS, GitHub, GitLab, OpenAI, Anthropic, GCP, Azure, Slack, Datadog, PyPI, npm, private keys, and LangSmith tokens.
LangSmith Engine integration	Policy violations surface as issues in LangSmith Engine. Click through from a violation to the trace that produced it.
Audit logging	Administrative changes to gateway configuration and gateway invocations are logged.
Tracing	Every gateway-proxied call appears in the same LangSmith workspace as the rest of your agent’s traces. Routing through the gateway does not fragment your observability.

Supported providers

Provider	Gateway path	Secret name
Anthropic	/anthropic	ANTHROPIC_API_KEY
AWS Bedrock	/bedrock	AWS_BEARER_TOKEN_BEDROCK
Baseten	/baseten	BASETEN_API_KEY
Fireworks	/fireworks	FIREWORKS_API_KEY
Google Gemini	/gemini	GOOGLE_API_KEY
Google Vertex AI	/vertex	VERTEX_SERVICE_ACCOUNT_JSON
OpenAI	/openai	OPENAI_API_KEY

Known limitations

• Claude Desktop: no marketplace plugins supported. The Chat tab not visible, but Cowork is functional.
• Codex Desktop: no marketplace plugins supported.
• Cursor: IDE integration is not yet available.
• ChatGPT Desktop / Gemini Desktop: not configurable.

Resources

Spend policies

Set and manage cost limits across your organization.

PII and secrets redaction

Prevent sensitive data from reaching LLM providers or trace storage.

For further questions on LLM Gateway, contact support.langchain.com.

---

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.