Manage evaluators

View and manage evaluators at the workspace level in LangSmith.

Evaluators in LangSmith are workspace-level resources. You can attach a single evaluator to multiple tracing projects and datasets, so you can apply consistent evaluation logic across your work without recreating it each time.

The LangSmith Engine suggests custom evaluators for detected issues and can deploy them with one click.

View evaluators

In the LangSmith UI, select Evaluators in the left sidebar to view all evaluators in your workspace.

The evaluators table shows the following columns:

Column	Description
Name	The evaluator name
Type	LLM as a judge or Code. Composite score evaluators are scoped to individual tracing projects and datasets and do not appear here.
Feedback Key	The feedback key the evaluator produces
Projects & Datasets	Tracing projects and datasets this evaluator is attached to
Evaluator Trace Count (this week)	Number of traces this evaluator ran on in the past week. Only shown when spend tracking is enabled; – for Code evaluators or evaluators with no attached rules.
Spend (this week)	Estimated USD spend for this evaluator in the past week. Only shown when spend tracking is enabled; – for Code evaluators or evaluators with no attached rules.
Spend Status	Whether the evaluator is Under limits, Unlimited, or has hit one or more configured spend limits. Only shown when spend tracking is enabled; – for Code evaluators.
Created By	The workspace member who created the evaluator
Updated At	When the evaluator was last modified
Created At	When the evaluator was created

Create an evaluator

You can create an evaluator in the LangSmith UI or programmatically with the SDK. Evaluators created either way are workspace-level resources that appear in the Evaluators table.

Create an evaluator in the UI

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Click + Evaluator to open the new evaluator panel.
3. The panel lets you:

   • Create from scratch: Build a new LLM-as-a-Judge or Code evaluator.

   • Create from a template: Start from a ready-made evaluator (also known as a prebuilt evaluator) for common evaluation patterns. A Recommended section surfaces popular templates first, followed by templates organized by the following categories:

     Category	Description
     Security	Detect leaks, injections, and adversarial inputs.
     Safety	Evaluate content safety and moderation.
     Quality	Measure output quality and accuracy.
     Conversation	Evaluate conversational quality and user experience.
     Trajectory	Evaluate agent tool use and decision paths.
     Image Evaluations	Evaluate image content quality and safety.
     Voice Evaluation	Evaluate voice and audio interaction quality.

You can also add an evaluator directly from a tracing project or dataset. In that flow, you can additionally attach an existing evaluator from your workspace, or create a Composite evaluator. Refer to Set up LLM-as-a-judge online evaluators and Automatically run evaluators on experiments.

Create an evaluator with the SDK

Use the LangSmith SDK to create evaluators programmatically. The SDK is available for Python and TypeScript. Evaluators created through the SDK appear in the Evaluators table alongside those created in the UI.

Managing evaluators through the SDK requires langsmith>=0.9.8 (Python, PyPI) or langsmith>=0.7.16 (TypeScript, npm).

Python

import asyncio

from langsmith import Client


async def main():
    client = Client()

    created = await client.evaluators.create(
        name="Correctness evaluator",
        type="code",
        code_evaluator={
            "code": "def perform_eval(run, example):\n    return {'score': 1}",
            "language": "python",
        },
    )
    print("Created evaluator:", created.evaluator.id)


asyncio.run(main())

import { Client } from "langsmith";

const client = new Client();

const created = await client.evaluators.create({
  name: "Correctness evaluator",
  type: "code",
  code_evaluator: {
    code: "def perform_eval(run, example):\n    return {'score': 1}",
    language: "python",
  },
});
console.log("Created evaluator:", created.evaluator?.id);

To create an LLM-as-a-judge evaluator and to retrieve, update, list, or delete evaluators, refer to Manage evaluators with the SDK.

View evaluator details

Click any evaluator in the table to open its detail view. The detail view has four tabs:

• Overview: The evaluator’s feedback configuration and prompt or code definition.
• Traces: Traces processed by this evaluator across all attached resources.
• Logs: Execution logs for this evaluator across all attached resources.
• Projects & Datasets: The tracing projects and datasets this evaluator is attached to, with each attachment’s weekly spend and limit.

Edit an evaluator

Open an evaluator. In the Overview tab, click the Edit evaluator icon to open the Configure Evaluator panel. Update the evaluator’s configuration. Click Save.

Because the evaluator is shared, changes apply across all tracing projects and datasets it is attached to.

Manage evaluator trace retention

When an online evaluator scores a trace, it attaches feedback to that trace, which auto-upgrades the trace to extended retention. Extended retention keeps the trace longer but costs more. When you set up an online evaluator on a tracing project, you can opt out of this upgrade so that scored traces stay at the project’s base retention.

This control is available only when the project’s default retention is the base tier. If the project defaults to extended retention (set at the project or workspace level), traces scored by the evaluator follow that default and the option is locked.

To opt out of extending retention for scored traces:

1. When you create or edit an online evaluator, set the source to a tracing project, rather than a dataset.
2. Expand the Advanced section in the evaluator configuration panel.
3. Clear Extend trace retention.

The change applies to traces scored after you save the evaluator. Existing scored traces keep their current retention tier.

Delete an evaluator

You cannot delete an evaluator while it is attached to a tracing project or dataset. To delete an evaluator:

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Select the evaluator you want to delete.
3. Open the Projects & Datasets tab. For each attached tracing project and dataset, select Detach in the Actions menu at the right of the row.
4. Return to the Evaluators page and click Delete at the top of the page.

---

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.