Manage evaluators

View and manage evaluators at the workspace level in LangSmith.

Evaluators in LangSmith are workspace-level resources. You can attach a single evaluator to multiple trace containers and datasets, so you can apply consistent evaluation logic across your work without recreating it each time.

What an evaluator attaches to depends on how your workspace is organized.

Agent-based workspaces (beta). This section applies if your workspace organizes traces by agent and environment. To check, look at the control at the top left: it names your workspace. If it shows the LangSmith logo instead, and the sidebar has an Application section with an application picker, your workspace is project-based. Skip this section and read Tracing projects.

An evaluator attaches to an environment of an agent. Attach it to Production alone to score production traffic without also scoring local runs.

Project-based workspaces. This section applies if your workspace organizes traces by tracing project. To check, look at the control at the top left. In a project-based workspace, it shows the LangSmith logo, and the sidebar has an Application section with an application picker. If the control shows your workspace name instead, your workspace is agent-based, which is in beta. Skip this section and read Agents.

An evaluator attaches to a tracing project.

Evaluator scores are a high-priority signal for the LangSmith Engine: it pulls low-scoring traces when choosing what to analyze, so attaching an evaluator to a project sharpens the issues Engine finds there.

View evaluators

In the LangSmith UI, select Evaluators in the left sidebar to view all evaluators in your workspace.

The evaluators table shows the following columns:

Column	Description
Name	The evaluator name
Type	LLM as a judge or Code. Composite score evaluators are scoped to individual tracing projects and datasets and do not appear here.
Feedback Key	The feedback key the evaluator produces
Projects & Datasets	Tracing projects and datasets this evaluator is attached to. In an agent-based workspace, this column is Environments & Datasets and lists agent environments and datasets.
Evaluator Trace Count (this week)	Number of traces this evaluator ran on in the past week. Only shown when spend tracking is enabled; – for Code evaluators or evaluators with no attached rules.
Spend (this week)	Estimated USD spend for this evaluator in the past week. Only shown when spend tracking is enabled; – for Code evaluators or evaluators with no attached rules.
Spend Status	Whether the evaluator is Under limits, Unlimited, or has hit one or more configured spend limits. Only shown when spend tracking is enabled; – for Code evaluators.
Created By	The workspace member who created the evaluator
Updated At	When the evaluator was last modified
Created At	When the evaluator was created

Create an evaluator

You can create an evaluator in the LangSmith UI or programmatically with the SDK. Evaluators created either way are workspace-level resources that appear in the Evaluators table.

Create an evaluator in the UI

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Click + Evaluator to open the new evaluator panel.
3. The panel lets you:

   • Create from scratch: Build a new LLM-as-a-Judge or Code evaluator. To use a decision model as the judge, select LLM-as-a-Judge Evaluator, then select a decision model.

   • Add a LangChain Tuned Evaluator: Attach a specialized judge managed by LangChain to a compatible tracing project without configuring a prompt, model, or API key.

   • Create from a template: Start from a ready-made evaluator (also known as a prebuilt evaluator) for common evaluation patterns. A Recommended section surfaces popular templates first, followed by templates organized by the following categories:

     Category	Description
     Security	Detect leaks, injections, and adversarial inputs.
     Safety	Evaluate content safety and moderation.
     Quality	Measure output quality and accuracy.
     Conversation	Evaluate conversational quality and user experience.
     Trajectory	Evaluate agent tool use and decision paths.
     Image Evaluations	Evaluate image content quality and safety.
     Voice Evaluation	Evaluate voice and audio interaction quality.

You can also add an evaluator directly from a tracing project, an agent environment, or a dataset. In that flow, you can additionally attach an existing evaluator from your workspace, or create a Composite evaluator. Refer to Set up LLM-as-a-judge online evaluators and Automatically run evaluators on experiments.

Create an evaluator with the SDK

Use the LangSmith SDK to create evaluators programmatically. The SDK is available for Python and TypeScript. Evaluators created through the SDK appear in the Evaluators table alongside those created in the UI.

The SDK does not support decision model evaluators yet. To create one, use the UI.

Managing evaluators through the SDK requires langsmith>=0.9.8 (Python, PyPI) or langsmith>=0.7.16 (TypeScript, npm).

Python

import asyncio

from langsmith import Client


async def main():
    client = Client()

    created = await client.evaluators.create(
        name="Correctness evaluator",
        type="code",
        code_evaluator={
            "code": "def perform_eval(run, example):\n    return {'score': 1}",
            "language": "python",
        },
    )
    print("Created evaluator:", created.evaluator.id)


asyncio.run(main())

To create an LLM-as-a-judge evaluator and to retrieve, update, list, or delete evaluators, refer to Manage evaluators with the SDK.

View evaluator details

Click any evaluator in the table to open its detail view. The detail view has four tabs:

• Overview: The evaluator’s feedback configuration and prompt or code definition.
• Traces: Traces processed by this evaluator across all attached resources.
• Logs: Execution logs for this evaluator across all attached resources.
• Projects & Datasets (Environments & Datasets in an agent-based workspace): The tracing projects, agent environments, and datasets this evaluator is attached to, with each attachment’s weekly spend and limit.

Edit an evaluator

Open an evaluator. In the Overview tab, click the Edit evaluator icon to open the Configure Evaluator panel. Update the evaluator’s configuration. Click Save.

Because the evaluator is shared, changes apply across every trace container and dataset it is attached to.

Manage evaluator trace retention

When an online evaluator scores a trace, it attaches feedback to the trace. This can auto-upgrade the trace to extended retention, depending on the evaluator’s retention setting. Extended retention keeps the trace longer but costs more. When you set up an online evaluator on a tracing project, or on an agent environment, you can opt out of this upgrade so that scored traces stay at the base retention.

This control is available only when the project’s default retention is the base tier. If the project defaults to extended retention (set at the project or workspace level), traces scored by the evaluator follow that default and the option is locked.

To opt out of extending retention for scored traces:

1. When you create or edit an online evaluator, set the source to a tracing project or an agent environment, rather than a dataset.
2. Expand the Advanced section in the evaluator configuration panel.
3. Clear Extend trace retention.

The change applies to traces scored after you save the evaluator. Existing scored traces keep their current retention tier.

The Extend trace retention toggle described above applies to both trace-level and thread-level (multi-turn) online evaluators. For more information on multi-turn evaluators, see Set up multi-turn online evaluators.

Include extended stats

Use Include extended stats (feedback, costs, tokens) in a run-level evaluator to evaluate feedback statistics, token usage, or cost data from the run. The feedback_stats field contains feedback statistics, including the number and average for each feedback key. This option is not available for multi-turn (thread-level) evaluators.

LangSmith fetches additional data for evaluators with this option enabled. Enable it only when your evaluation logic or prompt requires these fields.

To include extended stats:

1. When you create or edit a run-level evaluator, expand the Advanced section in the evaluator configuration panel.
2. Select Include extended stats (feedback, costs, tokens).

Access extended stats

Code evaluators receive these fields in the run object, including feedback_stats, total_tokens, and total_cost. They also receive individual feedback records in run["feedback"], regardless of whether you enable extended stats. For LLM-as-a-judge evaluators, map the corresponding run.* field to a prompt variable. For example, map {{correctness_average}} to run.feedback_stats.correctness.avg to include a correctness feedback average in the prompt. The available run.* fields also include prompt and completion token, cost, and detail fields. For the full run data schema, see Run data format.

Chain evaluators

A chained evaluator is an online evaluator that triggers only on runs another evaluator has already scored. Chaining lets you use a more specific evaluator or reserve a more expensive evaluator for the runs that need it. For example, you can run an LLM-as-a-judge evaluator only when an earlier correctness score is below a threshold.

The second evaluator must be a run-level evaluator because multi-turn (thread-level) evaluators cannot filter on feedback. The first evaluator can be either type. A multi-turn evaluator attaches its feedback to the last run in the thread, so a second evaluator chained onto it evaluates that run, not the full thread.

To chain a second evaluator onto a first:

1. Create or open the second evaluator and expand its filter. For more information, see Apply a filter to runs that trigger the evaluator for LLM-as-a-judge evaluators or Create a filter for code evaluators.

2. Filter on the first evaluator’s feedback key. The filter matches on the key rather than the evaluator that produced it, so feedback from any source with that key triggers the second evaluator, including user feedback. Use a key that only the first evaluator produces.

   • To trigger on every run the first evaluator scored, filter on the key alone:

     eq(feedback_key, "correctness")
     

   • To trigger only when the score meets a condition, combine the key with a feedback_score comparison. For example, to trigger only when correctness is below 0.5:

     and(eq(feedback_key, "correctness"), lt(feedback_score, 0.5))
     

3. If the second evaluator reads the first score, include extended stats. LLM-as-a-judge evaluators require it, and code evaluators use it to read the score reliably. For more information, see Read the first score.

4. Save the evaluator.

The second evaluator picks up a run only after the first evaluator’s score is attached to it, so the order in which the evaluators run does not matter. For the full filter syntax, see Trace query syntax.

Read the first score

How the second evaluator reads the first evaluator’s score depends on its type:

• LLM-as-a-judge evaluators: Include extended stats, then map a prompt variable to run.feedback_stats.<feedback_key>.avg. For example, map {{correctness_average}} to run.feedback_stats.correctness.avg.
• Code evaluators: Read the feedback record in run["feedback"] whose key is the first evaluator’s feedback key, and use its score. run["feedback"] holds at most 100 feedback records across each batch of runs the evaluator processes, and the oldest records are dropped first, so the score can be missing when the batch carries many feedback records. To read the score reliably, include extended stats and read the average from run["feedback_stats"]["<feedback_key>"]["avg"].

The avg field averages every feedback record with that key on the run, so it equals the first evaluator’s score only when the run has one record with that key. Another record with the same key, such as user feedback, changes the average.

Delete an evaluator

You cannot delete an evaluator while it is attached to a trace container or dataset. To delete an evaluator:

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Select the evaluator you want to delete.
3. Open the Projects & Datasets tab (Environments & Datasets in an agent-based workspace). For each attachment listed there, select Detach in the Actions menu at the right of the row.
4. Return to the Evaluators page and click Delete at the top of the page.

---

Connect these docs to your agent of choice via MCP for real-time answers.

Edit this page on GitHub or file an issue.