Validate Rule

Validate a rule by executing it with test data without creating a saved rule.

This endpoint allows testing LLM-as-judge evaluators before saving them. It accepts a rule configuration (same as rule creation) and test data, executes the evaluator, and returns the evaluation results in the same format as batch_invoke_evaluator.

Only LLM-as-judge rules (evaluators) are supported. Code evaluators are not allowed.

The evaluator execution traces are written to the database (in the “evaluators” project), which allows users to see the evaluator execution history.

POST /api/v1/runs/rules/validate

Error

A valid request URL is required to generate request examples

200

[
  {}
]

{
  "detail": [
    {
      "loc": [
        "<string>"
      ],
      "msg": "<string>",
      "type": "<string>"
    }
  ]
}

Authorizations

API KeyTenant IDBearer Auth

X-API-Key · string · header · required

Body

application/json

Schema for validating rules without creating them.

Extends RunRulesCreateSchema with test data fields for validation. Only LLM-as-judge rules (evaluators) are supported, not code_evaluators.

For trace-level evaluators, provide test_inputs / test_outputs. For thread evaluators (group_by="thread_id"), provide test_thread_id + session_id instead; the backend fetches and assembles all turns automatically.

display_name · string · required

sampling_rate · number · required

session_id · string<uuid> | null

is_enabled · boolean · default: true

dataset_id · string<uuid> | null

filter · string | null

trace_filter · string | null

tree_filter · string | null

backfill_from · string<date-time> | null

use_corrections_dataset · boolean · default: false

num_few_shot_examples · integer | null

extend_only · boolean · default: false

is_tracing_disabled · boolean · default: false

extend_evaluator_trace_retention · boolean · default: false

extend_dataset_trace_retention · boolean | null

extend_annotation_queue_trace_retention · boolean | null

extend_webhook_trace_retention · boolean | null

transient · boolean · default: false

add_to_annotation_queue_id · string<uuid> | null

add_to_dataset_id · string<uuid> | null

add_to_dataset_prefer_correction · boolean · default: false

evaluators · EvaluatorTopLevel · object[] | null

Show child attributes

code_evaluators · CodeEvaluatorTopLevel · object[] | null

Show child attributes

evaluator_id · string<uuid> | null

alerts · RunRulesPagerdutyAlertSchema · object[] | null

Show child attributes

webhooks · RunRulesWebhookSchema · object[] | null

Show child attributes

evaluator_version · integer | null

create_alignment_queue · boolean · default: false

include_extended_stats · boolean · default: false

group_by · string | null

Allowed value: "thread_id"

spend_limit · RunRuleSpendLimitSchema · object | null

Show child attributes

tracer_session_issue_id · string<uuid> | null

test_inputs · Test Inputs · object | null

test_outputs · Test Outputs · object | null

test_reference_outputs · Test Reference Outputs · object | null

test_attachments · Test Attachments · object | null

test_thread_id · string | null

Response

200

application/json

Successful Response