Connection check
verified live · 20h ago
evals
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Tools
54
GitHub stars
—
Installs / wk
—
Licence
—
Transport
streamable-http
Last checked
20h ago
Tools & capabilities
54 toolsRead from the running server on 20h ago.
agreements_create
noterun_id*verdict*metric_id*created_byresponse_id*
+1
Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'. Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.
agreements_list
run_idmetric_idcreated_byresponse_id
List agreements. Filter by run_id, response_id, metric_id, or created_by. List agreements. Filter by run_id, response_id, metric_id, or created_by.
datasets_create
name*csv_data*tag_names
Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and… Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.
datasets_create_from_url
url*name*tag_names
Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data n… Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.
datasets_delete
id*
Delete a dataset Delete a dataset
datasets_get
id*
Get a dataset by ID Get a dataset by ID
datasets_list
List all datasets List all datasets
datasets_update
id*namecsv_datatag_names
Update a dataset Update a dataset
judges_compare
metric_id*metric_version_a_id*metric_version_b_id*
Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unav… Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.
judges_replay
name*metric_id*dataset_id*judge_model*output_column
Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call… Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.
metric_groups_create
name*tag_namesmetric_idsdescription
Create a metric group Create a metric group
metric_groups_delete
id*
Delete a metric group Delete a metric group
metric_groups_get
id*
Get a metric group by ID Get a metric group by ID
metric_groups_list
List all metric groups List all metric groups
metric_groups_update
id*nametag_namesmetric_idsdescription
Update a metric group Update a metric group
metrics_create
name*tag_namesinstructionmetric_typecheck_configrubric_bands
Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern… Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
metrics_delete
id*
Delete a metric Delete a metric
metrics_get
id*
Get a metric by ID Get a metric by ID
metrics_list
List all metrics List all metrics
metrics_suggest_variants
countmodelmetric_id*
Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion".… Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.
metrics_update
id*nametag_namesinstructionmetric_typecheck_config
+1
Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expect… Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid_json takes no extra keys. target_path is required when target is json_path. For contains, not_contains, and equals, set compare_to:"expected" to grade against each row's own expected_output (ground truth) instead of a constant value (drop value); add expected_path to dig into the expected value when it is JSON.
metric_versions_dismiss
metric_version_id*
Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as cu… Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead.
metric_versions_list
metric_id*
List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps. List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.
metric_versions_publish
metric_version_id*
Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, d… Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.
promptfoo_import
config*
Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-… Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.
prompts_create
name*template*llm_model*tag_namesdescription
Create a prompt Create a prompt
prompts_delete
id*
Delete a prompt Delete a prompt
prompts_get
id*
Get a prompt by ID Get a prompt by ID
prompts_list
List all prompts List all prompts
prompts_publish
id*
Publish a prompt version, making it the current version Publish a prompt version, making it the current version
prompts_suggest_improvement
run_id*
Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewri… Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).
prompts_update
id*nametemplatellm_modeltag_namesdescription
Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so… Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.
provider_credentials_create
api_key*provider*api_versionapi_endpoint
Create a provider credential Create a provider credential
provider_credentials_delete
id*
Delete a provider credential Delete a provider credential
provider_credentials_get
id*
Get a provider credential by ID (API key is not exposed) Get a provider credential by ID (API key is not exposed)
provider_credentials_list
List all provider credentials (API keys are not exposed) List all provider credentials (API keys are not exposed)
provider_credentials_update
id*api_keyproviderapi_versionapi_endpoint
Update a provider credential Update a provider credential
responses_get
id*run_id*
Get a specific response Get a specific response
responses_list
sortlimitfieldsoffsetrun_id*status
+2
List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "… List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.
runs_create
name*prompt_idtag_namesdataset_idmax_tokensmetric_ids
+6
Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones. Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.
runs_delete
id*
Delete a run Delete a run
runs_generate
id*
Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and gr… Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.
runs_get
id*
Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. U… Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.
runs_list
List all runs List all runs
runs_regrade
id*
Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run. Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.
runs_rerun
id*
Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions. Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.
runs_retry_failures
id*only
Re-run only the failed responses of a run, optionally limited to specific response ids via "only". Re-run only the failed responses of a run, optionally limited to specific response ids via "only".
runs_update
id*nametag_namesdataset_idmax_tokensmetric_ids
+6
Update a run Update a run
tags_create
name*
Create a tag. Color is auto-assigned. Create a tag. Color is auto-assigned.
tags_delete
id*
Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset. Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.
tags_get
id*
Get a tag by ID Get a tag by ID
tags_list
List all tags List all tags
tags_update
id*name
Rename a tag. Rename a tag.
usage_get