Operator - Evals API Reference

POST

/api/v1/operator/failure_groups/:id/generate_eval_draft Web Session

Generate an incident-based eval draft

Description

Creates a draft incident-eval suggestion from a failure group so operators can refine and promote it into a suite.

Auth

Web Session

Request Body

Field	Type	Required	Description
incident_eval_suggestion	object	No	Incident eval suggestion attributes
incident_eval_suggestion.proposed_name	string	No	Optional display name override
incident_eval_suggestion.target_eval_suite_id	string	No	Optional destination eval suite UUID
incident_eval_suggestion.proposed_description	string	No	Optional draft description override

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Failure group UUID

Responses

201

Incident eval draft created

Returns: IncidentEvalSuggestion

404

Failure group not found

GET

/api/v1/operator/eval_suites Web Session

List eval suites

Description

Returns the full unpaginated eval suite list for the current workspace. Only nested filter[status] is validated (active | archived). limit, offset, eval_suite_id, and run_kind are not list parameters on this endpoint.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
filter[status]	query	string	No	Filter by suite lifecycle status

Responses

200

Full unpaginated list of eval suites

Returns: EvalSuiteList

GET

/api/v1/operator/eval_suites/:id Web Session

Get eval suite details

Description

Returns details of a specific eval suite.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval suite UUID

Responses

200

Eval suite details

Returns: EvalSuite

404

Eval suite not found

PATCH

/api/v1/operator/eval_suites/:id Web Session

Update eval suite

Description

Updates an eval suite's configuration.

Auth

Web Session

Request Body

Field	Type	Required	Description
eval_suite	object	Yes	Eval suite attributes. The controller pattern-matches {"eval_suite" => attrs}; a flat body does not match a clause.
eval_suite.name	string	No	Suite name
eval_suite.status	string (active, archived)	No	Suite lifecycle status
eval_suite.description	string	No	Suite description
eval_suite.canary_enabled	boolean	No	Enables canary regression-based release gate checks
eval_suite.canary_max_regression	string	No	Maximum allowed pass-rate regression in percentage points
eval_suite.pass_threshold	string	No	Passing threshold percentage (e.g. 85.0)
eval_suite.schedule_enabled	boolean	No	Enables periodic automated scheduling
eval_suite.schedule_interval_minutes	integer	No	Scheduling interval in minutes
eval_suite.baseline_version_id	string	No	Baseline workflow version UUID
eval_suite.is_release_gate	boolean	No	Whether this suite gates release readiness

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval suite UUID

Responses

200

Eval suite updated

Returns: EvalSuite

404

Eval suite not found

DELETE

/api/v1/operator/eval_suites/:id Web Session

Delete eval suite

Description

Deletes an eval suite and cascades to its eval cases, eval runs, and related review history.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval suite UUID

Responses

204

Eval suite deleted

404

Eval suite not found

POST

/api/v1/operator/eval_suites/:id/clone Web Session

Clone eval suite

Description

Creates a copy of an eval suite from suite fields only. clone/3 does not copy eval_cases. Schedule and canary flags are reset. The body must be wrapped as {"eval_suite": {"name": "..."}}; a flat body does not match a clause.

Auth

Web Session

Request Body

Field	Type	Required	Description
eval_suite	object	Yes	Override attributes for the cloned suite
eval_suite.name	string	No	Name for the cloned suite (default: Copy of <original>)

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval suite UUID to clone

Responses

201

Eval suite cloned

Returns: EvalSuite

404

Eval suite not found

GET

/api/v1/operator/eval_runs Web Session

List eval runs (operator)

Description

Returns a paginated eval-run list for the current workspace. Nested filter[status|eval_suite_id|workflow_target_id|run_kind|hiring_id|provider|provider_run_id|engagement_id|eval_run_id] plus limit (0-100, default 50) and offset (0-10000) are applied. Correlation filters remain workspace-scoped. The JSON envelope is {eval_runs, pagination}.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
filter[status]	query	string	No	Filter by run status
filter[eval_suite_id]	query	string	No	Eval suite UUID
filter[workflow_target_id]	query	string	No	Workflow target UUID
filter[run_kind]	query	string	No	Filter by run kind
filter[provider]	query	string	No	Canonical provider name in the correlation envelope
filter[hiring_id]	query	string	No	Agrenting hiring identifier
filter[provider_run_id]	query	string	No	AgentFarm provider run identifier
filter[engagement_id]	query	string	No	AgentOps engagement UUID
filter[eval_run_id]	query	string	No	Correlated AgentOps eval UUID
limit	query	integer	No	Page size from 0 to 100 (default 50)
offset	query	integer	No	Zero-based offset up to 10000 (default 0)

Responses

200

Paginated list of eval runs

Returns: EvalRunList

Example

{
  "eval_runs": [
    {
      "correlation": {
        "contract_version": "correlation/v1",
        "eval_run_id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
        "provider": "agentfarm",
        "provider_run_id": "provider-run-example"
      },
      "eval_suite_id": "9e1f1538-ff37-48f0-a2f9-65066d7d6699",
      "id": "d0b91d0c-0f4b-4a28-8d9d-ccf0c427d6e8",
      "inserted_at": "2026-03-20T14:00:00Z",
      "score": 0.85,
      "status": "completed",
      "updated_at": "2026-03-20T14:05:00Z",
      "workspace_id": "4ce4d170-e6d5-4118-8f4d-fac69dc099ea"
    }
  ],
  "pagination": {
    "limit": 50,
    "offset": 0
  }
}

GET

/api/v1/operator/eval_runs/:id Web Session

Get eval run (operator)

Description

Returns detailed information for a single eval run.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval run UUID

Responses

200

Eval run details

Returns: EvalRun

404

Eval run not found

POST

/api/v1/operator/eval_runs/:id/retry Web Session

Retry eval run

Description

Retries a failed eval run.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval run UUID

Responses

201

New eval run created

Returns: EvalRun

404

Eval run not found

POST

/api/v1/operator/eval_runs/:id/cancel Web Session

Cancel eval run

Description

Cancels a queued eval run. Only status == queued is updated; any other existing run returns 400 cannot_cancel.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval run UUID

Responses

200

Eval run cancelled

Returns: EvalRun

400

cannot_cancel — the run is not queued

404

Eval run not found

GET

/api/v1/operator/eval_reviews Web Session

List eval review tasks

Description

Returns pending or completed review tasks created from failed, errored, or human-review-required eval results.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
status	query	string	No	Filter by review status
eval_run_id	query	string	No	Filter by eval run UUID

Responses

200

List of eval review tasks

Returns: EvalReviewTaskList

POST

/api/v1/operator/eval_reviews/:id/resolve Web Session

Resolve an eval review task

Description

Approves an automated result or revises the final score and pass state for an eval case result.

Auth

Web Session

Request Body

Field	Type	Required	Description
decision	string (approved, revised)	No	Review decision
decision_notes	string	No	Operator notes for the review decision
final_passed	boolean	No	Final pass/fail when revising
final_score	string	No	Final score when revising

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Eval review task UUID

Responses

200

Eval review task resolved

Returns: EvalReviewTask

404

Eval review task not found

GET

/api/v1/operator/incident_eval_suggestions Web Session

List incident-generated eval drafts

Description

Returns incident-eval suggestions generated from failure groups, including drafts awaiting review or promotion.

Auth

Web Session

Parameters

Name	In	Type	Required	Description
status	query	string	No	Filter by suggestion status
failure_group_id	query	string	No	Filter by failure group UUID
workflow_target_id	query	string	No	Filter by workflow target UUID

Responses

200

List of incident eval suggestions

Returns: IncidentEvalSuggestionList

PATCH

/api/v1/operator/incident_eval_suggestions/:id Web Session

Update an incident eval suggestion

Description

Edits a draft incident suggestion or marks it approved or rejected before promotion.

Auth

Web Session

Request Body

Field	Type	Required	Description
status	string (draft, approved, rejected)	No	Suggestion status
proposed_name	string	No	Draft eval case name
target_eval_suite_id	string	No	Destination eval suite UUID
proposed_description	string	No	Draft eval case description
proposed_expected_assertions	object	No	Suggested assertions
proposed_input_payload	object	No	Suggested input payload
proposed_policy_expectations	object	No	Suggested policy expectations
rationale	string	No	Why this incident should become a test

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Incident eval suggestion UUID

Responses

200

Incident eval suggestion updated

Returns: IncidentEvalSuggestion

400

Invalid status transition or target eval suite

404

Incident eval suggestion not found

409

Incident eval suggestion is already finalized

POST

/api/v1/operator/incident_eval_suggestions/:id/promote Web Session

Promote an incident suggestion into an eval case

Description

Converts an approved incident suggestion into a real eval case in the selected suite.

Auth

Web Session

Request Body

Field	Type	Required	Description
target_eval_suite_id	string	No	Destination eval suite UUID when not already set on the suggestion

Parameters

Name	In	Type	Required	Description
id	path	string	Yes	Incident eval suggestion UUID

Responses

201

Incident suggestion promoted

Returns: IncidentEvalSuggestion

400

Invalid target eval suite

404

Incident eval suggestion not found

409

Incident eval suggestion has already been promoted or rejected

Operator - Evals

Endpoints

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Example

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth

Description

Auth