Evaluation

Availability

Open core · Cloud requires the Pro plan or higher

Evaluation lets you measure whether your agent produces the right answers, and whether a change made things better or worse. You push a dataset of examples, run your agent yourself to produce results, and hand those results to Splyntra, which scores them against the ground truth and records the run.

Splyntra scores results — it does not run your agent

An evaluation run scores caller-produced results against expected outputs. You execute your agent in your own code, collect its outputs, and submit them. Splyntra joins each result to the matching dataset item by its input and applies the scorers. It never invokes your agent for you.

Datasets

A dataset is a named set of examples. Each item has up to three fields:

Field	Purpose
input	The prompt or question given to the agent. Also the join key between a dataset item and a submitted result.
expected_output	The ground-truth answer the scorers compare against.
context	Optional supporting text (retrieved documents, source passages). Required by the groundedness and faithfulness scorers.

Pushing a dataset returns a dataset id you use to run against it. Push a new version whenever your ground truth changes.

Runs

A run scores a list of results against a dataset with one or more scorers. Each result carries the input (used to find its dataset item) and the actual output your agent produced. Splyntra joins result to item by input, applies every scorer, and stores per-item and aggregate scores.

Python

from splyntra import eval as ev

# 1. Push a dataset of ground-truth examples.
dataset_id = ev.push_dataset(
    "refund-policy-qa",
    items=[
        {"input": "How long do refunds take?", "expected_output": "5–7 business days."},
        {"input": "Can I refund a gift card?", "expected_output": "No, gift cards are non-refundable."},
    ],
)

# 2. Run YOUR agent to produce results, then submit them for scoring.
results = [
    {"input": "How long do refunds take?", "actual": my_agent("How long do refunds take?")},
    {"input": "Can I refund a gift card?", "actual": my_agent("Can I refund a gift card?")},
]

ev.run(dataset_id, results=results, scorers=["exact_match"])

TypeScript

import { pushDataset, runEval } from "@splyntra/sdk";

// 1. Push a dataset of ground-truth examples.
const datasetId = await pushDataset("refund-policy-qa", [
  { input: "How long do refunds take?", expected_output: "5–7 business days." },
  { input: "Can I refund a gift card?", expected_output: "No, gift cards are non-refundable." },
]);

// 2. Run YOUR agent to produce results, then submit them for scoring.
const results = [
  { input: "How long do refunds take?", actual: await myAgent("How long do refunds take?") },
  { input: "Can I refund a gift card?", actual: await myAgent("Can I refund a gift card?") },
];

const res = await runEval(datasetId, results, { gate: false });

Baselines and the leaderboard

• Baseline — mark a run as the reference point for a dataset. Later runs are compared against it to detect regressions. Setting a baseline is how you pin "known good" behavior. See CI regression gates.
• Leaderboard — the Evaluation screen ranks runs for a dataset by their scores, so you can see version-over-version movement at a glance and pick the best-performing revision.

Next steps

• Scorers — the built-in scorers and the commercial llm_as_judge.
• CI regression gates — fail a build when a run regresses against the baseline.
• CLI — the splyntra CLI for pushing datasets and running gated evals.