CI regression gates

Availability

Open core · Cloud requires the Pro plan or higher

A regression gate turns an evaluation run into a pass/fail check. When a run's scores drop below the recorded baseline, the gate fails and your CI build exits non-zero — stopping a regression before it merges.

Gate on a run

Pass gate=True (Python) / gate: true (TypeScript) when running an evaluation. The run is scored against the dataset's baseline; if it regresses, the call signals failure and the process exits non-zero, which fails the CI step.

Python

from splyntra import eval as ev

dataset_id = ev.push_dataset("refund-policy-qa", items=[...])

results = [{"input": q, "actual": my_agent(q)} for q in questions]

# Fails (exits non-zero) if this run regresses vs. the baseline.
ev.run(dataset_id, results=results, scorers=["exact_match"], gate=True)

TypeScript

import { pushDataset, runEval } from "@splyntra/sdk";

const datasetId = await pushDataset("refund-policy-qa", items);

const results = questions.map((q) => ({ input: q, actual: myAgent(q) }));

// res.passed is false if this run regresses vs. the baseline.
const res = await runEval(datasetId, results, { gate: true });
if (!res.passed) process.exit(1);

Set the baseline

The baseline is the reference run future runs are compared against. Establish it once from a known-good revision by setting set_baseline=True on the run; subsequent gated runs compare against it.

Python

# Record this run as the new baseline for the dataset.
ev.run(dataset_id, results=results, scorers=["exact_match"], set_baseline=True)

TypeScript

// Record this run as the new baseline for the dataset.
await runEval(datasetId, results, { setBaseline: true });

Gate from the CLI

The splyntra CLI runs the same flow without writing SDK glue, which is convenient in CI:

# Push (or update) the dataset from a file.
splyntra eval push refund-policy-qa --file datasets/refund-qa.jsonl

# Run and gate against the baseline — exits non-zero on regression.
splyntra eval run refund-policy-qa --results results.jsonl --gate

See the CLI reference.

GitHub Actions

Because a failed gate exits non-zero, wiring it into CI is a single step:

- name: Evaluation regression gate
  env:
    SPLYNTRA_API_KEY: ${{ secrets.SPLYNTRA_API_KEY }}
    SPLYNTRA_ENDPOINT: https://ingest.splyntra.com
  run: |
    splyntra eval run refund-policy-qa --results results.jsonl --gate

If the run regresses against the baseline the step fails, blocking the merge.

Next steps

• Evaluation — datasets, runs, baselines, and the leaderboard.
• Scorers — choose what the gate measures.