Score agent accuracy

Grade an agent against a known answer (exact match, set overlap, or numeric tolerance) and record the score on the run that produced it.

Agents fail quietly. A misrouted ticket, a wrong extracted value, or a fix that touched the wrong files looks like a successful run: no exception is thrown, the function returns 200, and the trace is green. The only thing that went wrong is the answer.

When you have ground truth (a labeled category, an expected value, the set of files a fix actually touched), accuracy is the most direct score you can record. You compare what the agent produced against the known answer and attach a 0–1 score to the run.

Inngest records the score where the run already executed, so it sits next to the trace, inputs, and outputs that produced it. The known answer can be available at runtime, or it can arrive later as an outcome event. When it arrives later, you score the run by its ID instead of guessing at runtime.

Use this pattern when you have a labeled set, a golden answer, or a verifiable result the agent should match: classification, extraction, routing, or file localization.

§Which accuracy measure?

The first three are exact, cheap, and deterministic. Reach for them whenever the answer has a definite shape:

• Exact match fits a single category or label: score 1 when the agent's choice equals the known answer, else 0. This is the measure for classification and routing.
• Overlap (Jaccard) fits set-valued answers like cited files, extracted entities, or applied tags: score the fraction of the correct set the agent got, so partial credit tells you how wrong, not just that it was wrong. It's also the right shape for file localization, where you compare the files the agent proposed against the set the merged fix actually touched.
• Tolerance fits numeric estimates and forecasts: score 1 inside an acceptable band, then decay toward 0 as the error grows.

Free-text similarity is the fuzzy case; when there's no single correct string, grade it with a model instead (that's an LLM judge, not accuracy).

§A worked example

Define each measure once as a pure function, so the same check is reusable across functions and experiment variants:

01// measures.ts: define once, reuse everywhere
02export const exactMatch = (a: string, b: string): number => (a === b ? 1 : 0);
03
04// Jaccard: |intersection| / |union|, in [0, 1]
05export const jaccard = (a: string[], b: string[]): number => {
06  const A = new Set(a);
07  const B = new Set(b);
08  const shared = [...A].filter((x) => B.has(x)).length;
09  return shared / (A.size + B.size - shared);
10};
11
12// Full credit within `band`, decaying to 0 past it
13export const tolerance = (predicted: number, actual: number, band = 0.1) => {
14  const error = Math.abs(predicted - actual) / actual;
15  return error <= band ? 1 : Math.max(0, 1 - error);
16};

Then call the measure that matches your answer shape from inside the run, scoring only when ground truth is present so unlabeled production traffic doesn't drag the metric down:

01import { inngest } from "./client";
02import { exactMatch } from "./measures";
03
04export default inngest.createFunction(
05  { id: "route-ticket", triggers: { event: "support/ticket.created" } },
06  async ({ event, step }) => {
07    const team = await step.run("route", () => route(event.data));
08
09    if (event.data.knownTeam) {
10      await step.score("score-routing-accuracy", {
11        name: "routing-accuracy",
12        value: exactMatch(team, event.data.knownTeam),
13      });
14    }
15
16    return team;
17  }
18);

step.score() attaches the score to the current run. The first argument is a stable step id, which memoizes the score across retries and replays the same way step.run() IDs do; name is the series the score aggregates under in dashboards. Swap exactMatch for jaccard or tolerance when the answer is a set or a number.

When the correct answer only emerges after the run finishes (a human corrects the output, a downstream system confirms the result), move the check into a deferred scorer that waits for the outcome event and scores the original run by its ID.

§Best practices

• Only score accuracy when you actually have ground truth. If you're scoring against a model's opinion, that's an LLM judge, not accuracy. Keep the two as separate score names so the trends don't blur.
• Prefer partial-credit measures (overlap, tolerance) over exact match for anything set- or value-shaped. They tell you how wrong, not just that it was wrong.
• Keep the score name stable so accuracy stays in one trend series. A renamed score starts a new, disconnected line.
• Give each score step a stable id. Like step.run(), it memoizes across retries and replays so a score isn't counted twice.
• When the correct answer only emerges later, score the run by its runId instead of forcing a guess at runtime.

§Related docs

• Score a function run
• Build a deferred scorer for outcomes that arrive after the run
• inngest.score() reference
• Run experiments in production