Score agents on real outcomes

Attach scores to runs and steps, and defer scoring until the real-world signal arrives. Measure agents on what actually happened, not on proxy metrics.

Use deferred scoring to grade an agent on what actually happened: human feedback, a resolved ticket, or ground truth that only arrives after the run ends.

The most honest signal for an agent is whether the real-world outcome was good, and that usually isn't known when the run finishes.

A triage agent posts a root-cause analysis now; whether it pointed at the right file is only known once an engineer ships the fix.

A support agent answers now; whether it resolved the ticket is known hours later.

Because Inngest runs the agent, you can attach a score to that original run whenever the truth arrives; that can be inline if it's known during the run, or deferred against the run's ID if it lands later. The score lives next to the run, step, and trace that produced it, with no separate eval pipeline to ship traces to.

This pattern is useful when you want to grade against a resolved outcome, capture a human 👍/👎, compare an agent's claim to ground truth, or measure goal completion rather than a model-judged proxy.

§Which kind of score do you need?

A score is a number (commonly 0–1) attached to a run, a specific step, or a session. You decide what it measures.

§Score inline when the outcome is known

If the agent can be graded against a known answer at the time it runs, whether a fixed value range or a deterministic algorithm, score it inline with step.score(). The score is memoized like any other step.

01export default inngest.createFunction(
02  { id: "classify-ticket", triggers: { event: "support/ticket.created" } },
03  async ({ event, step }) => {
04    const predicted = await step.run("classify", () =>
05      classify(event.data.body)
06    );
07
08    if (event.data.knownLabel) {
09      await step.score({
10        name: "classification-accuracy",
11        value: predicted === event.data.knownLabel ? 1 : 0,
12      });
13    }
14
15    return predicted;
16  }
17);

§Defer the score until the real outcome lands

This is the common case for agents in production: the run finishes, and the outcome only arrives later. Define a scorer with createScorer() and kick it off from the agent run with defer(). The scorer runs as its own function: it waits for the outcome signal, then returns a { name, value } score that attaches to the run that deferred it. The agent run is never blocked, and there's no glue table mapping runs to outcomes.

In the incident-triage example, the agent posts a root-cause analysis and cites the files it believes are responsible. The score can't be computed yet (it depends on the files the eventual fix touches), so the scorer waits for incident/fix.shipped, then grades the cited files against what the fix actually changed.

01import { createScorer } from "inngest/experimental";
02import { z } from "zod";
03import { inngest } from "./client";
04
05// The scorer is its own function. It waits for the fix to ship,
06// then grades the agent's cited files against ground truth.
07export const triageScorer = createScorer(
08  inngest,
09  {
10    id: "triage-localization-scorer",
11    schema: z.object({
12      incidentId: z.string(),
13      cited: z.array(z.string()),
14    }),
15  },
16  async ({ event, step }) => {
17    const fix = await step.waitForEvent("wait-for-fix", {
18      event: "incident/fix.shipped",
19      timeout: "30d",
20      if: `async.data.incidentId == '${event.data.input.incidentId}'`,
21    });
22
23    // No fix shipped inside the window: record a zero rather than nothing.
24    if (!fix) {
25      return { name: "localization", value: 0 };
26    }
27
28    return {
29      name: "localization",
30      value: jaccard(event.data.input.cited, fix.data.fixFiles), // overlap, 0–1
31    };
32  }
33);

Trigger the scorer from the agent run with defer(). It's enqueued when the run finalizes, so the agent returns immediately and the grading happens in the background.

01export default inngest.createFunction(
02  { id: "triage-incident", triggers: { event: "incident/opened" } },
03  async ({ event, step, defer }) => {
04    const rca = await step.run("investigate", () => investigate(event.data));
05    await step.run("post-rca", () => postRca(event.data.incidentId, rca));
06
07    // Score this run once the fix ships, in the background.
08    defer("score-localization", {
09      function: triageScorer,
10      data: { incidentId: event.data.incidentId, cited: rca.citedFiles },
11    });
12
13    return rca;
14  }
15);

The returned score attaches to the triage run, so its trace, cost, and quality live in one place. Because the scorer is defined once, the same logic can grade any run that defers to it, including each variant of a group.experiment().

§Capture human feedback as a score

When the outcome already arrives as an event that carries the run's ID, you don't need a scorer; score the run directly from a normal function. A thumbs-up/down from your UI is the typical case: send it as an event and score the run it refers to.

01export const scoreFeedback = inngest.createFunction(
02  { id: "score-feedback", triggers: { event: "agent/response.rated" } },
03  async ({ event }) => {
04    await inngest.score({
05      name: "human-feedback",
06      runId: event.data.runId,
07      value: event.data.rating === "up" ? 1 : 0,
08    });
09  }
10);

§Score the session, not just the run

A conversation or an incident thread spans many runs. When the outcome reflects the whole flow rather than one step, score the session instead of a single run, using the same session key you group runs by.

01await inngest.score("goal-completed", {
02  session: { conversation_id: event.data.conversationId },
03  value: event.data.resolved ? 1 : 0,
04});

§Best practices

• Score the outcome you care about, not a proxy for it. Prefer "did the ticket resolve" over "did the answer look good."
• Use a stable score name. Names appear in the dashboard and in Insights, and renaming splits your history.
• Keep value in [0, 1]. Map pass/fail to 1/0 and graded results to a fraction.
• When you score directly from an outcome event, put the run's ID on that event so the handler knows which run to score. Deferred scorers don't need this: the score attaches to the run that deferred them.
• Reach for a deferred score before an LLM-as-judge score. A real outcome is a better signal than a model's opinion of the output.

§Related docs

• Score a function run: how-to for inline and external scoring
• Build a deferred scorer: score outcomes that resolve later
• Scoring reference: inngest.score(), step.score(), createScorer(), defer()
• Deferred functions reference: defer() for fire-and-forget background work
• Patterns: Run experiments in production: split traffic and compare variants
• group.experiment() reference: score each variant in an experiment