# Deferred scoring

> Score outcomes that arrive after a run finishes with scorer functions that wait for the real signal.

Often you don't know how well a run went until later. A customer clicks "helpful" tomorrow. A ticket reopens next week. A fix ships that shows whether the agent pointed at the right files. A **deferred scorer** handles this: it runs as its own function, waits for the signal, and scores the run that started it. The original run finishes without waiting.

Deferred scoring is a beta API. `createScorer` comes from `inngest/experimental` and may change before general availability. In Python, define the scorer with `create_defer()` from `inngest.experimental` and write its score with `inngest_client.score()` or `inngest_client.score_experiment()`.

## Create a scorer

Define a scorer with `createScorer()`. The optional `schema` validates the data the scorer receives. The handler can use any step tool, such as `step.waitForEvent()` or `step.run()`.

```typescript {{ title: "TypeScript" }}
import { createScorer } from "inngest/experimental";
import { z } from "zod";
import { inngest } from "./client";

export const feedbackScorer = createScorer(
  inngest,
  {
    id: "feedback-scorer",
    schema: z.object({ ticketId: z.string() }),
  },
  async ({ event, step }) => {
    const feedback = await step.waitForEvent("wait-for-feedback", {
      event: "support/feedback.received",
      timeout: "7d",
      if: `async.data.ticketId == '${event.data.ticketId}'`,
    });

    if (!feedback) return null;
    return { name: "customer-helpful", value: feedback.data.helpful };
  }
);
```

```python {{ title: "Python" }}
# Requires the inngest release after 0.5.19.
import datetime

import inngest
from inngest.experimental import create_defer

from .client import inngest_client

@create_defer(inngest_client, fn_id="feedback-scorer")
async def feedback_scorer(ctx: inngest.Context) -> None:
    ticket_id = str(ctx.event.data["ticketId"])

    feedback = await ctx.step.wait_for_event(
        "wait-for-feedback",
        event="support/feedback.received",
        timeout=datetime.timedelta(days=7),
        if_exp=f"async.data.ticketId == '{ticket_id}'",
    )
    if feedback is None:
        return  # Write nothing: the outcome is unknown.

    # Python scorers write the score themselves. The run that called
    # ctx.defer(), and its experiment variant, are on ctx.parents[0].
    parent = ctx.parents[0]
    helpful = feedback.data.get("helpful") is True

    async def write_score() -> None:
        if parent.experiment is not None:
            await inngest_client.score_experiment(
                name="customer-helpful",
                value=helpful,
                experiment=parent.experiment,
                run_id=parent.run_id,
            )
        else:
            await inngest_client.score(
                name="customer-helpful", value=helpful, run_id=parent.run_id
            )

    await ctx.step.run("score", write_score)
```

Return `{ name, value }` to write one score. Return `null` or `undefined` to write nothing. Register the scorer with your other functions:

```typescript {{ title: "TypeScript" }}
serve({ client: inngest, functions: [answerTicket, feedbackScorer] });
```

```python {{ title: "Python" }}
inngest.fast_api.serve(app, inngest_client, [answer_ticket, feedback_scorer])
```

## Start the scorer from a run

Call `defer()` from the function that produced the result. Pass the data the scorer needs; you don't pass the run ID.

```typescript {{ title: "TypeScript" }}
export const answerTicket = inngest.createFunction(
  { id: "answer-ticket", triggers: { event: "support/ticket.created" } },
  async ({ event, step, defer }) => {
    const answer = await step.run("write-answer", () =>
      writeAnswer(event.data.message)
    );

    defer("score-feedback", {
      function: feedbackScorer,
      data: { ticketId: event.data.ticketId },
    });

    return answer;
  }
);
```

```python {{ title: "Python" }}
# Requires the inngest release after 0.5.19.
import inngest

from .client import inngest_client
from .scorer import feedback_scorer
from .stubs import write_answer

@inngest_client.create_function(
    fn_id="answer-ticket",
    trigger=inngest.TriggerEvent(event="support/ticket.created"),
)
async def answer_ticket(ctx: inngest.Context) -> str:
    message = str(ctx.event.data["message"])

    async def write() -> str:
        return await write_answer(message)

    answer = await ctx.step.run("write-answer", write)

    ctx.defer(
        "score-feedback",
        function=feedback_scorer,
        data={"ticketId": ctx.event.data["ticketId"]},
    )

    return answer
```

`defer()` returns `void`; don't await a score from it. Each `defer()` ID must be unique within the run. When the scorer returns a score, Inngest attributes it to the run that called `defer()`.

If the result came from an [experiment](/docs-markdown/agent-evals/experiments), pass the returned `experimentRef` so the score counts toward the variant that served it:

```typescript {{ title: "TypeScript" }}
defer("score-feedback", {
  function: feedbackScorer,
  data: { ticketId: event.data.ticketId },
  experiment: experimentRef,
});
```

```python {{ title: "Python" }}
# Requires the inngest release after 0.5.19.
ctx.defer(
    "score-feedback",
    function=feedback_scorer,
    data={"ticketId": ctx.event.data["ticketId"]},
    experiment=experiment_ref,
)
```

## Send the signal

The scorer above waits for `support/feedback.received`. Your app sends it when the customer responds:

```typescript {{ title: "TypeScript" }}
await inngest.send({
  name: "support/feedback.received",
  data: { ticketId: "tk_123", helpful: true },
});
```

```python {{ title: "Python" }}
await inngest_client.send(
    inngest.Event(
        name="support/feedback.received",
        data={"ticketId": "tk_123", "helpful": True},
    )
)
```

Inngest resumes the scorer whose `if` expression matches the event's `ticketId`. The scorer returns the score, and Inngest writes it for the original run.

## How deferred scoring works

- The scorer is a separate function run. Inngest schedules it when the parent run finishes, not when the signal arrives.
- If the scorer waits with `step.waitForEvent()`, it pauses on its own. The parent run is never affected by the scorer's lifecycle, retries, or failures.
- A wait only catches events sent after the wait starts. If feedback can arrive very quickly, send it after the parent run finishes, or score the run directly with its `runId`.
- A scorer accepts normal function options such as `retries`, `concurrency`, and `throttle`. It can't use `onFailure` or event batching.
- A scorer run and its steps count as executions. The score write is one more step.

## Decide what a timeout means

If the signal never arrives, the scorer's timeout path decides the result. Returning `null` records no score, which means the outcome is unknown. Returning `0` records a negative outcome. Pick one deliberately and keep it the same for every variant you compare. Most comparisons should treat missing feedback as missing, not as failure. See [Read results and roll out](/docs-markdown/agent-evals/guides/interpreting-results#let-later-outcomes-catch-up).

## Write several scores

To write more than one score, or to control attribution yourself, write the scores from the scorer and return nothing. The parent run and its variant are on `parents[0]`:

```typescript {{ title: "TypeScript" }}
export const reviewScorer = createScorer(
  inngest,
  { id: "review-scorer", schema: z.object({ ticketId: z.string() }) },
  async ({ event, step, parents }) => {
    const { runId, experiment } = parents[0];
    const review = await step.waitForEvent("wait-for-review", {
      event: "support/review.completed",
      timeout: "14d",
      if: `async.data.ticketId == '${event.data.ticketId}'`,
    });
    if (!review) return null;

    await step.run("write-scores", async () => {
      for (const [name, value] of Object.entries(review.data.scores)) {
        if (experiment) {
          await inngest.score.experiment({ name, value, experiment, runId });
        } else {
          await inngest.score({ name, value, runId });
        }
      }
    });
    return null;
  }
);
```

```python {{ title: "Python" }}
# Requires the inngest release after 0.5.19.
import datetime

import inngest
from inngest.experimental import create_defer

from .client import inngest_client

@create_defer(inngest_client, fn_id="review-scorer")
async def review_scorer(ctx: inngest.Context) -> None:
    parent = ctx.parents[0]
    ticket_id = str(ctx.event.data["ticketId"])

    review = await ctx.step.wait_for_event(
        "wait-for-review",
        event="support/review.completed",
        timeout=datetime.timedelta(days=14),
        if_exp=f"async.data.ticketId == '{ticket_id}'",
    )
    if review is None:
        return

    scores = review.data.get("scores")

    async def write_scores() -> None:
        if not isinstance(scores, dict):
            return
        for name, value in scores.items():
            if not isinstance(value, (bool, int, float)):
                continue
            if parent.experiment is not None:
                await inngest_client.score_experiment(
                    name=name,
                    value=value,
                    experiment=parent.experiment,
                    run_id=parent.run_id,
                )
            else:
                await inngest_client.score(
                    name=name, value=value, run_id=parent.run_id
                )

    await ctx.step.run("write-scores", write_scores)
```

## Deferred or direct?

- **Score directly** with [`step.score()`](/docs-markdown/agent-evals/scores) when you know the outcome before the function finishes: a guardrail passed, JSON parsed, or a tool returned the expected format.
- **Score from another service** with [`inngest.score({ runId })`](/docs-markdown/agent-evals/scores#score-from-outside-the-run) when that service already receives the outcome and knows the run ID. This avoids a scorer run.
- **Defer a scorer** when the outcome needs its own work: waiting for an event, calling a model judge, or checking an external system.

## Next steps

- [Score user feedback](/docs-markdown/agent-evals/guides/user-feedback) shows both the deferred and direct approaches.
- [Experiments](/docs-markdown/agent-evals/experiments) credits deferred scores to variants.
- [Reference](/docs-markdown/agent-evals/reference#deferred-scoring) lists every option.