Handling Failures

Define any failure handlers for your function with the onFailure option. This function will be automatically called when your function fails after it's maximum number of retries. Alternatively, you can use the "inngest/function.failed" system event to handle failures across all functions.

import { inngest } from "./client";

export default inngest.createFunction(
  {
    id: "import-product-images",
    onFailure: async ({ error, event, step }) => {
      // This is the failure handler which can be used to
      // send an alert, notification, or whatever you need to do
    },
  },
  { event: "shop/product.imported" },
  async ({ event, step, runId }) => {
    // This is the main function handler's code
  }
);

The failure handler is very useful for:

  • Sending alerts to your team
  • Sending metrics to a third party monitoring tool (e.g. Datadog)
  • Send a notification to your team or user that the job has failed
  • Perform a rollback of the transaction (i.e. undo work partially completed by the main handler)

Failures should not be confused with Errors which will be retried. Read the error handling & retries documentation for more context.


How onFailure works

The onFailure handler is a helper that actually creates a separate Inngest function used specifically for handling failures for your main function handler.

The separate Inngest function utilizes an "inngest/function.failed" system event that gets sent to your account any time a function fails. The function created with onFailure will appear as a separate function in your dashboard with the name format: "<Your function name> (failure)".

onFailure({ error, event, step, runId })

The onFailure handler function has the same arguments as the main function handler when creating a function, but also received an error argument.

error

The JavaScript Error object as thrown from the last retry in your main function handler.

The Inngest SDK attempts to serialize and deserialize the Error object to the best of its ability and any custom error classes (e.g. Prisma.PrismaClientKnownRequestError or MyCustomErrorType) that may be thrown will be deserialized as the default Error object. This means you cannot use instance of within onFailure to infer the type of error.

event

The "inngest/function.failed" system event payload object. This object is similar to any event payload, but it contains data specific to the failed function's final retry attempt. See the complete reference for this event payload below.

step

See the step reference in the create function documentation.

runId

This will be the function run ID for the error handling function, not the function that failed. To get the failed function's run ID, use event.data.run_id. Learn more about runId here.

The "inngest/function.failed" event

Inngest sends an "inngest/function.failed" event to your environment whenever any function fails after exhausting all of it's retry attempts. You can use this event as an event trigger within your own function or use the onFailure helper option as documented above.

  • Name
    name
    Type
    string: "inngest/function.failed"
    Description

    The inngest/ event prefix is reserved for system events in each environment.

  • Name
    data
    Type
    object
    Description

    The event payload data.

    Properties
    • Name
      error
      Type
      object
      Description

      Data about the error payload as returned from the failed function.

    • Name
      event
      Type
      string
      Description

      The failed function's original event payload.

    • Name
      function_id
      Type
      string
      Description

      The failed function's id

    • Name
      run_id
      Type
      string
      Description

      The failed function's function run ID.

  • Name
    ts
    Type
    number
    Description

    The timestamp integer in milliseconds at which the failure occurred.

Example "inngest/function.failed" event payload
{
  "name": "inngest/function.failed",
  "data": {
    "error": {
      "__serialized": true,
      "error": "invalid status code: 500",
      "message": "taylor@ok.com is already a list member. Use PUT to insert or update list members.",
      "name": "Error",
      "stack": "Error: taylor@ok.com is already a list member. Use PUT to insert or update list members.\n    at /var/task/.next/server/pages/api/inngest.js:2430:23\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async InngestFunction.runFn (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestFunction.js:378:32)\n    at async InngestCommHandler.runStep (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:459:25)\n    at async InngestCommHandler.handleAction (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/components/InngestCommHandler.js:359:33)\n    at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)\n    at async ServerTiming.wrap (/var/task/node_modules/.pnpm/inngest@2.6.0_typescript@5.1.6/node_modules/inngest/helpers/ServerTiming.js:69:21)"
    },
    "event": {
      "data": { "billingPlan": "pro" },
      "id": "01H0TPSHZTVFF6SFVTR6E25MTC",
      "name": "user.signup",
      "ts": 1684523501562,
      "user": { "external_id": "6463da8211cdbbcb191dd7da" }
    },
    "function_id": "my-gcp-cloud-functions-app-hello-inngest",
    "run_id": "01H0TPSJ576QY54R6JJ8MEX6JH"
  },
  "id": "01H0TPW7KB4KCR739TG2J3FTHT",
  "ts": 1684523589227
}

Examples

Send a Slack notification when a function fails

In this example, the function attempts to sync all products from a Shopify store, and if it fails, it sends a message to the team's #eng-alerts Slack channel using the Slack Web Api's chat.postMessage (docs) API.

import { client } from "@slack/web-api";
import { inngest } from "./client";

export default inngest.createFunction(
  {
    id: "sync-shopify-products",
    // Your handler should be an async function:
    onFailure: async ({ error, event }) => {
      const originalEvent = event.data.event;

      // Post a message to the Engineering team's alerts channel in Slack:
      const result = await client.chat.postMessage({
        token: process.env.SLACK_TOKEN,
        channel: "C12345",
        blocks: [
          {
            type: "section",
            text: {
              type: "mrkdwn",
              text: `Sync Shopify function failed for Store ${
                originalEvent.storeId
              }: ${error.toString()}`,
            },
          },
        ],
      });

      return result;
    },
  },
  { event: "shop/product_sync.requested" },
  async ({ event, step, runId }) => {
    // This is the main function handler's code
    const products = await step.run("fetch-products", async () => {
      const storeId = event.data.storeId;
      // The function might fail here or...
    });
    await step.run("save-products", async () => {
      // The function might fail here after the maximum number of retries
    });
  }
);

Track all function failures in Datadog

You can use the "inngest/function.failed" event to handle all failed functions in a single place. This can enable you to send metrics, alerts, or events to external systems like Datadog or Sentry for all of your Inngest functions. Here's an example using Datadog's Events API to send all failures the Datadog event stream.

import { client, v1 } from "@datadog/datadog-api-client";
import { inngest } from "./client";

const configuration = client.createConfiguration();
const apiInstance = new v1.EventsApi(configuration);

export default inngest.createFunction(
  {
    name: "Send failures to Datadog",
    id: "send-failed-function-events-to-datadog"
  },
  { event: "inngest/function.failed" },
  async ({ event, step }) => {
    // This is a normal Inngest function, so we can use steps as we normally do:
    await step.run("send-event-to-datadog", async () => {
      const error = event.data.error;

      // Create the Datadog event body using information about the failed function:
      const params: v1.EventsApiCreateEventRequest = {
        body: {
          title: "Inngest Function Failed",
          alert_type: "error",
          text: `The ${event.data.function_id} function failed with the error: ${error.message}`,
          tags: [
            // Add a tag with the Inngest function id:
            `inngest_function_id:${event.data.function_id}`,
          ],
        },
      };

      // Send to Datadog:
      const data = await apiInstance.createEvent(params);

      // Return the data to Inngest for viewing in function logs:
      return { message: "Event sent successfully", data };
    });
  }
);

Capture all failure errors with Sentry

Similar to the above example, you can capture and all failed functions' errors and send them to a singular place. Here's an example using Sentry's node.js library to capture and send all failure errors to Sentry.

import * as Sentry from "@sentry/node";
import { inngest } from "./client";

Sentry.init({
  dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
});

export default inngest.createFunction(
  {
    name: "Send failures to Sentry",
    id: "send-failed-function-errors-to-sentry"
  },
  { event: "inngest/function.failed" },
  async ({ event, step }) => {

    // The error is serialized as JSON, so we must re-construct it for Sentry's error handling:
    const error = event.data.error;
    const reconstructedEvent = new Error(error.message);
    // Set the name in the newly created event:
    // You can even customize the name here if you'd like,
    // e.g. `Function Failure: ${event.} - ${error.name}`
    reconstructedEvent.name = error.name;

    // Add the stack trace to the error:
    reconstructedEvent.stack = error.stack;

    // Capture the error with Sentry and append any additional tags or metadata:
    Sentry.captureException(reconstructedEvent,{
      extra: {
        function_id,
      },
    });

    // Flush the Sentry queue to ensure the error is sent:
    return await Sentry.flush();
  }
);