How cubic reduced false positives by 51% by orchestrating their multi-agent system with Inngest
“Inngest's event and step architecture are essential to cubic. Our Agents trigger numerous nested and parallel steps; The ability to string them together with good traceability and observability is a game-changer!”
cubic revolutionizes the code review experience by operating AI Agents that quickly identify bugs, suggest fixes, and generate diagrams of architectural changes, freeing developers from low-level review tasks.
cubic's AI Agents seamlessly integrate with GitHub to build a long-term memory that learns codebase patterns over time, helping teams at Granola, n8n, and Browser Use merge Pull Requests 4x faster.
The observability challenge of multi-agent systems
Like many YC startups, cubic began its journey as a PoC quickly brought to market.
The first cubic version released to customers revealed new challenges. cubic, integrated with larger codebases, encountered edge cases that caused its multi-step, agentic system to fail or become stuck unexpectedly, compounded by limited observability that made diagnosing issues difficult.
The AI review would sometimes fail without explanation. I quickly realized we needed an async queuing system to handle this on serverless while integrating with our existing observability system.
Allis immediately started researching serverless queuing solutions that would match their requirements:
- DX: a smooth learning curve, easy to integrate into an existing codebase
- Fully managed infrastructure that perfectly integrates with serverless to tackle timeouts
- Orchestration capabilities: retries and support for long-running processes (multi-steps)
- Observability and Tracing: best-in-class observability to operate Agents at scale
A thorough benchmarking of serverless queuing solutions led Allis to quickly start a PoC with Inngest, which only took a few hours to finish during a flight.
From manual logs tagging to fine-grained traces
Setting up Inngest brought state-of-the-art observability to cubic's AI Agents, replacing manual tagging and grepping of logs with fine-grained and searchable traces:
Compared to what I was doing before, which was essentially tagging everything with trace IDs and doing hardcore grepping, it was much easier to see all the steps and sub-steps triggered by an event
Inngest's dashboard is critical in enabling cubic to operate its multi-agent system, filling the gap between user interactions (Pull Requests) and Agent runs:
- The Runs search enables cubic to find all Agent runs related to a specific Pull Request, as well as quickly filter Agent runs based on custom data (eg, token used).
- The Traces view provides a detailed insight into a given Agent run, allowing for the inspection of its steps, outputs, and actions.
I really liked the Inngest dashboard. I'm often looking for specific customers and PR numbers. I appreciate how easy it is to query for those as well as for specific function names.
Reducing false positives by 51% by leveraging a multi-agent architecture
Inngest has also played a critical role in helping cubic improve the accuracy of its AI Agents.
Their "Learnings from building AI Agents" article states that the initial public release of cubic, designed as a 'Single, Do-Everything Agent,' led to an influx of Pull Requests filled with low-value comments, minor nitpicks, or many false positives.
Following this observation, an overhaul of their product took place, transitioning from a single-agent system to a multi-agent system, which resulted in a 51% reduction in false positives.
Leverage Inngest's orchestrator to tackle the challenges of multi-agent systems
A significant improvement involved splitting the 'Single, Do-Everything Agent' into a multi-agent system organized by expertise:
- Planner: Quickly assesses changes and identifies necessary checks.
- Security Agent: Detects vulnerabilities such as injection or insecure authentication.
- Duplication Agent: Flags repeated or copied code.
- Filtering Agent: Designed to filter for false positives and confirm issues found.
Inngest's step APIs facilitated a smooth shift to a multi-agent system by offering built-in support for parallel execution and enabling agents to connect through its event-driven architecture.
While improving the overall accuracy of the product, moving to a multi-agent system presented several challenges, where Inngest proved to be a valuable aid:
- Race conditions among multiple parallel Agents were solved by leveraging Inngest's idempotency feature
- The intermittent serverless timeouts were gracefully handled with Inngest's timeout callbacks
Building robust Agent tools with Flow Control
Another key milestone of the product overhaul involved transitioning from providing a lot of context to Agents (”context-pushing”) to enabling Agents to pull context on demand using tools (”context-pulling”).
Tools connect Agents to third-party APIs and a secure sandbox environment, which comes with its set of external constraints, such as provisioning or rate limiting.
Inngest's Flow Control features ensure that cubic Agents get reliable tools by handling rate limits, spreading load evenly with debouncing, and prioritizing tasks.
Iterating on Agents' reasoning with Inngest Traces
All of the above enhancements were achieved by incorporating detailed logging of the Agent's outputs, which helped clarify which tools were effective and clarified the overall decision-making process.
Inngest's Traces, available from local development to deployed applications, turned out to be the perfect tool for identifying flawed reasoning patterns and creating a foundation to diagnose and resolve root causes in production.
Ship AI Agents from PoC to production with Inngest
Building and deploying AI Agents presents unique challenges, from quickly building a PoC to reliably operating numerous parallel Agents execution and associated tool calls at scale. cubic's journey demonstrates how Inngest provided the infrastructure needed to transform their product from a proof of concept to a production-ready, multi-agent system that delivers timely and accurate Pull Request reviews.
For teams looking to build and ship AI Agent systems, cubic's experience highlights the importance of selecting the right orchestration layer. As AI Agents grow more complex and specialized, tools like Inngest become essential for managing the intricate dance of events, steps, and parallel processes that power next-generation AI products.
Ready to build AI products with Inngest? Book a call with our experts today to learn how we can help you orchestrate reliable, scalable AI workflows for your business.