
Building Agentic Workflows That Query Millions of Rows: A Real-World Guide with AgentKit
Jakob Evangelista · 8/8/2025 · 9 min read
How We Built an AI Agent to Query 3.5M Powerlifting Records Using AgentKit
AI agents are quickly becoming the backbone of modern applications—powering everything from natural language interfaces to complex backend workflows. But building reliable, multi-step agents still comes with challenges: managing state, minimizing latency, and maintaining context across steps.
At Inngest we've been working on how to simplify agent orchestration with **AgentKit. A developer first framework for** building agentic workflows using only code
We've tested Agentkit, by building a multi-agent system that queries and analyzes 3.5 million powerlifting records using natural language.
We wanted to answer a deceptively simple question:
“Let users ask anything about powerlifting meets and return real, structured results.”
This meant solving:
- Multi-agent orchestration
- Real-time inference and result streaming
The Stack
- Frontend: Next.js AI Chat Template (Vercel)
- Agent Framework: Inngest AgentKit
- LLMs: Claude Haiku, Claude Opus, Gemini 2.5 Pro
- Database: ClickHouse, Convex
Multi-Agent Architecture with AgentKit
We structured our agent system using AgentKit to orchestrate three roles:
-
Routing Agent (Claude Haiku)
Determines whether a query requires database access or not. -
Query Agent (Claude Opus)
Converts natural language into structured queries for ClickHouse. -
Summary Agent (Gemini 2.5 Pro)
Interprets the raw query results and summarizes them for the end user.
Each agent is defined in isolation, tested independently, and composed into a single network for full orchestration. AgentKit provides fine-grained control over execution flow, streaming behavior, and model selection.
Letting LLMs Query 3.5M Rows
To enable LLMs to access millions of structured records, we implemented a language-to-SQL pattern:
- Loaded the OpenPowerlifting dataset into ClickHouse
- Primed the model with the schema using few-shot examples
- Converted each user query into a structured SQL-style call
Rather than relying on embeddings or retrieval-based search, this approach allowed precise control, schema validation, and reliable filtering.
Designing a Structured Prompt for Code-Based Tooling
We defined a dedicated agent the Meet performance analyst responsible for converting natural language into structured tool calls. This agent exclusively invoked the getMeetResults()
function, which abstracts over a SQL query targeting a ClickHouse database.
The system prompt provided:
- A detailed schema definition
- Supported operators and column names
- Strict formatting rules (no commentary, markdown, or natural language output)
- Several few-shot examples for context
Here’s the agent definition:
const meetQueryAgent = createAgent({
name: "Meet performance analyst",
description:
"Answers questions about powerlifting meet results by constructing detailed queries.",
system: `
You are an expert powerlifting data analyst. Your only job is to answer user questions by calling the "get_meet_results" tool. You must convert the user's natural language question into the structured JSON parameters that the tool expects.
Do not add any extra text, markdown, or commentary. Your only output should be the tool call.
The tool queries a database of powerlifting meet results. Each row represents a single lifter's performance at a single competition.
**Here is an explanation of the columns:**
- **Name**: The lifter's name. Duplicates are handled with a suffix (e.g., 'John Doe #1').
- **Sex**: 'M' for male, 'F' for female, or 'Mx' for gender-neutral.
- **Event**: The competition type: 'SBD' (Squat-Bench-Deadlift), 'B' (Bench-only), etc.
- **Equipment**: Equipment category: 'Raw', 'Wraps', 'Single-ply', 'Multi-ply'.
- **Age**: Lifter's age on meet day. A value like 23.5 means the lifter could be 23 or 24.
- **AgeClass**: Age category based on exact age, e.g., '40-45'.
- **BirthYearClass**: Age category based on birth year, used by IPF. e.g., '40-49'.
- **Division**: Free-form text describing the competition division, e.g., 'Open'.
- **BodyweightKg**: Lifter's official bodyweight.
- **WeightClassKg**: The weight class, e.g., '90' (up to 90kg) or '90+' (above 90kg).
- **Attempt Columns (e.g., Squat1Kg, Bench2Kg)**: Lift attempts in Kg. Negative values mean the attempt was failed.
- **Squat4Kg, Bench4Kg, Deadlift4Kg**: Fourth attempts for setting records; do not count towards the total.
- **Best3SquatKg, Best3BenchKg, Best3DeadliftKg**: The best successful attempt from the first three attempts.
- **TotalKg**: The sum of the three best lifts. Only present if all three lifts were successful.
- **Place**: A number for the official placing, or a code: 'G' (Guest), 'DQ' (Disqualified), 'DD' (Doping Disqualification).
- **Dots, Wilks, Glossbrenner, Goodlift**: Different formulas for calculating a lifter's score relative to others. Higher is better. 'Goodlift' points are also called 'IPF GL Points'.
- **Tested**: 'Yes' if the competition category was drug-tested.
- **Country, State**: The lifter's home country and state/province.
- **Federation**: The organizing body for the meet, e.g., 'USAPL'.
- **ParentFederation**: The international sanctioning body, e.g., 'IPF'.
- **Date**: The meet's start date in 'YYYY-MM-DD' format.
- **MeetCountry, MeetState, MeetTown, MeetName**: Details about where the competition was held and its name.
- **Sanctioned**: 'Yes' if the meet was officially recognized by a federation.
**The full list of available columns is:**
\`\`\`
${allColumns.join(", ")}
\`\`\`
The tool is very powerful and can filter, sort, and limit results from the database. You must convert the user's natural language question into the structured JSON format the tool expects.
**Tool Schema:**
\`get_meet_results(filters: Array<{column, operator, value}>, orderBy?: string, sortDirection?: 'ASC' | 'DESC', limit?: number)\`
**Column Names:** \`Name\`, \`Sex\`, \`Event\`, \`Equipment\`, \`Age\`, \`TotalKg\`, \`Best3SquatKg\`, \`Date\`, \`MeetTown\`, etc.
**Operators:** \`=\`, \`!=\`, \`>\`, \`<\`, \`>=\`, \`<=\`, \`ILIKE\`, \`IS NULL\`, \`IS NOT NULL\`
**Examples:**
1. **User:** "How did Jakob do at his last meet?"
**Tool Call:** \`get_meet_results({ filters: [{ column: 'Name', operator: 'ILIKE', value: 'Jakob' }], orderBy: 'Date', sortDirection: 'DESC', limit: 1 })\`
2. **User:** "who had the biggest squat in houston in the month of february 2024"
**Tool Call:** \`get_meet_results({ filters: [{ column: 'MeetTown', operator: 'ILIKE', value: 'houston' }, { column: 'Date', operator: '>=', value: '2024-02-01' }, { column: 'Date', operator: '<=', value: '2024-02-29' }], orderBy: 'Best3SquatKg', sortDirection: 'DESC', limit: 1 })\`
3. **User:** "Top 5 women's totals in the 'USAPL' federation"
**Tool Call:** \`get_meet_results({ filters: [{ column: 'Sex', operator: '=', value: 'F' }, { column: 'Federation', operator: 'ILIKE', value: 'USAPL' }], orderBy: 'TotalKg', sortDirection: 'DESC', limit: 5 })\`
Think step-by-step to deconstruct the user's query into filters, ordering, and limits. Pay close attention to dates and infer date ranges when a user specifies a month or year. For "biggest" or "best", use \`orderBy\` and \`sortDirection: 'DESC'\`. For "smallest" or "worst", use \`'ASC'\`. Always set a limit.
`,
model: anthropic({
model: "claude-sonnet-4-20250514",
defaultParameters: {
temperature: 0.0,
max_tokens: 5000,
},
}),
tools: [getMeetResults],
});
This agent was tightly scoped. It had one responsibility: take natural language input and convert it into a structured call to get_meet_results()
, a tool that supports filtering, ordering, and limiting rows from a 3.5M+ row ClickHouse table.
Here is that tool:
const getMeetResults = createTool({
name: "get_meet_results",
description:
"Returns available meet data based on a set of filters, sorting, and limits. Use this to find how lifters performed, compare them, or find lifters that meet certain criteria.",
parameters: z.object({
filters: z
.array(
z.object({
column: z.enum(allColumns),
operator: z
.enum([
"=",
"!=",
">",
"<",
">=",
"<=",
"ILIKE",
"NOT ILIKE",
"IS NULL",
"IS NOT NULL",
])
.describe("The operator to use for the filter."),
value: z
.union([z.string(), z.number()])
.optional()
.describe(
"The value to filter by. Not required for IS NULL or IS NOT NULL.",
),
}),
)
.optional()
.describe("An array of filters to apply to the query."),
orderBy: z
.enum(allColumns)
.optional()
.describe("The column to sort the results by."),
sortDirection: z
.enum(["ASC", "DESC"])
.optional()
.describe("The direction to sort the results."),
limit: z
.number()
.max(100)
.optional()
.describe("The maximum number of results to return."),
}),
handler: async (
{ filters, orderBy, sortDirection, limit },
{ network },
) => {
const query_params: Record<string, unknown> = {};
let paramIndex = 0;
const whereClauses =
filters
?.map((filter) => {
const { column, operator, value } = filter;
if (operator === "IS NULL" || operator === "IS NOT NULL") {
return `${column} ${operator}`;
}
if (value === undefined || value === null) {
return ""; // Skip invalid filters
}
const paramName = `param${paramIndex++}`;
if (operator === "ILIKE" || operator === "NOT ILIKE") {
query_params[paramName] = `%${value}%`;
return `${column} ${operator} {${paramName}:String}`;
}
query_params[paramName] = value;
const paramType = typeof value === "number" ? "Int64" : "String";
return `${column} ${operator} {${paramName}:${paramType}}`;
})
.filter(Boolean)
.join(" AND ") || "";
const orderByClause = orderBy
? `ORDER BY ${orderBy} ${sortDirection || "DESC"}`
: "ORDER BY Name, Date DESC";
const limitClause = limit ? `LIMIT ${limit}` : "LIMIT 20";
const sql = `
SELECT
${allColumns.join(",\n ")}
FROM 'powerlifting-records'
${whereClauses ? `WHERE ${whereClauses}` : ""}
${orderByClause}
${limitClause};
`;
try {
const result = await client.query({
query: sql,
query_params,
format: "JSON",
});
const data = (await result.json()).data;
// Store results in network state for other agents to use
network.state.data.meetResults = data;
return data;
} catch (err) {
const error = { error: `Query failed: ${(err as Error).message}` };
return error;
}
},
});
Example conversions:
User: "Who had the biggest squat in Houston in February 2024?"
Tool Call:
get_meet_results({
filters: [
{ column: 'MeetTown', operator: 'ILIKE', value: 'houston' },
{ column: 'Date', operator: '>=', value: '2024-02-01' },
{ column: 'Date', operator: '<=', value: '2024-02-29' }
],
orderBy: 'Best3SquatKg',
sortDirection: 'DESC',
limit: 1
})
This pattern gave us:
- Accurate and deterministic data access
- Fully inspectable logic via prompt + few-shot examples
- A clean separation of concerns between intent parsing and data summarization
And because AgentKit allows agents to be composed like functions, this logic was encapsulated in a single createAgent()
call and plugged directly into our orchestration network.
Takeaways
- Agentic applications benefit from clear boundaries between intent parsing, data access, and summarization.
- Prompting structured tool calls gives LLMs more control and accuracy over structured datasets.
- Inngest AgentKit simplifies the process of defining, executing, and orchestrating agents with built-in real-time and production-grade primitives.
Try It Yourself
- Start building: Inngest AgentKit
- Live demo: https://strengthsync.app
- Source code: GitHub repo
Want to build an agent that does more than chat? Get started with AgentKit →