Frontier LLMs fail real BI tasks — specialized data tools close most of the gap

Frontier LLMs fail more than half the time on real business intelligence tasks — not because the questions are tricky, but because the data is a mess before any question gets asked. BI-Bench makes this concrete: 100 test cases drawn from actual Power BI dashboards, each requiring a system to search for relevant tables, reshape non-relational spreadsheet data, predict join keys across complex schemas, and only then answer the analytical question. The best frontier model tested, o4-mini, tops out at 48.2% accuracy using SQL.

The gap between NL2SQL benchmarks and this one is striking. Models that score above 70% on BIRD and Spider 2.0-lite collapse to single digits or low twenties on BI-Bench. The reason is structural: NL2SQL benchmarks hand you clean, already-joined tables. Real BI projects hand you pivot tables imported from Excel, surrogate key columns with overlapping ID ranges, and schemas with up to 94 join relationships.

BI-Agent's response to this is to stop asking LLMs to do things they reliably fail at. Table-reshaping transformations — unpivot, transpose, pivot — are handled by a dedicated transform tool built on data management algorithms that inspect full table structure rather than sampled rows. Join prediction uses a specialized algorithm tuned for star and snowflake schemas, computing value containment statistics over complete tables rather than relying on the LLM to guess from a truncated prompt. A search tool prunes irrelevant tables before the LLM ever sees them. Adding all three tools improves accuracy by an average of 14 percentage points in SQL across the models tested, with gains that are statistically significant in 19 of 20 comparisons.

Post-training pushes further. A Qwen3-8B model fine-tuned on synthesized trajectories from real BI projects, then refined with GRPO-based reinforcement learning, improves by nearly 30 percentage points in SQL with tools — reaching accuracy comparable to GPT-4o while costing roughly $0.19 to run the entire benchmark, versus over $10 for frontier models. The RL reward design matters: removing the partial credit for trajectories that return a wrong-but-non-empty table, or dropping the syntax-error penalty, causes accuracy to fall below even the SFT baseline.

The generalization result is worth noting. Post-training happened entirely on BI tasks, yet the same Qwen3-8B-RL model improves on Spider 2.0-lite, a completely different NL2SQL benchmark. The tool-use habits and coding discipline transfer.

The benchmark itself required over 400 person-hours to build. Each query was verified by checking that two independent frontier LLMs could reproduce the result given full hints; cases where they couldn't triggered additional manual review rather than automatic exclusion. That rigor matters — a benchmark that only keeps easy cases would understate the problem.