Why are LLMs missing on Tabular Benchmark Leaderboards?

Tabular benchmarks don't include LLMs. In this article, we explore why.

Almost every week, someone tells me LLMs can do everything. Feed them a spreadsheet, and they'll predict the missing column. Ask them to forecast churn, and they'll give you an answer. That's because LLMs are trained to predict the next likely token, and they're good at it. But predicting the next likely word in a sentence isn't the same as producing an accurate prediction on structured data. An LLM can give you a plausible-sounding number. That doesn't make it correct.
Instead of LLMs, tabular foundation models (TFMs), such as TabPFN, have emerged as a strong general-purpose solution for accurate predictions on structured data. TFMs are designed specifically for predictions on structured data; trained from the ground up on the structure and statistics of tables, not on language.

The next question is almost always the same (including, admittedly, from myself - Tuana - coming from a pure AI engineer/LLM world): "Okay, so how do TFMs compare to LLMs on this?" And that's where things get awkward, because the honest answer is that the benchmarks to answer that question properly barely exist (yet).

Despite this awkward situation, it's somehow become common knowledge in academia and industry that LLMs aren't actually good at this kind of prediction task (to name a few papers, Why Large Language Models Fail at Tabular Prediction, The Illusion of Generalization in Tabular Language Models, Robustness is Important: Limitations of LLMs for Data Fitting and Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models). Yet nobody has exhaustive benchmark numbers to back it up.  In contrast, tabular foundation models (TFMs), such as TabPFN, are exhaustively benchmarked in academia and industry.

In this article, we’ll discuss why we think there are no reliable benchmark numbers for LLMs (yet).

What Tabular Benchmarks Cover Today

A good benchmark is more than a pile of datasets. It needs a validation protocol, an evaluation protocol, clear task definitions, agreed-upon baselines, and a pipeline that runs everything in the same fair way every time. Judged by that bar, not everything that gets called a benchmark actually is one.

Tabular ML has a deep history of benchmarking efforts, though the quality varies. Benchmarks and dataset collections have emerged from model development (RealMLP, TabM, TabPFNv2), from tabular deep learning comparisons (TALENT, ScoringBench, Gorishniy et al., Shwartz-Ziv & Armon, Grinsztajn et al., TabZilla, Shmuel et al., Zabërgja et al., MultiTab, OmniTabBench), from AutoML development (Hanussek et al., Zöller & Huber, Assembled-OpenML, Jiang et al., AMLB, Jurado et al.), from a data-centric view (OpenML100, OpenML benchmarking suites, PMLB, PMLB v1.0, OpenML-CTR23, TabRepo, Tschalzev et al., Kohli et al., TabReD, RamanBench), or from multimodal tabular data tasks (Shi et al., AutoGluon-Multimodal, CARTE, TextTabBench, TIME, STRABLE, MulTaBench).

Last year, TabArena distilled much of the deep tabular benchmarking history into a single "living benchmark": a maintained, versioned leaderboard rather than a static paper release. It covers around 50 curated datasets with a strict protocol to avoid the usual sins of tabular benchmarking: duplicated datasets, unrepresentative data domains, non-predictive tasks, and data quality issues such as target leakage. This year, BeyondArena extended TabArena to around 140 datasets, including non-IID data, larger datasets, tabular data with text, and wider domain coverage.

These extensive resources and benchmarks enable us to compare TFMs and traditional models, such as MLPs and gradient-boosted decision trees, in detail. Yet, LLMs are consistently absent in such standardized benchmarks.

Literature comparing TFM or traditional models with LLMs is rare and often immature. When LLMs emerged, several efforts investigated how well LLMs can solve predictive tasks by treating tables as text (TaBERT, TabLLM, Tab2Text, TabuLa-8B, JOLT, TabGemma, Fang et al., Pavlidis et al.), yet these efforts rarely generalized to more rigorous evaluations and comparisons with TFMs or traditional models (CARTE, TabICLv2, ClinicalBench, Elephants Never Forget,  Silvestri et al., Liu et al., Gorla & Puduppully).
From this research push, the best evaluations of LLMs occurred for tabular few-shot learning tasks: LLMs do well when there are only a handful (≤64) of labeled rows, where semantic understanding trumps statistical learning. This direction deserves more study, including the hard question of whether few-shot tabular prediction is a real task at all or an artifact of how we construct benchmarks. We should establish that the task exists before racing to win at it.

Why Benchmarking LLMs on Tabular Data Is Genuinely Hard

Building a fair benchmark that includes LLMs alongside tabular models runs into a few real problems. Besides that, quite honestly, it is very annoying to do.

First, serialization. An LLM doesn't see a table; it sees whatever text you turn that table into, and the format you choose (CSV, JSON, Markdown) can change the results unfairly.
Second, cost. Running a large LLM over hundreds of datasets and thousands of rows costs far, far, far more than running any other machine learning method, which limits how comprehensive an LLM-inclusive benchmark can be. Third, contamination. Many public tabular datasets have been available on the web with machine-readable access for years, so large LLMs have memorized the correct answers during pretraining rather than actually predicting them. Once a model has seen the right answer, it can repeat it convincingly, which looks like good performance without being it.

This is why a fair LLM-inclusive benchmark would need to be different from past tabular benchmarks. In particular, LLM-inclusive benchmarks would need to become exclusive by being closed and hiding tasks and datasets from researchers. A trade-off that would hinder the progress of open research for TFMs or traditional models. So, why has it not been worth doing this (yet)?

1. Unspoken, implicit constraints. When someone develops a predictive model for real-world use, that goal comes with a long tail of constraints nobody writes down. Practitioners often assume a standard sklearn fit/predict API; that inference has to be reasonably fast for a given task at a reasonable cost; and that it can be deployed. Moreover, in a lot of the domains where tabular data actually lives, the data can't leave the building at all. Elo on a benchmark says nothing about any of this, and today's benchmarks mostly don't ask. And when today’s benchmarks ask, they do so poorly, e.g., by producing Pareto fronts composed of models that are rarely optimized for real-world deployment efficiency or given the chance to trade off accuracy with speed. If we want LLMs and TFMs in the same table, benchmarks need to start explicitly categorizing these constraints, tuning for them where it matters, and enforcing them.
2. Too much $$$ for too little signal. Running a frontier LLM across hundreds of datasets is expensive, and every empirical and theoretical signal we have from the literature points in the same direction: LLMs are not good enough at typical tabular prediction tasks. That combination makes the cost hard to justify. It's worth being clear here: if a method appeared tomorrow with a rigorous signal for useful tabular prediction, it would be on TabArena the next day, and academia and industry would happily set aside the cost or leakage concerns to find out. The absence of LLMs from these leaderboards isn't gatekeeping. It's that nobody has shown up with a result worth the computing bill, or with the motivation to foot the overhead and bills themselves. This lack of motivation also tells us a lot about how much people believe in their own LLM-based solutions for tabular data.
3. Let's be real, it's annoying. Even knowing that someone should do it, the day-to-day of benchmarking work is miserable. Maybe even more so for LLMs, as frontier model versions can drift under you mid-experiment. And when the numbers come out unflattering, the reply is reliably that you used the model incorrectly. Nobody (really, nobody) wants to do benchmarking full-time. The basics are drudgery, and you rarely get far enough to reach the fun part, where you design experiments that actually spark curiosity and insights. Instead, it is almost always time to ship yet another leaderboard with more baselines and more datasets.

Where we are now

The core reason you can't find a good head-to-head comparison of TFMs versus LLMs for tabular prediction is that valid benchmarks with LLMs require hidden evaluation datasets to prevent contamination, which is so far missing from rigorous large-scale benchmarks, as well as substantial capital investment and time.

Despite all we said above, it is very likely that LLMs will appear on future tabular leaderboards. Why? With the industry moving towards closed-source APIs for TFMs and the advent of agentic systems, we will, in any case, need benchmarks with hidden evaluation datasets to avoid foul play. Once we have that, someone will be willing to pay the price to get LLMs on this leaderboard, either to win a heated online argument or to please reviewer #2, who is asking for LLM baselines.

If you want to work together with us on benchmarking (of LLMs), reach out to Lennart for collaborations. We are also hiring for work like this https://priorlabs.ai/careers!

‍