Introducing RelArena-alpa, TabPFN-Rel and RPI: Predictive Queries for Relational Databases
Most of the data that actually matters to a business- users, transactions, sessions, orders- lives in relational databases, not in a single spreadsheet. SQL is built for backward-looking analysis: what happened, how much, when. It can't tell you whether a given user will churn in the next 30 days. The common practice is to derive a single table from multiple interconnected tables and treat is a supervised tabular learning problem. However, in reality this is a relational machine learning problem: making predictions directly over the structure of a database instead of a single flattened table. It's still a young field, where results are often hard to reproduce, hard to compare across methods, and rarely tested outside benchmarks like RelBench. We argue that these problems have slowed progress in relational learning.
In response, we're releasing three synergetic pieces of open-source software (all in alpha stages of development):
- RelArena-α: A benchmarking framework that finally enables trustworthy comparison of relational models
- TabPFN-Rel: A relational harness for TabPFN-3/TabPFN-3-Plus, which is currently the No. 1 model submission on RelArena.
- and our Relational Predictive Interface (RPI): An interface that enables practioniers, for the first time, to easily apply all the different baselines in RelArena-α, including TabPFN-Rel, to their real-world problems.
Alongside the release, we're publishing a detailed report. This post is the high-level version for anyone who wants the gist.
⭐️ We’ve open-sourced all of this in the relarena repo.RelArena-α: A benchmarking framework for Reproducible and Comparable Relational Learning
At Prior Labs, we believe that great benchmarks are the key driver of scientific progress. For you, as the reader, a great benchmark means you can confidently choose the right model for your use case. You can understand which models work and which don’t without relying on what one paper claims here and what another claims there. This is as important for researchers as it is for practitioners, because it lets them see what actually works and what doesn’t. Over time, selecting the right small or big changes to methods adds up, leading to rapid progress in fields with great benchmarks. We have seen this firsthand with TabArena in tabular learning.
So where does relational learning stand? Relational Learning has tasks and databases, like the ones in RelBench v1, that researchers use to evaluate their models. What is missing is a standard for evaluating all different models out there. Currently, everyone evaluates their own model and then compares their results against results reported by all the other researchers in the field. The problem is just that you can’t compare these results. There are so many reasons for this, which we outline in our model report, but in short:
- Some results just cannot be reproduced: This doesn’t mean that the people lied about their results, but this means if you would try to use their model you either couldn’t get the same results or would have a really, really hard time.
- Results can’t be compared: There are so many reasons for this, like evaluating on different states of the same datasets, comparing tuned with untuned methods, different evaluation regimes etc.. Any of these factors mean that if A is better than B based on the results of the authors, we don’t actually know if A is actually better than B or if it just looks like it.
- No common aggregate metric: A benchmark has normally many datasets and I want to know if a model is on average better than another. For this, you somehow need to aggregate across datasets scores to get a common metric that tells if A is better than B. This metric is super important, because it is what you and the research community will make a lot of decision based on. Currently, there is no such common metric in the relational learning community and instead there are many metrics being used that we think don’t make a lot of sense.
- Important baselines are missing: Imagine you look at a big table of results to decide what model to chose and the model that actually works for you isn’t even on there. That’s exactly what has been happening in relational learning. Exactly the methods that practioners use, namely flattening a databased into table, have gotten very little love in the community in the past years, leading to weak or outright buggy flattening-based baselines, which can mislead practioniers and researchers alike.
We are trying to solve all of these aspects with the release of RelArena-α: RelArena-α standardizes how models are run, tuned, evaluated, what data they get, and so on. All the great things learned from TabArena, are enabling trustworthy comparisons on the most popular set of relational learning tasks: the RelBench v1 entity-level task suite.
As we mentioned above, a great benchmark needs the right baselines represented. In addition to picking a comprehensive set of baselines from the literature, we also added one of our own, TabPFN-Rel, which demonstrates that flattening to a table might not be such a bad idea…
TabPFN-Rel: Our relational harness for TabPFN-3
TabPFN-Rel is our relational harness for TabPFN-3, and it automates the same workflow most practitioners already use to solve predictive tasks over relational databases: First, flattening the relational prediction tasks into a single table and then applying tabular machine learning methods. It’s currently the No. 1 ranked model submission on RelArena and is our contribution to a line of work that includes papers like featuretools and RDBLearn, which have shown that this flattening process can be automated.
TabPFN-3 and TabPFN-Rel were co-developed to achieve optimal performance on relational data, and we will continue to iterate on both the model and the harness in the future.
.png)
RPI: An Interface for Predicting on Your Own Database
If you come from tabular learning, this might sound odd, but there is currently no convenient way to actually run a relational learning model on your database. No scikit-learn interface or installable packages. What does that mean? You need to copy all the code for the model you want to use and carefully convert the database into the specific format the model uses, which is very different from what you are starting with.
RPI, our Relational Predictive Interface, is what lets you take any RelArena model, including TabPFN-Rel, outside a benchmark and onto your own data. You define a predictive task on your own database with a YAML config, and RPI runs the model directly against it.
Here’s a snippet from the example we walk through in the introductory cookbook:
# Predict seller churn over the Olist Brazilian E-Commerce DB: among sellers active
# in the past 30 days, will they have NO order in the next 30 days? (binary)
# Mirrors relbench's user-churn pattern (seed on backward window, label on forward).
# The database schema lives in a separate file, referenced below.
database: olist_database.yaml
entity_table: sellers
entity_col: seller_id
time_col: timestamp
target_col: churn
task_type: binary_classification
timedelta: 30 days
...
entities: all
# Seed: sellers with >=1 order in the backward window (timestamp - td, timestamp].
# Label churn=1 if they have NO order in the forward window (timestamp, timestamp + td].
query: |
SELECT timestamp, seller_id,
CAST(NOT EXISTS (
SELECT 1 FROM order_items
WHERE order_items.seller_id = sellers.seller_id
AND purchase_ts > timestamp AND purchase_ts <= timestamp + INTERVAL '{timedelta}'
) AS INTEGER) AS churn
FROM timestamp_df, sellers
WHERE EXISTS (
SELECT 1 FROM order_items
WHERE order_items.seller_id = sellers.seller_id
AND purchase_ts > timestamp - INTERVAL '{timedelta}' AND purchase_ts <= timestamp
)As far as we know, RPI is the first open-source, model-agnostic predictive interface of its kind, and we think it matters for reasons beyond convenience. Real-world data rarely looks like a pre-defined benchmark, and we think the field has spent more time optimizing for benchmarks than testing whether methods actually hold up on the problems practitioners have in front of them.
We're calling all three alpha releases on purpose. This is a release for the research community and early-adopter practitioners, not yet for industry practitioners in production. Specifying a relational prediction task is still meaningfully harder than specifying a tabular one, and RPI currently doesn't yet guard against every way that can go wrong. We'd rather put this in front of people now and improve it with real feedback than build something that does not add value to the academic or scientific community. We go into the full motivation, design decisions, and early results in our accompanying report.
How to Get Started
