SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering

Comparison of feature generation mechanisms and interpretability across AutoFE paradigms

Traditional AutoFE rely on blind operator stacks, yielding high model scores but poor interpretability. LLM-based AutoFE offer semantic clarity but suffer from costly iterations, hallucinations, and performance instability. SymboLLM-FE synergizes symbolic regression with LLM refinement to achieve both high predictive accuracy and human-interpretable transparency.

Abstract

Tabular data, as a core data format in machine learning, often lacks the discriminative power needed for high-performance modeling due to insufficient feature informativeness. Automated Feature Engineering (AutoFE) overcomes this by automating feature generation and selection, ensuring both model performance and operational efficiency.

However, traditional AutoFE often yields features with poor interpretability because they rely on blind mathematical transformations, while large language model (LLM)-based AutoFE faces challenges in requiring costly multi-round iterations to generate high-utility features, compounded by inherent risks of bias and hallucination.

In this paper, we combine symbolic regression with LLMs for feature engineering (SymboLLM-FE) to solve these challenges. We extract mathematically expressive formulas strongly correlated with the target via symbolic regression — which enhances model performance — then refine them by LLMs with rich prior knowledge to ensure interpretability. Empirical results on six real-world datasets and four Kaggle competitions demonstrate that SymboLLM-FE outperforms existing AutoFE, addressing the dual challenges of poor interpretability and numerous iterations via a statistical prior-grounded LLM refinement mechanism with only single-digit LLM calls.

1. The AutoFE Dilemma

Existing AutoFE approaches are caught in a dichotomy: traditional methods maximize model performance at the cost of interpretability, while LLM-based methods prioritize semantic clarity but sacrifice efficiency and reliability. A comparative analysis (Table 1 below) reveals critical bottlenecks in both efficiency and feature quality.

Traditional AutoFE

Relies on a blind operator stack: features are opaque mathematical compositions with no semantic grounding. Large feature sets (e.g., OpenFE generates 1,436 features) fail to provide actionable insights and generalize poorly.

LLM-based AutoFE

Generates semantically coherent features from task background, but requires extensive iterative trial-and-error with downstream predictors (up to ~50 rounds) and remains susceptible to hallucinations and implicit biases that degrade reliability.

2. SymboLLM-FE Framework

SymboLLM-FE is a two-stage collaborative pipeline. In Stage 1, a Spearman correlation-guided expanding-sliding window reduces the subset search space from exponential O(2ⁿ) to polynomial O(n²), and symbolic regression mines explicit, mathematically grounded candidate formulas into a formula repository. In Stage 2, the LLM acts as a deterministic feature integrator: grounded in the discovered formulas and their validation scores, it injects domain priors via chain-of-thought reasoning, synthesizing semantically meaningful, executable feature-engineering code that is validated by a downstream predictor through an iterative refinement loop.

Overview of the two-stage SymboLLM-FE framework

Stage 1 constructs a formula repository via correlation-guided symbolic regression; Stage 2 refines formulas into interpretable features through a grounded LLM loop with downstream predictor validation.

3. Experimental Results

SymboLLM-FE is evaluated on six real-world datasets (binary / multi-class classification and regression) from OpenML and Kaggle, with four downstream predictors (CatBoost, XGBoost, MLP, TabPFN) against two traditional and five LLM-based AutoFE baselines. It achieves statistically significant improvements over traditional AutoFE (average gain of 1.23%) and approximately 1% higher accuracy than LLM-based AutoFE, with an average +2.5pp score improvement over vanilla TabPFN across four Kaggle competitions.

Table 1: Efficiency and cost comparison across AutoFE. Model Score is the average classification accuracy (downstream TabPFN).
Method	Generated Features	Model Score	LLM API Calls
AutoFeat	2	79.75	–
OpenFE	1436	79.53	–
CAAFE	30	79.94	10
OcTree	50	79.52	50
FEBP	200	79.36	29
LLM-FE	30	79.59	35
LLM-RANK	–	78.86	7
SymboLLM-FE	70	80.02	4

Table 2: Main results (downstream TabPFN). ↑ higher is better (Accuracy); ↓ lower is better (RMSE). Best in bold, second best underlined.
FE Method	Credit-g ↑	Spaceship ↑	Cmc ↑	Academic ↑	Ailerons ↓	Tesla ↓
FE Method	Acc.	Acc.	Acc.	Acc.	RMSE	RMSE
Baseline	77.03±0.47	80.79±1.27	57.85±0.89	77.33±0.67	5.10±0.48	2.39±0.09
AutoFeat	77.83±1.43	80.76±1.17	57.85±0.32	76.80±0.46	5.04±0.45	2.38±0.00
OpenFE	76.50±3.34	80.30±1.00	57.85±2.35	76.42±0.61	5.26±0.49	2.18±0.06
CAAFE	78.00±0.71	80.99±1.00	57.78±1.28	77.29±0.56	5.03±0.46	2.66±0.09
OcTree	76.50±0.82	80.79±1.15	57.85±1.52	77.33±0.70	5.09±0.47	2.43±0.10
FEBP	77.50±0.41	80.85±1.29	57.93±0.48	77.36±0.35	5.08±0.48	2.41±0.08
LLM-FE	76.67±0.62	80.22±0.96	57.29±2.20	76.42±0.61	5.05±0.46	2.39±0.09
LLM-RANK	77.50±0.82	80.70±1.15	57.74±0.85	77.25±0.19	5.10±0.48	5.87±0.19
SymboLLM-FE	77.00±1.63	81.27±1.31	57.97±0.73	77.89±0.23	5.02±0.46	2.16±0.06

BibTeX

@inproceedings{cheng2026symbollm-fe,
  title     = {SymboLLM-FE: LLM-Accelerated Symbolic Regression
               for Automated Feature Engineering on Tabular Data},
  author    = {Cheng, Zi-Jian and Jia, Zi-Yi and Zhou, Zhi
               and Li, Yu-Feng and Guo, Lan-Zhe},
  booktitle = {Proceedings of the 2026 Conference on Empirical
               Methods in Natural Language Processing},
  year      = {2026}
}