TabPFN and Factor Neutral Equity Returns: A Research Proposal

What is the current state-of-the-art financial machine learning model, and can it be used to create implementable and effective portfolios by predicting stock returns neutralized against traditional factor models?

Financial machine learning and traditional econometrics represent two competing cultures in quantitative finance, as identified by Kelly and Xiu ^[1]. Where econometrics prioritises interpretability and parsimony, machine learning trades model transparency for predictive accuracy — scaling to billions of parameters to achieve superior out-of-sample performance. This trade-off has driven machine learning adoption broadly across finance, with applications spanning macroeconomic policy, credit risk, and cryptocurrency markets ^[2].

Within the field of machine learning there is persistent effort to evaluate and compare the best performing methods, with websites such as TabArena providing live, maintained benchmarks for tabular machine learning models such as those implemented on financial datasets ^[3].

Tree-Based Methods vs. Neural Networks

A large amount of research has been directed towards applying deep learning techniques to tabular data. Despite their popular appeal, neural networks have not yet been established as the state of the art when applied to this subset of machine learning. Gradient boosted decision trees such as LightGBM and XGBoost are their greatest competitors. Debate exists within academia as to whether neural networks or tree-based methods are superior when it comes to modelling small to medium sized datasets such as those present in financial markets ^[4]. Grinsztajn et al. highlight the differences in the fundamental characteristics of both architectures, claiming that tree-based methods are indeed both more accurate and have lower computational requirements.

Tabular Foundation Models & the Transformer

More recently, researchers are trying to take advantage of a specific mechanism that redefined modern neural network architectures known as the Transformer. By extending the principle of Attention defined in the landmark paper Attention is All You Need ^[5], as well as shifting the layout of the problem from predicting samples to predicting entire datasets, tabular prior data fitted networks intend to extend the popular capabilities of deep learning into a widely applicable tabular foundation model ^[6].

TabPFN

Their efforts seem to be finding success. TabPFN, a popular implementation of a tabular prior data fitted network, has achieved recognition for its success across diverse datasets and its outperformance of traditional deep learning as well as gradient boosted decision trees. Since its initial release in 2023, the input capacity of TabPFN has scaled from datasets containing only ten thousand samples to beyond one million, and the model has found applications across a wide variety of scientific fields such as health care and life sciences as well as energy and utilities ^[7].

Prior Labs, the developers of TabPFN, released an expansive list of use cases where the model has been successfully applied in the TabPFN-3 Technical Report ^[7]. Considering its impact on the field, the effectiveness of its predictions, and its competitive computational advantages, TabPFN has the potential to be established as the new state-of-the-art within the machine learning subdomain of tabular modelling.

Application to Financial Markets

Unanswered questions remain. Of the two hundred and twelve papers and applications listed by Prior Labs in their report, only seven fall under the fields of Financial Services, Banking, and Insurance. This signals a general lack of academic input within these areas and an opportunity for impactful research. Only one of those seven publications was an academic paper related to the prediction of stock returns ^[8].

Wang and Lera used prior fitted networks to forecast Chinese and American stock returns, comparing the performance of PFNs against traditional neural networks and gradient boosted decision trees. The focus of the paper was directed towards dealing with shifting regimes, a challenging characteristic of financial markets that causes the relationships between predictive features and target variables to change over time. They concluded that prior fitted networks produced results superior to neural networks and tree-based benchmarks — an important result that supports the hypothesis that prior fitted networks have the potential to become the new state-of-the-art in tabular machine learning applied to financial markets.

In terms of academic implications, further work still needs to be done. The scope of their paper was limited by the novelty of the PFN model architecture. At the time of publication, prior fitted networks were only capable of processing small datasets of less than ten thousand samples. Inputs given to their PFN were limited to only the data from the previous date of inference, rather than across the entire history of the dataset. The scale of dataset size that prior fitted networks are now capable of handling has since increased substantially, opening the door to even further potential gains in performance.

Factor Models & Factor-Neutral Returns

Regarding the practicality of this research, there is an important consideration that remains unexplored: that of factor models and factor-neutral returns. Financial machine learning models rely on groups of features called factors that describe fundamental characteristics of companies such as earnings trends, stock price volatility, and quality ^[9]. The association between expected returns and factors has been documented in academia. Models that rely purely on factors are prone to significant practical weaknesses, such as high portfolio turnover rates that lead to trading costs so severe that the portfolio is not economically feasible.

Methods exist that are capable of neutralising exposure to these factors. This was not accounted for by Wang and Lera. Further research needs to explore the potential of prior fitted networks to model stock returns that have been neutralised to common factors, and then compared with the ability of neural networks and gradient boosted trees to do so as well.

Conclusion

The prediction of stock returns will always be of interest to practitioners invested in financial markets and to academics endeavouring to understand them. Financial machine learning depends on the continuous development of modelling tabular data, and prior fitted networks have demonstrated the potential to replace traditional neural networks and gradient boosted decision trees as the state-of-the-art foundation model for small to medium sized tabular datasets.

Despite wide application of PFNs across various academic fields, there exists a distinct lack of publicised research within the field of finance, and in particular, the prediction of stock returns. Although PFNs have displayed the ability to adjust to changes in market regime, their capability to create implementable and economically feasible factor-neutral portfolios remains unevaluated, and represents an opportunity for impactful independent research.

References