Two and a Half Years of Quantitative Research

I develop machine learning models to predict long term global equity returns. When asked to explain what I do for work to someone who doesn't know too much about finance or ML, I tell them I use AI to predict the stock market. Unfortunately, it isn't as easy as asking ChatGPT whether Microsoft's share price will go up or down today…

Shortly after finishing my undergraduate degree in mathematics, my research in financial machine learning began as a data science internship with a local hedge fund startup.

My primary task has been to develop models to compete with in the Numerai Data Science Tournament. Numerai is a hedge fund based in San Francisco that crowdsources the intelligence of data scientists across the world to power their global equity portfolio.

There are three things that contribute to developing a successful machine learning model for Numerai: backtesting, pipeline development, and exposure to frontier research.

Backtesting

Backtesting is training and evaluating the performance of models on historical data. Important considerations include preventing data leakage, which is ensuring that your models do not have access to data that would be unseen in real life circumstances. If you are testing the performance of your model in the year 2018, then you must make sure that all of the data that it has had access to was only available at the end of 2017.

Something else to watch out for is the statistical issue of overfitting, when your model learns to predict patterns within data that do not necessarily generalise. Models or practitioners that place too much emphasis on short periods of high performance are biased and are less likely to work out of sample on live data. There are a couple of techniques used to prevent overfitting. Firstly, we make sure that the test set — the period of time that we evaluate the performance of our models on — is a sufficiently large. A few months is not enough time to know whether a model is effective or not. Secondly, we use multiple seperate evaluation periods. A strategy that may have worked from 2010 until 2015 may not be as effective from 2015 to 2020.

Pipeline Development

Once you have identified your best model, the next step is to build a pipeline that feeds it data and generates live predictions. This is a meticulous process. The slightest of mistakes can result in outputs that are different to what you thought you were examining during testing, and subsequent poor performance. The processes that you used to differentiate your model and the data that it uses as input need to be replicated exactly. Unit tests are short programs that can verify that your model is behaving the way you expect it to. Another way of making sure everything is in order is by keeping track of your model's churn, which in the context of Numerai refers to the autocorrelation of the predictions produced by your model from one week to the next.

Frontier Research & Alpha

Succeeding at Numerai is not simple. The metrics which are used to evaluate your models are not based on raw performance alone, but on what your model provides that the models of other competitors do not. You are rewarded for discovering unique models that uncover patterns yet to become common knowledge.

Finding techniques that are unique and effective — often described as alpha in the world of quantitative finance — is synonymous with being a successful Numerai data scientist. How one goes about acquiring them is an individualised process that depends on a combination of creativity and knowledge within the fields of mathematics, statistics, machine learning and finance. Academics in all four of these fields are releasing papers constantly, and all of them may be hiding valuable secrets that could be taken advantage of.

Beyond keeping up with the latest machine learning literature, one's own skill as a data scientist, which comprises data analysis, engineering, and experimentation, also proves vital, as individuals use their own insight and experience to uncover the statistical gems hidden in the mines of huge, noisy datasets.

I will now discuss two characteristics of quantitative finance that have guided my research process thus far: feature neutralisation and the application of tree-based statistical learning methods as opposed to neural network architectures.

Feature Neutralisation

We are given a universe of stocks in the form of a two-dimensional table of rows and columns, where each row represents values associated with a stock at a given date. Among the columns that describe each row is the label that identifies the stock, as well as descriptive characteristics of the company called features — for example its current market capitalisation, or the amount of revenue made over the past year. A third type of column is forward-looking: it describes whether purchasing the stock today will result in a positive or negative return on investment. The target is "forward-looking" because it can only be determined after the fact.

The task given to us by Numerai is to use the features to predict the targets, and is a classic regression problem. Linear regression is a tried and tested method that has been around for many years, but in the context of modelling stock returns it has significant drawbacks. Correlations between features and targets are unstable and small in magnitude, so strategies based on linear models tend to underperform and have high volatility.

To prevent taking the risks associated with betting on linear models, Numerai encourages a method called linear feature neutralisation. It is possible to mitigate these risks by subtracting away the prediction of linear models from the target column. This "neutralised target" can then be trained on by more complex non-linear models that will hopefully be able to find subtle and stable non-linear feature-target relationships, providing both an improvement in long run average performance as well as a reduction in the volatility of the strategy.

Model Methodologies

The complex models capable of learning non-linear relationships that are being referred to usually exist in two different forms: decision trees and neural networks. One of the most commonly used tree-based methods is LightGBM, developed and released by Microsoft in the late 2010s. It provides an easy-to-use and efficient implementation of a gradient boosted decision tree — a modern statistical learning algorithm that leverages differential calculus to build an ensemble of increasingly accurate decision trees, producing a final prediction by averaging the output of all trees constructed during training.

Neural networks are the brains behind recent advances in large language models like ChatGPT and Claude, and considering their incredible performance it is no surprise that there have been attempts to apply similar techniques on financial data. They are however not as easy to train and optimise as their tree-based counterparts. There is academic debate over which of these architectures are more effective at uncovering the information within medium-sized tabular datasets like those that describe financial markets. A data scientist like myself might opt for the tree-based methods simply because the time and energy saved on not implementing complicated neural networks can be spent on engineering and analysing datasets, which often produces valuable insight.

AI in Research

I believe it to be important for any scientist describing their past few years of research to include at least a brief discussion of how their experimental process has been impacted by recent advances in artificial intelligence. I find myself a cautious yet optimistic observer of AI who takes advantage of the productivity gains provided by tools like ChatGPT and agents, whilst still acknowledging their limitations. The introduction of these tools marks a shift in the skillset required by any individual attempting to use computational power to achieve research goals, in that the medium of communication between man and machine is moving closer towards natural language and beyond high-level programming languages like Python.

Conclusion

Developing a successful model of the stock market has proven to be elusive and challenging. Leakage-free backtesting and preventing overfitting are only a fundamental platform upon which further development can be made, and techniques like tree-based methods and neural networks are already available to all of your competitors. Creativity, experience, and persistence will always be required, when the reality is that at any point in time a sudden significant drop in performance can bring you right back down to square one.

Even though it is incredibly difficult, with enough AI-backed research and a bit of luck, one might become acquainted with quantitative finance well enough to walk its hidden paths to success. As for my career as a quantitative researcher, I am ready to widen both the scope of my knowledge and the breadth of its applications, whilst perhaps providing some light for those that seek to join me.