What does this mean for capitalism?

The use of machine learning in finance is nothing new, but traditionally, such models operate as a so-called ‘black box’. That is, an AI algorithm sifts through mountains of data to make future predictions without revealing how it makes its determinations.

Now, researchers at Cornell University have developed a way to use machine learning to transparently assess the effectiveness of tools used to make these predictions – and to accurately predict future market movements.

Given the massive amounts of data and high volatility involved, this presents an extraordinarily undertaking.

“What we were trying to do is bring the power of machine learning techniques to not only evaluate how well our current methods and models work, but also to help us extend these in a way that we never could do without machine learning,” said Maureen O’Hara, the Robert W. Purcell Professor of Management at the SC Johnson College of Business, and co-author of the paper “Microstructure in the Machine Age.”

“Trying to estimate these sorts of things using standard techniques gets very tricky, because the databases are so big. The beauty of machine learning is that it’s a different way to analyze the data,” O’Hara said.

“The key thing we show in this paper is that in some cases, these microstructure features that attach to one contract are so powerful, they can predict the movements of other contracts. So we can pick up the patterns of how markets affect other markets, which is very difficult to do using standard tools.”

Predicting the unpredictable

In the study, the researchers assessed their algorithms using a dataset of 87 futures contracts. “Our sample is basically all active futures contracts around the world for five years, and we use every single trade – tens of millions of them – in our analysis,” O’Hara said.

“What we did is use machine learning to try to understand how well microstructure tools developed for less complex market settings work to predict the future price process both within a contract and then collectively across contracts. We find that some of the variables work very, very well – and some of them, not so great.”

The model is not yet developed enough to have tangible applications within market pattern prediction, but that is evidently the end goal for both the researchers and the financial institutions that partnered with the team on the study. Accurately predicting market activity – something of a Holy Grail for those in finance – stands to have an enormous impact on our capitalism-driven world, and this research marks the first step towards significant change.