The backtest ends in 2024: An MFT trader’s perspective

About the author

Alex is based out of southern Portugal. He has traded since 1995, spent 13 years on a large broker's institutional desk, then left the industry in 2017. He joined BHFT in 2023 and spent his first year and a half helping it build a traditional-finance business.

In late 2024, Alex started building his own medium-frequency strategy on weakly cointegrated pairs, mostly Chinese exchanges against Western markets. He wrote it in Julia with the help of AI tools, and it has been trading live since October 2025.

The point of convergence

My first algorithmic strategy lost money as soon as it went live. Before that I'd spent several months reading what the literature on arXiv had to say about trading this kind of strategy, writing my own simulator and running the strategy on it, first against synthetic price processes and then on real market data. The simulator said the strategy worked. Production disagreed.

A year on, the two finally agree. The simulator now produces roughly the same equity curve and roughly the same final number as the live book. Getting there took live fills to calibrate the execution model, a hunt through a data vendor's timestamps, and a decision about how far back historical data is worth looking. My answer to that last question is: no further than 2024.

This piece explains why. It covers what has changed in the markets, what a historical backtest can't show you, how I calibrate execution on live trading, and why a medium-frequency strategy (MFT) takes so much longer to judge than an HFT one. It ends with the rules I now work by. The arguments are mine, and some of them are open to dispute.

Where the history stops

I don't backtest on anything earlier than 2024. That leaves about two and a half years of data, and I don't think more would help.

Every technological wave has changed how markets work. Floors where traders yelled instructions at each other gave way to brokers who took orders by phone and entered them into electronic systems. Then came direct electronic access, which let people send orders to the exchange themselves. Later, algorithms began making the trading decisions as well. Each step changed who is on the other side of your trade. I think the past few years of AI and more capable automated systems are another such step, and markets don't go back.

The practical problem is fitting. Fit a model to five years of data and a large share of those five years describes a market with different participants, a different ratio of people to algorithms, and different patterns. The model learns things that have stopped being true and then trades in a market where they aren't.

I used to think the Covid period was worth testing on because it contained frightening moves. Practice showed that frightening moves now happen regularly without a pandemic. In July 2025 COMEX copper had its steepest one-day fall on record, 22%, when the White House left refined copper out of a 50% import tariff the market had already priced in. Oil spent two months jumping several per cent on every statement about Iran. Each of these episodes looks different from the last, and a longer archive gives you no more of the next one.

Why history goes stale

Fewer people in the chain

Much of the investment industry now runs without anyone looking at a screen. A US employee's pension contribution goes to a fund. An algorithm decides how much goes to which manager. The managers' systems decide what to buy, and execution algorithms work the orders on the exchange. By the time I stopped trading equities, manual intervention was falling by roughly half every year.

The IMF's October 2024 Global Financial Stability Report shows how fast the mix of participants is changing. The share of AI content in patent applications for algorithmic trading rose from 19% in 2017 to over 50% every year since 2020, and the IMF's analysis of trading data already shows changes in some markets consistent with AI adoption. When the participants change, the statistics change with them. A backtest over a long window mixes several populations into one sample.

Regulation has pushed in the same direction. After every sharp fall someone has lost money and someone has to be blamed, and the usual response is another restriction. Short-selling bans are the clearest case. In a sell-off, short sellers closing at a profit are natural buyers who take on no new risk.

Ban them, and everyone who is long is trying to sell to people who would have to open new positions. Korea ran the longest short-selling ban in its history, from November 2023 to March 2025. A review by the Korea Capital Market Institute found that the ban did nothing to lift prices: Korean stocks were the weakest performers among the major Asian markets. Locally it looks as though the ban helped. In practice it makes the market more fragile at exactly the moments that matter.

The order book emptied for a similar reason, and regulators played a part in it. They punish spoofing, placing large orders you don’t intend to fill. Scalpers who front-run a large visible order are treated as a normal part of the market. If I bid for 50 lots, a robot buys one lot in front of me; if I raise my price, it sells to me a tick higher. So nobody shows size or interest any more, and the books have thinned out. I remember screens with $100m or $200m on the bid. If your model found an alpha in equity order books in 2007, by 2012 there was only noise left in them. 

Today you see one to ten contracts and icebergs everywhere. Level 2 and level 3 data from ten years ago show depth that no longer exists, and a model trained on them learns a market nobody trades in. Crypto venues are a partial exception: displayed size there is more often real size, though nothing stops anyone from slicing an order into millisecond pieces.

Everyone is mining the same data

There are millions of people with training in maths and statistics going through the same cheap, easily available data, US equities first, looking for alphas, and ML models now run much of that search for them. If you've found a pattern and it pays, someone else will find it too. A pattern your machine found in data from three years ago won't work now.

Traders without their markets

We once took on a trader to run a separate venue. He had years of manual trading behind him and millions of contracts traded. He'd then spent eighteen months out of the market. For three months his results hovered around zero, and he decided not to carry on. His skill hadn't got worse. The environment it worked in had gone.

What doesn't go stale

Long history doesn't help with the pairs I trade either. Start with the word "trend". Language says a stock is rising, and the listener pictures a continuous line with a slope that carries on for a while, the way a differentiable function does. A price is a sequence of separate trades. The tenth trade may be higher than the first, and that is all you can say: there's no tangent and no continuation. Much of what gets called trend-following is an attempt to fit tangents to scattered points.

A trend describes a single price. A pair is about how two prices relate to each other, and there are two kinds of relationship to tell apart: correlation and cointegration.

Correlation is a real relationship, and it's hard to make money on. When the dollar falls, commodities rise, but by the time you see it they have already risen, because robots reprice them instantly. You can trade correlation as lead-lag: get the data out of Chicago first and buy a lagging market, a Brazilian index, say, before it catches up. That's a business of speed and technology, picking up pennies in front of a train. It's a respectable business, but it's engineering more than quantitative research.

Cointegration is different. Two prices can drift apart, but not without limit, because there's a spring between them. Here are some examples.

• Copper in London and copper in Chicago can't settle at a twofold difference: metal can be bought in one place and shipped to the other, and industrial buyers switch to whichever is cheaper.

• WTI and Brent aren't interchangeable in a given refinery, but both end up as petrol, so if one gets too expensive, demand moves to the other. A factory in Italy can buy coal from Brazil or from Australia, with freight at 20–30% of the price depending on diesel.

• Gold trades on futures exchanges around the world, and the gaps between them rarely exceed 2%: moving gold costs almost nothing relative to its value, and a central bank can simply buy wherever it's cheaper.

• Soybeans in China are a weak spring. Domestic beans cost 50–60% more than imported ones and can range from parity to double, because shipping bulk cargo is expensive and substitution is slow.

Trading a weak spring

My strategy trades pairs with what I call weak cointegration. A strong spring might drift between 0.3% and 0.5% over a week, and moves of 0.05–0.07% are enough to make money, provided you trade regularly and don't get stuck in positions. The spring is economic: freight, tariffs, refinery configuration.

Whether it works today is a question about today's economics, and a decade of old prices answers it badly. The copper episode above was exactly that: a spring stretched by a tariff threat and then snapping back in a single session.

What history won't show

Your own footprint

The one thing a historical record can't contain is your own orders. Take a deliberately grotesque example. In a thin market you buy at 100 and sell at 101, on volume the market can't digest. Your own flow holds the price inside that corridor, and there's no trace of it in the history you backtested on, because you weren't there.

A 2024 study of this effect tracked every trading account on the Tokyo Stock Exchange over eight years. For every liquid stock, the average price impact of a large order grew with the square root of its size. A backtest implicitly assumed the impact was zero.

The practical version is harsher. In the backtest the instrument ranges from 90 to 110. In production it ranges from 97 to 103, and the break out of that range comes when my position is fully built. While I'm building it, my orders hold the range; once I stop, that support fails. The result is an asymmetry: I earn less than the backtest in quiet periods and lose more in bad ones. The backtest is wrong on both sides in the same direction.

Figure 2. The same instrument as the backtest sees it and as production sees it. The live range is narrower because the strategy's own orders are inside it, and it breaks once the position is complete. Schematic.

Returns that are something else

A good-looking backtest often hides a different exposure. I once interviewed a series of crypto traders, all with backtests showing 100–200% a year. Then a colleague who joined to review candidates showed me a quick test: overlay the candidate's P&L on the bitcoin price. The curves almost matched. The strategy was, in effect, long bitcoin, and it would work until bitcoin fell.

Options newcomers quickly discover that selling far out-of-the-money options pays for two or three years in a row. Then a crisis takes the whole capital in one go. Mean-reversion spreads, which is what I trade, carry the same shape of risk: the spread moves within a percentage point either way for months and then jumps 20%, and if you're still in the position you give back a year's earnings.

The execution model is calibrated in production

If a historical backtest can't show your own footprint or the exposure hiding behind its returns, the check has to come from somewhere else. For me it comes from a simulator built to match production, and a simulator like that has one job. Run over a period you've already traded, it should reproduce what production did: roughly the same curve and roughly the same result. 

Matching the final number alone is easy to fake. If one curve goes up steadily and the other lurches, they are different strategies, whatever the totals say. Getting to that point needs three things: clean data, faithfulness to production code, and an execution model. The third is the hardest, and it can only be built with production data.

Figure 3. The three parts of a simulator worth trusting. Only the execution model needs live trading to calibrate, and it's the part that decides whether the simulated P&L means anything.

Clean data

Any historical dataset, from any vendor or straight from the exchange, contains glitches: a bad print, a price that existed for 10 seconds. Leave them in and the simulator will happily sell there and buy back lower, earning money on a price nobody could have traded.

The subtler problem is time. Two weeks before my simulator and production finally converged, the simulator was still earning about 20 percentage points a year more than the live book: 60% against 40%. I'd been cleaning pessimistically, smoothing and removing whatever looked suspicious, and was close to giving up. Then I checked how often the last trade of each minute in our historical data matched what production had actually received. The answer was 60–65%. I'd assumed it should be close to 100%.

The data came from a vendor who records it on its own collection servers. The files carry a single timestamp and don't say which one it is; it's probably not the exchange's. My own servers also see the data later than the exchange does, about 10 ms later in China and around 100 ms in Chicago. That's normal, since it's the data I trade on.

The upshot was that the historical data reached the model earlier than production could ever have seen it, which gave the model a small, hidden look into the future. I searched for the shift that gave the best match with my production feed. For China it was almost half a second, for Chicago it was 200 ms. The match rose to about 85%, and the last 20 points of excess return disappeared. The curves converged.

Figure 4. Aligning vendor timestamps with what production actually received. A shift of one snapshot in China and 200 ms in Chicago raised the match rate and removed a look-ahead worth 20 percentage points of annual return.

That makes a great bootcamp task: here is raw data, here is a simulator that makes 80% a year on it, production makes 10%, find what's wrong with the data. 

Faithfulness

The decision module in the simulator has to be the production module, line for line. It's tempting to rewrite it for speed, because production runs checks that a simulation doesn't need. The moment you do, the simulator is testing a different strategy, and it will usually test a better one, because it has features production doesn't.

The surrounding work is more fiddly than hard. In production the system runs in real time. In the simulator you walk through a database, so every event that happens when there's no data has to be inserted by hand: the open and close of each session, a check scheduled five minutes after the data for the day has ended.

AI makes this part much easier. It also remembers my dead ends: when I propose fitting coefficients to the last two weeks of data, it now points out that we've already established this doesn't work.

Execution

When the decision module says "buy the spread at this price", production sends orders and something happens. The order fills, partly fills or misses. The market moves. I wanted to buy at 10 and paid 10.05, because one leg filled and the other ran before I could complete it.

The simulator has only history, so it has to estimate all of this. That estimate is the execution model. The obvious shortcut is to calibrate an impact model on public market data. A 2025 study shows where that leads. The researchers fitted both linear models and a neural network to Nasdaq order book data and used them to simulate a large order. The predicted price paths looked nothing like real executions: the price rose in an almost straight line during the order and barely reverted afterwards.

The sequence I use on a new instrument is fixed. I start with a naive execution model, launch at small size, collect about a month of orders and trades, and calibrate the model on pairs of signal and outcome: what the strategy asked for, and what it got. My current model took a year to grow from the naive version. It gained a lot of features along the way, most of which were later dropped. For a new pair, say, LME lead against Shanghai lead, I can set the starting parameters from similar pairs, and they will be roughly right. After 100–200 real trades I'll revise them, and only then does the simulator gain credibility on that pair.

How fast the answer arrives

HFT is more expensive to build and easier to judge. An HFT strategy should make money nearly every day. If it loses two days in a row, you can switch it off. A week at small size produces enough events to give you an answer.

A medium-frequency strategy gives no such answer. With an annual Sharpe ratio of 2, which is a good strategy, any given month has a better than one-in-four chance of ending in a loss, and two consecutive months in the red are unremarkable.

So at what point do you switch it off and go back to research? A week tells you nothing. A month barely counts. You can keep testing indefinitely, and the time between launching and realising something is wrong is simply too long.

Figure 4. The chance that a strategy with positive expected return shows a loss over a given window. At an annual Sharpe of 2, one month in four and one quarter in six are losers even when nothing is wrong. Computed under a normal i.i.d. assumption; real returns have fatter tails.

My own path shows the problem. The first version went live and lost money straight away: in early 2026 I ran into silver’s most volatile day on record, in a market people have been trading for centuries. I rebuilt the strategy and it started making money, then lost again on oil. 

In both cases the strategy had looked like it worked for weeks beforehand. That's the typical medium-frequency pattern: three months of results that look fine, and then a shock, a 20% move on the news, which is what you actually needed to know about.

Detecting regime changes in advance is another philosopher's stone. I don't know anyone who really nailed it. What I use is primitive: when volatility spikes, the strategy closes its position and sits out. It doesn't know which way the market will go, only that something has changed. In effect it's a stop-loss. No robot, however modern, can weigh world events properly. The probability distribution itself keeps changing, and you're playing against other people.

Why I wrote my own simulator

The platform backtester we had was built for HFT. It models queue position and microstructure in detail. For a strategy that trades about once a day, none of that matters. What matters is running three years of history in a minute and a half instead of three days.

My strategy makes a decision once a minute. In production, spending 100 ms on each decision costs nothing: a tenth of a second out of a minute changes little. In the simulator the same 100 ms is fatal. Mine takes about 1.5 ms per step, which lets me run around 30,000 simulations a day on our Slurm cluster. At 150 ms per step, the same work would take a hundred days. That's the difference between a day of research and three months of it. The simulator can't be written in Python and has to be optimised as far as it will go.

I also tried the holy grail: simulating the exchange itself, with agents trading against each other. I built simple agents that bought and sold at random, and others that behaved like a crowd, borrowing the Ising model from magnetism, in which spins align as temperature falls. The crowd degenerated almost instantly. After a little trading the price would shoot up or down a thousandfold, which doesn't happen in real markets. I no longer think it's worth the effort.

What to do about it

These are the rules I'd give someone building an MFT strategy today.

Match the data window to the holding period. For intraday and medium-frequency trading, two to three years is enough and more is harmful. Strategies that hold positions for months and rebalance quarterly depend on economic trends and geopolitics, and they may justify longer history. Even there I'm sceptical: it's hard to see what the 1990s or 2008 can tell you about the current market run largely by machines.

Treat the backtest as a filter. A backtest is good at killing ideas and bad at proving them. Arian, Norouzi and Seco tested common out-of-sample methods on synthetic markets where the true answer is known, and standard walk-forward testing did worst at catching overfitting. Novy-Marx and Velikov show how cheap a convincing story has become. They mined more than 30,000 candidate return predictors, kept the 96 that passed strict tests, and had language models write hundreds of complete academic papers about them, each with its own theory. Modern neural networks make this worse. They're very good at finding structure, including in random noise.

Keep size small enough that the price doesn't react to it. A backtest assumes your orders don't move the market, because your orders aren't in the historical data. In reality they do. Start smaller than the backtest suggests, and increase size only when production fills show that the price doesn't move against you. 

Treat small-size production as a stage of testing. Launch on a naive execution model, collect a month of fills, calibrate, and only then trust the simulator on that instrument.

Check the data before the model. Find out which timestamp your vendor gives you, measure how often the historical data matches what production received, and remove prints nobody could have traded. A simulator that beats production is usually seeing the future.

Decide the stop rule before you launch. For HFT, a couple of losing days. For MFT, something that doesn't depend on the P&L looking bad, because at a Sharpe of 2 it will look bad regularly. Mine is a volatility trigger that takes the strategy flat.

Alpha or gamma

Serious firms invest less in finding new alphas than in testing whether the ones they already have make money. I think they know that miraculous alphas don't exist. For a medium-frequency strategy that ability is a simulator that behaves like production. The only way to build one is to trade live, at small size, and calibrate against what happens.

Without that simulator you can find any number of alphas. The question is how you know that what you found is alpha, and not gamma, or simply the market going up. My next model goes live after the week-long break in Chinese markets. The simulator's numbers for it look almost too good, and I don't believe them yet. Production will tell me within a few months whether I should.

Sources

1. S&P Global Market Intelligence. Prices decline as US 50% tariff excludes refined copper, August 2025. https://www.spglobal.com/market-intelligence/en/news-insights/research/prices-decline-as-us-50-tariff-excludes-refined-copper

2. International Monetary Fund. Advances in Artificial Intelligence: Implications for Capital Market Activities. Global Financial Stability Report, Chapter 3, October 2024. https://www.elibrary.imf.org/display/book/9798400277573/CH003.xml

3. Korea Capital Market Institute. Short Selling Resumption in Korea: Regulatory Reform and Market Implications. April 2025. https://www.kcmi.re.kr/en/publications/pub_detail_view?syear=2025&zcd=002001017&zno=1840&cno=6517

4. Sato, Y., Kanazawa, K. Strict Universality of the Square-Root Law in Price Impact across Stocks: A Complete Survey of the Tokyo Stock Exchange. Physical Review Letters, 2025. https://arxiv.org/abs/2411.13965

5. Naviglio, M., Bormetti, G., Campigli, F., Rodikov, G., Lillo, F. Why Is the Estimation of Metaorder Impact with Public Market Data So Challenging? arXiv, 2025. https://arxiv.org/pdf/2501.17096

6. Arian, H. R., Norouzi Mobarekeh, D., Seco, L. A. Backtest Overfitting in the Machine Learning Era: A Comparison of Out-of-Sample Testing Methods in a Synthetic Controlled Environment. Knowledge-Based Systems, 2024. https://www.ssrn.com/abstract=4778909

7. Novy-Marx, R., Velikov, M. AI-Powered (Finance) Scholarship. NBER Working Paper 33363, 2025. https://www.nber.org/papers/w33363