I've got a strange one. I have a system I run a backtest on every day. On May 18th, the system was performing normally - consistent with past runs of the system. On May 19th, suddenly the system results changed radically. The trades were different, and the results were different.
Now, there are no code changes between the two days - I've verified this via Report Explorer, comparing the code between the two systems, and the Watchlist the system runs against hasn't changed.
So I'm at a loss as to how to troubleshoot this. The only thing I can think of is that my data has changed. I leverage the free US stocks data from Yahoo - and that's the only variable I can think of, but I wanted to ask the forum if anyone else saw a radical change in performance between May 18th and 19th leveraging Yahoo data. Any other thoughts on what to look at/how to troubleshoot would be welcome.
@droskill, without seeing what your code operates on and what elements are calculated to determine its "performance" it is impossible to say what the cause is, but it will not have escaped you that May 18, 2022 on the American markets was the worst day since 2020..
So I guess your system is quite sensitive to such significant variations in a single day.
The code is a pretty straightforward rotational system - and again, the code between the days is exactly the same, as is the list of stocks it is applied to. But yet the trades it generated (including past and open trades) are different between the two days. Which is to say that it isn't about the system sensitivity - something materially changed between those two days such that it impacted both past and open trades. It's all just a bit strange.
If the historical trades have changed, then the most likely cause is that the underlying data has changed. However, if you don't use an APX file to save all your settings, you should also verify that things like Periodicity, Commissions, Pad & Align, etc. are the same for the runs on the 18th and 19th.
If that doesn't yield answers, I would compare the trade lists and dig into the differences between them and try to answer questions like these:
Are there many trade differences, or just a few that "cascade" over time?
Do the symbols from the May 18 backtest still exist in the database?
Are the May 18 symbols the most highly ranked in the universe using the current data? Presumably not unless your backtest is really messed up, so could anything besides the price data have caused the ranking to change?
I'm sure it would also be very informative to compare your current system performance to the same AFL running on a different system with either Yahoo data or data from another provider. Obviously not everyone has that option though.
I figured it out - first, thanks for the reply. I do use APX files to ensure consistency in settings, etc.
Here's what I just remembered: On the day in question, I recognized that I didn't have complete history for a number of stocks in my watchlist. So, I ran an update on all the stocks in the watchlist. This obviously pull more history for some stocks, and therefore changed the outcome.
Moral of the story: the accuracy of your system is dependent on having full history of all stocks in the watchlist. Don't make my mistake!
