Volatility in optimisation results

AmiBroker
1

hi all
i am using the optimize function for testing /optimising a trading strategy varying lenght of moving averages for a close price.

i am using free data from yahoo which goes back 20 years. However results of optimisation have huge variances from one increment to the next (eg using 70 day MA vs 80day MA) which seems extremely counterintitive. My strategy is relatively simple so wasnt hoping to see that level of variance.

Could this be caused by Yahoos incorrect data. I am using yahoo's free data and running teh test on S&P500 stocks list which i have in a .tls file. Everyone says its better to invest in paid data like norgate, will this address my optimisation result issues? not sure what is the incremental value from teh paid data.

2

@sydneysider, when backtesting a trading strategy on an index like the S&P 500, Nasdaq 100, or Russell 2000, it's essential not only to have historical price data, but also to know which stocks were actually in the index at each point in time.

This is where Norgate’s "Historical Index Constituents" series becomes extremely valuable, and it's fully supported in AmiBroker via their plugin and associated functions.

Index membership changes frequently: companies get added, removed, acquired, or delisted. If you’re backtesting using only today’s list of index members (as you likely are doing with Yahoo data), then you're testing with a distorted view of history. You may be including stocks that weren't even in the index during your test period, and excluding others that were part of the index but later delisted or went bankrupt.

This introduces two major problems:

  1. Survivorship bias: by only including stocks that still exist today, you ignore the "losers" that dropped out of the index, often due to poor performance. This gives an artificially positive view of your strategy’s effectiveness.
  2. Constituency bias: even if your price data is complete, if you don’t know which stocks were in the index on a given date, your backtest may use an incorrect universe testing on companies that were not eligible at the time.

Norgate solves this by providing special lists of historical index constituents, which can be used directly in AmiBroker. These allow your strategy to operate on the correct universe for each point in time reflecting how the index actually looked then, not now.

You can read more about this feature in Norgate’s documentation, or search this forum for examples using the keyword NorgateIndexConstituentTimeSeries

To be clear, using free Yahoo data is perfectly fine while you're still learning how to use AmiBroker and exploring initial ideas. It’s a great way to get familiar with the platform and test out basic strategies.

However, once you move toward deploying strategies with real money, this kind of data upgrade IMO is no longer a luxury: it’s a necessity.
Without historical constituent data, your backtest will likely be biased and overly optimistic: great for your ego, but dangerous for your capital. If you're serious about systematic trading, accurate and complete data is a good investment.

4 Likes
3

Thank you , I think I will make the investment and buy historical data for both US and Australia stocks to improve my backtesting robustness.

From an ongoing update perspective, do you have any suggestions if I should subscribe to EOD updates as well or is that something which I am better off saving money moving to InteractiveBroker platform who apparently have plug-ins for EOD data into Amibroker.

Norgate will cost me $50-60 a month for ongoing updates so just assessing if its worth it.

4

As @beppe said,

You get what you pay for. If you are serious about your trading and robust back testing then you need to subscribe to a EOD data provider such as Norgate. When you subscribe to the EOD data it comes with historical data.