Sudden very slow backtesting and optimization with norgate data

Hello,

I've been using AB 6.30.5 for a couple weeks and all was fine and fast. Now today all of a sudden running backtests or optimization is 10-20x slower than usual.

I'm using norgate data and run the latest updates.

Anyone has an idea as to what can cause the issue ?

Of course i've rebooted my PC multiple times.

Using a local DB it seems like everything is fast as usual so it could be norgate related...

Any idea of what the issue can be?

Thanks,

Looking at the info pane it seems like it's a "data" problem (does it mean a norgate issue?)

Timings: data 74.52, setup 0.01, afl 24.08 job 0.92 lock 0.00 pbt 0.43 ui thread 74.96 worker threads 25/24.99

if I understand the stats correctly it means almost half of the time of this 4 years backtest completed in 163 seconds, is spent getting the data from the norgate DB, am I right?

I already recreated a brand new norgate DB, what else can I try?

thanks,

I'm inclined to think it is Norgate. About 22 hours ago I was unable to download any data with NDU, and following that I was unable to open AB at all - it just stayed at 'Opening Charts'.

At the time I suspected some change in my Avira Prime Anti-vir and spent some time isolating each possibility but no joy. I went away to participate in a sporting activity and when I came back home about 18 hours ago I tried NDU again - it did an app update, downloaded data, and I was able to open AB no problem at all.

I just did a backtest over 25 years worth of data which took all of 20 secs first time around and then only 1.7 secs the second time.

Try downloading data with Norgate again. If no change, are you by any chance using a new .afl file for the first time which may be causing the slowness?

Thanks for your answer.

Like you it seems like the first round using a watchlist after a NDU updates is VERY VERY slow...

and if like me you combine first round with a watchlist like the russell 3K old and present + Optimize 1 parameters with 10 possibles values, it will take forever during the first round.

Looks like the best is to not optimize during the first round, I suspect there is some caching happening behind the scene that slows down AB when it's not cached, not sure if that's on Norgate or AB side.

I'm not using any antivirus so it can't be the problem.

  1. Go to File->Database Settings
  2. Change "Data source" to "(local database)" - you are effectively disconnecting from Norgate.
  3. Run your backtest

Should be back to normal speed.

1 Like

Thanks for the tip.

Actually it seems like the first time an optimization or backtest is made right after a DB update is slow, not sure why, maybe some caching mechanism in the background (?)

But else it seems to be back to normal. I'll just try to avoid running long stuff the first time after a norgate refresh.

This is expected behaviour - an update to the database can be small (last value) or large (historical correction, historical adjustment due to dividend/stock split etc.). Simply comparing the last bar's value won't reveal anything other than the last bar has changed, so, erring on the side of caution, we reload the data.

If you're running automatic background updates, you should definitely disable this prior to doing any sort of backtesting/optimization run. This is easily settable in Norgate Data Updater with Settings -> Update Mode, as follows:
image

After your initial run, all further price data access will be cached by AmiBroker. Make sure you set your Amibroker's cache settings high enough to handle the number of symbols and size of data required (Tools -> Preferences -> Data).

image

3 Likes

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.