Strategy Optimization Times With IQFEED DATA

AmiBroker
#1

Hello All,

I've noticed that when I run a backtest/optimization with IQFeed Data my PC takes considerably longer on daily data (1000 bars loaded - 3 years of data) than it does with norgate's database of the same settings. I'm having trouble understanding why it takes so much longer with IQfeed to run optimization and backtests compared to norgate. I suspect it could be because of the realtime feed that also somehow interferes with the AFL formula when trying to optimize. `maxpos = 10; // maximum number of open positions

SetOption("InitialEquity", 100000 ); // set initial equity = 100K
SetOption( "MaxOpenPositions", maxpos );
SetPositionSize( 10 / maxpos, spsPercentOfEquity );
ShortMA = Optimize("ShortMA", 9, 1, 100, 1);
LongMA = Optimize("LongMA", 21, 5, 100, 1);

FastMA = EMA(C, shortMA);
SlowMA = EMA(C, longMA);
FilterMA = EMA(C, 200);
Buy = Cross(FastMA, slowMA);

Sell = 0;
SellPrice = Close;
ApplyStop( stopTypeProfit,stopModePercent,10, ExitAtStop = 0, volatile = False, ReentryDelay = 0, ValidFrom = 0, ValidTo = -1);
ApplyStop( stopTypeLoss, stopModePercent, 1, ExitAtStop = 0, volatile = False, ReentryDelay = 0, ValidFrom = 0, ValidTo = -1);

Plot(FastMA,"shortMA", colorGreen, styleLine, 0, 0, 0, 0);
Plot(SlowMA, "SLow", colorBlue, styleLine, 0, 0, 0, 0);
Plot(FilterMA, "200", colorRed, styleLine, 0, 0, 0, 0);`

This is the code I am using.

#2

Side note, I am using currently only 20 symbols with the IQFeed data however, with norgate it was loading all symbols from its database for US stock exchanges. The optimization time of this code on 20 symbols with the iqfeed was approximately 2-3 hours~, with norgate's data feed and all of the stocks loaded it was 30seconds~

#3

Have you "Wait for backfill" turned on or off?

14

Otherwise see @Tomasz post here.

If you are into optimizations it is recommended to TURN OFF external data source (plugin) and run only native AmiBroker database ("Data source" set to "(local database)").
You may also want to increase "in-memory cache" setting in Preferences so it is more than number of symbols under test.

1 Like
#4

Thank you so much, also I have a question, for the in-memory cache, is that value per symbol or the cache as a whole? Also, I'm assuming the cache is using ram. Also, one last thing, would changing the AR interval also help as well?

#5

No, it is not per symbol.

Preferences window

  • In-memory cache (max. symbols) - defines how many symbols data should be kept in RAM (for very fast access) - this works together with the next setting
  • In-memory cache (max. MegaBytes) - defines how many MB of RAM should be used for temporary data cache (for very fast access)

In-memory caching

By default AmiBroker holds only 10 the most recently accessed symbols' data in RAM. This takes up about 320 KB (yes, kilobytes) of memory for 1000 bars per symbol loaded. You can enlarge "In memory cache" (Tools->Preferences : "Data" tab) to 100 (approx. 3.2MB additional RAM consumption ) or 1000 (approx. 32MB additional RAM consumption) or even more to get much better performance for subsequent data access (once data are in RAM AmiBroker does not need to ask plugin again and again)

Performance tuning tips

  • The "in-memory cache" size defined in Tools->Preferences, "Data" tab controls the number of symbols in the cache and maximum amount of memory consumed by the cache. While larger cache generally speeds up processing because data do not need to be read from disk and written back all the time, you may run out of memory on certain situations when your cache is too large, especially on 32-bit operating system that limits the available virtual memory per process to 2GB. To avoid running out of memory, go to Tools->Preferences, "Data" and decrease the size of in-memory cache. To set it to minimum, enter "11" (eleven) into "in-memory cache (max. symbols)".
    IMPORTANT: It is advised NOT to decrease the cache when your databases are relatively small (few hundred MB) as larger cache will speed up the access. Large cache is good thing as long as you do not run out of memory.

AR in optimization?
Why don't you turn it off?

1 Like
#6

Thank you so much again, I did turn it off and everything is now working about 3x faster (about 50min for optimization with 9600 steps from 3hours~ for the same optimization). I changed my memory cache to 4000mb from the 512 (for people who reference this post later). This is still considerably slower than norgate data's performance but thank you so much for all the advice you've given me. I still am stuck with the problem of wondering why norgate is much quicker than IQfeed but other than that, the performance tips you've given me have helped immensely.

#7

You are comparing apples to oranges. Norgate is EOD off-line data source. IQFeed is on-line real time streaming intraday.
This means typically you get hundreds of times more data to process in intraday (IQFeed) than you have in Norgate.

This can be compared as counting stars in single galaxy (Norgate) compared to counting stars in entire universe (IQFeed intraday). That's UNCOMPARABLE.

The number of BARS per symbol has the BIGGEST impact on performance. Nothing else comes even close to that.

You really MUST read this:

And for ALL folks trying to optimize their systems:

While you are running optimization Use LOCAL DATABASE or at least make sure that in-memory cache is LARGER than number of symbols you are testing on.

Local database is orders of magnitude faster than ANY external thing.

You can just go to File->Database Settings and change Data source to "(local database)". Recommended reading:

#8

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.