How to optimize backfill with IQFeed

[traderlabs]

Hi,

I have a python script downloading 750 instruments from IQFeed (1 month history of 1 minute data) , store data in database, compute statistics and update instrument with latest data and statistics. This take about 25-30 seconds. So I can scan this list every few minutes.

On the same list of instrument, if I do a backfill of the data to explore in AB, it takes 13 minutes. How could I optimize it to get similar time ?

Stephane

[traderlabs]

Reducing the number of data to 1000 doenst look to improve and AB is repetitively backfilling the data.

Is there a way to quickly keep data updated and then having explore working on data from database ?

[Milosz]

... AB is repetitively backfilling the data.

I don't use IQFeed, so I might not be able to help you, but that seems very strange. Have you enabled Local data storage in Database settings?

First/initial Scan/Exploration might take a lot of time but when it is completed and the database is updated, next scans should be performed much more quickly.

You might also be interested in some issues raised in this thread:

Architecture for taking Amibroker /IB live to large universe of stocks on intraday data?

Hi, I was curious if anyone out there had good architectural solutions for trying to adapt Amibroker to actually automatically scan and trade all 6000 nyse/nasdaq listed stocks - and how to run multiple trading systems on that? It would seem like the actually opportunity to continually scan and place orders run into several different types of limits. I've got a few thoughts, but I was curious if the community (Bandy, Thomasz) had good solutions. In the past I've tried to port systems developed …

And there is a special IQFeed category on the forum. You might find a possible solution here:

http://forum.amibroker.com/c/data/iqfeed

Regards

[traderlabs]

Thanks @Milosz. Yes I do have the local data storage activated, but it seems to be refetching data every time I scan. I have activated option ‘wait for backfill’ but I think it will only get new data. And as I sent to support I have got also a weird behavior on the last bar, so maybe there is something wrong with my setup.

Thanks for the links I will give them a look now.

[Tomasz]

Registered (purchased) version of AmiBroker does NOT refetch data that was already downloaded before. But for that to work database saving must be enabled.

@traderlabs - we don’t have your email in registered users list.

Saving is disabled in TRIAL / unregistered edition.

The only way to “optimize” it is to buy the program.

And you also need to make sure that you have run FULL SETUP of AmiBroker instead of copying files from somewhere. If you fail to use full setup, your database may be READ ONLY (i.e. blocked by Windows User Access Control “feature” from writing).