Force backfill of IQFeed

Data Sources IQFeed
1

Hello,

I have recently started using IQFeed as a data supplier for the London Stock Exchange. I only use the data for EOD bars and scans.

I have my database setup as shown on this Amibroker page (https://www.amibroker.com/guide/h_iqfeed.html), the only exception being that under Intraday settings I don't allow mixing of EOD and intraday data as I don't use the RT quote feature.

One of the things that I can't figure out is how to have my local data storage updated for stock splits and bad data points as they are adjusted for by IQFeed.

By way of example - I have found several cases where there have been bad data points or incomplete histories in the IQFeed database. To IQFeed's credit, when I have notified them of these errors they have been reviewed and corrected within 24 hours. I can see these updates have been done and are available using the "Time and Sales" app supplied by IQFeed.

But in my database I need to "Force backfill" to refresh my data. The process of creating a simple scan as shown on this page (https://www.amibroker.com/guide/h_rtsource.html) and running it across all the symbols doesn't update the historical data stored in my local database. I can only achieve this by "Force backfill" on the individual symbol. I have read elsewhere on the Forum that the "Force backfill" deletes the existing data of the symbol and then re-downloads it from IQFeed.

My question is: can I "Force backfill" somehow across hundreds of symbols at once. The reason I want to do this is to make sure that my local database is current and up-to-date for stock splits and any other adjustments that may be made over time by IQFeed.

It is not practical to this by hand with hundreds of symbols in the database. Can someone suggest how this may be achieved across all symbols on a regular basis (say weekly), hopefully while maintaining my symbol watchlists, ie: not deleting all the symbols and reloading them again.

I have found several similar Forum posts but they seem to either stop short of addressing how to do a bulk forced backfill, or relate to updating the most current data while the computer was turned off or similar.

Thanks in advance for the help and responses

2

You don't need Force Backfill at all.

It is 'last resort' total wipe + fresh download only needed in very very rare cases when data are somehow corrupted.

But you need to keep in mind that IQFeed delivers data ON REQUEST so data are first REQUESTED and not delivered immediately but after few seconds when IQFeed servers respond with potentially megabytes of backfill data. This takes few seconds and AmiBroker does not "lock". It goes on with Analysis using data that are already stored locally, the requests for new data are queued and data will be delivered when your internet connection bandwidth allows (IQFeed can easily saturate it).

If you want to WAIT for data, so Analysis / Scan waits until data are delivered and uses new data to produce scan, you should be using "WAIT FOR BACKFILL" option.

Also normal backfills always requests few bars back so any bad tick fixes should already be in place.

It is all explained in Knowledge Base:

https://www.amibroker.com/kb/2014/09/23/how-to-backfill-all-symbols-in-rt-database/

3

Thanks Tomasz.

I think the problem may be that I'm not using the service as a true RT supplier, but rather relying on performing the simple Scan (taken from the the previously linked website and with "Wait for Backfill" turned on) to fill the daily EOD data.

I have done that process again today and not all of the symbols have backfilled today's EOD data, even though I know the data is there from the IQFeed price app. I have run the scan several times to make sure it wasn't an internet data supply issue with no change.

I also had some data points checked today and corrected by IQFeed and again the data is corrected in the IQFeed price app but won't update in my database without a manual "Force backfill" action.

I also tested the approach in your link by manually deleting some data points from a month ago in a large LSE listed symbol and then running a scan described by Amibroker and the data points are NOT replaced - the hole in my data remains.

So I don't understand how my local data will ever be updated and adjusted for splits or error corrections in the future.

One way around this may be to use Amiquote to access the EOD data from IQFeed. I can't find any information about this on the Forum/Amibroker website, and IQFeed technical team don't know of the URL to enter to access the data/CSV file.

Can anyone help with this or maybe has Amiquote and IQFeed running already?

Thanks again

4

The instructions are to be followed. You should configure IQFeed exactly as presented in manual https://www.amibroker.com/guide/h_iqfeed.html
If you are not following instructions you are asking for trouble.

IQFeed is not "EOD" data source. EOD data source is Yahoo Finance (AmiQuote).

5

Try using the Tools-> Database Purify.

Use an index as a reference symbol. This should pick up on splits. I generally cross referenced to some news of a stock split. I’d just manually force backfill for that symbol. Don’t use IQFeed now as my work schedule doesn’t provide any benefit to the RT Data.

The problem is that IQFeed doesn’t advertise on itself as a historical data provider. They provide it, but if AMibroker constantly requested IQFeed data backfills/reloads you’d violate their API rules from what I understand.

6

Thanks for the replies.

I ended up creating a dos batch file to delete the applicable database directories which will run once a week. Then on the next database load the historical data will be filled with any corrections/splits made by IQFeed.

Seems to be working well at present.

Thanks again to all for the help

8

Interesting thread as I'm also having issues with DTN. I followed the instructions outlined and I know that one is not supposed to Force Backfill, but for some reason data is missing after a while of running tests.

As an example, I forced all data for symbols yesterday, today after a few scans & backtests the data is no one gone even with "wait for backfill checked"

When I ran a backtest/scan nothing came up, even though yesterday it had several years of data. I find that even though I'm not supposed to force backfill, if I do too much with Ami (too many scans backtests) the data disappears and I have to force them all again, quite time consuming. I read a previous post that someone said you can automate the process with a scan of Buy = 1; but that doesn't work either.

Any idea of what is wrong or a way to automate backfills? I did follow https://www.amibroker.com/guide/h_iqfeed.html exactly and even have a lower amount of data than the limit but still having issues.

image
image

9

Backfills are automated out of the box. You should not be doing Force backfill. If you do subsequent requests are QUEUED because there can be at most 4 running backfills at a time in IQFeed plugin. If you overdo things - you can wait quite long for queue to be processed.

10

This note is for those who think they can circumvent IQFeed subscription limits and use more symbols than the limit (>1300 or whatever your subscription limit is). No you can't. It is prohibited by IQFeed and not supported by us.

Here is what Jay (head of development) from IQFeed wrote on the in this thread:
http://finance.groups.yahoo.com/group/amibroker/message/149624

Quote from IQFeed representative:

"With IQFeed, we don't limit through our API the number of symbols you receive backfill data on. We also don't have a technical limit to how often you can watch/unwatch symbols. This being said, we do have policies which don't allow API developers to
programatically circumvent the symbol limit in which a customer subscribes to.
Switching between portfolio's within the course of a trading day is perfectly
acceptible. Contantly watching and unwatching symbols in order to get around the
symbol limit is not. We have subscription options that go as high as 1800
symbols with DTN.IQ. "

Note that "constantly watching and unwatching" AND backfill of missing data (from last unwatch to "current" time) is exactly what would happen if you scan, explore, backtest, optimize for more than your subscription limit is. So no, it is not supported.

IQFeed people don't understand how modern analysis software works therefore they differentiate between "watch and unwatch" and backfill. In reality to do any analysis involving historical data (chart, scan, backtest) you have to do BOTH (i.e. watch the symbol AND backfill missing data). People at IQFeed / eSignal and other places think in terms of people displaying "charts" manually by typing the symbol (so you display 10 charts that HUMAN slowly looks at so they think you watch and unwatch one symbol per second or slower because you are HUMAN). Or manually changing the "portfolio". Like in previous century. They don't consider things like AUTOMATED computerized scan, exploration, backtest, where you can access 10000 symbols within seconds. And their systems are incapable in operating at those speeds.

Since RT sources are not capable of delivering data at those speeds, we have to slow down requests. If you don't do wait for backfill, the flood of requests gets QUEUED at the plugin level and at most 4 simultaneous backfills are done at a time. More requests are waiting in the queue until previous requests get completed. If you go bananas about requesting thousands of symbols, your request queue will be long and you will wait a lot.

For this reason keep your daily operation within subscription limit. You can have more symbols in the database but over the course of single session you should NEVER exceed the subscription limit.

IQFeed deliberately throttles backfills during market hours (to save bandwidth for streaming), therefore Backfills of more symbols can happen ONLY outside market hours and ONLY with "wait for backfill" turned ON, so it goes one backfill at a time waiting for IQFeed to deliver response, without creating large queue.
AmiBroker is way too fast for IQFeed (or any other existing real time streaming data source) and they are not able to cope with such speed without AmiBroker WAITING for data to arrive. That is why you should use "wait for backfill", it instructs exploration / scan to wait for data to be complete. Without that exploration would go further asynchronously and complete before data are delivered.

2 Likes
11

I just received the notice from IQFeed that they are now implementing new historical rate limiting on their backfill servers. The limit will be upto 50 requests per second.
AmIBroker is unaffected by this change because even without "wait for backfill" we don't do more than 4 simultaneous requests and WAIT for arrival of previous responses before requesting more (so not more than 4 simultaneous backfills are running at a time). Since ping time is typically well above > 20ms you technically not able to do more than 50 historical requests per second that way.

12

As a futures trader using continuous backadjusted contracts (e.g. @ES#C) for charting, is there any way to get the revised historical prices automatically for all contracts, without using Force Backfill on each symbol individually?

1 Like
13

Using Foreign() forces backfill. So it's good enough to rotate through all tickers and retrieve Close array. For instance:

#pragma enable_static_decl “backfill{chartid}”
static i;

tickers = CategoryGetSymbols( categoryAll, 0);
s = StrExtract(tickers, i);
if (s == "") {
	i = 0;
	s = StrExtract(tickers, i);
}
++i;

C = Foreign(s, "C"); O = Foreign(s, "O"); 
H = Foreign(s, "H"); L = Foreign(s, "L");
// Unnecessary, only for displaying something:
s = StrLeft(s, StrFind(s, "-")-1);
Title = s;
PlotOHLC(O,H,L,C, s, colorDefault, styleCandle);

Cheers.

2 Likes
14

Thanks for the clever idea, @alligator. I think your code would be more useful for a backfill on IBKR, because it only triggers a backfill on the most recent bars and the IBKR plugin can be set for a long period like 180 days.

But what I'm trying to do is to replace the entire price history with the revised historical data when the contract is rolled over and backadjusted. It's similar to a stock split where the past history is revised based on the split ratio. In the IQFeed plugin, "Force Backfill" clears out the old data and re-downloads a complete price history for the symbol, and as far as I can tell, this is the only way to get the backadjusted history.

15

Sorry I didn't get it. Actually you want to backadjust your data. You can store your existing data in a var/ticker/file, trigger a backfill, detect gap/delta, adjust the old time serie and save it to a file. Then import the file to update your ticker(s).

16

There isn't coding required and no export/import.

You may use split and merge of Symbol menu.

35

Post from old (gone) Yahoo AmiBroker list by A. Pipa:

Task: Merging contracts.

merge contracts 1

merge contracts 2

17

Good idea, but he wants to do it automatically, not manually so you need a script anyway using Split/Merge with OLE (I also suspect there's a bunch of tickers to adjust.)

18

This old post proposes a solution using AFL+Batch:

It should solve @PeterD 's problem.

19

Hi all,

Thank you for helping and showing me some things I didn't know about how to stitch a continuous backadjusted contract together. However, I already use IQFeed to provide the backadjusted prices automatically, e.g. the symbol @ES#C is the backadjusted version of the continuous @ES, which is the stitched-together version of the front-month ES contract.

It's simpler to just click the "Force backfill" button manually so I'll just keep doing that when the contracts rollover.

Many thanks for your time and attention!