New AB user and hitting a roadblock on backfilling my database, hoping someone can advise and help. Steps I have taken so far:
Signed up to IQFeed (currently on free trial)
Connected to the IQFeed PlugIn
Created my database for 1min bars and 4 days fill
Imported symbol list from IQFeed
With the free trial, you get 4 days of intraday data so I wanted to backfill this data as a test before backfilling the entire historical data.
The issue I'm seeing is that AB runs the backfill and states 100% complete in the Analysis window but stays in this state all day. I have to manually stop the bar. Maybe this is normal behavior but I'm not sure.
In addition, when I look at the details of the backfilling window, there's a number of symbols at the end of the list which are in 'ready' state and don't change status to 'Old' which all the other symbols do.
Unsure if this means that AB has completed backfilling those symbols or not.
I suspect something isn't working correctly here but as a new user I'm not 100% or if I have made an error with the set up. Any help or advice would be appreciated. Thank you
Your configuration is wrong. Intraday 1-minute database needs A LOT MORE than 2000 bars. It is recommended to use 100K or more (like 500K).
PLEASE DO READ THE MANUAL How to use AmiBroker with IQFeed
What you are showing is NOT progress of backfill but progress of Analysis. Depending on your settings, Analysis may or MAY NOT wait for backfill (check "WAIT FOR BACKFILL" flag.
The plugin shows that there are no pending backfill requests (good). Ready state is good. It means that symbols are up-to-date (NOT old). Symbols will become "old" if you go beyond your subscription limits.
You did not say how many symbols did you add to the database (IQFeed has LIMITS). Especially "free trial" of IQFeed has significant limits
How did you exactly "import" symbol list??? Did you read this part of the manual: How to use AmiBroker with Real-Time data plugins , especially " Things you should NOT do, or you should do very carefully"
It says:
So please do not import ASCII (especially EOD data) into a real-time intraday database fed by the plugin.
You may ask why this is not disabled at all. The answer is that sometimes it is useful, and sometimes it will work (but these are rare cases). For example, it will work if you import INTRADAY data into the intraday database fed by the QuoteTracker plugin, and both the database and imported data have exactly the same bar interval.
It also works if you import the data for symbols that are not present in the database. In this case, newly imported symbols are marked by the ASCII importer as "use only local database for this symbol" (See the Information Window for details), so they are excluded from the real-time update. This is useful if you want to import some other data (even non-quote data) and access it via the Foreign function while using your real-time database.
So ASCII import is not disabled in a real-time database, but you have to use it with extreme care and know what you are doing.
When I'm running the scan I have 'Wait for Backfill' checked from the settings menu.
Ok, understood.
I imported 12,492 symbols. Have also spoken to IQFeed technical support and they said there are no limits on free trial apart from the 4 day historical intraday.
Wrong. I don't know who told you that nonsense at IQFeed, but even fully paid subscription has 500 symbols limit by default. If you pay extra for extra symbols the limit may go up upto 1300 symbols max.
You must not exceed IQFeed limits.
Trial has severe limits. And I mean symbol limits.
And I am not asking how did you extract symbols but how you imported them. Did you check symbol flag that I mentioned?
I mean the quote from the manual I gave before with bold text.
The conversation I had with IQFeed support was a little confusing on this topic. They confirmed that there is a 500 symbol limit but gave no other solutions. Do you know if this applies to backfilling as well as RT data? All I'm trying to do is backfill the past 4 days of intraday 1 minute data.
I used File > Import ASCII and then imported an IGFEEDDB file for the symbols.
Sorry, I'm not following you on the 'check symbol flag' part that you mentioned. Is this referring to 'Use only local database for this symbol' from this link?
"With IQFeed, we don't limit through our API the number of symbols you receive backfill data on. We also
don't have a technical limit to how often you can watch/unwatch symbols. This
being said, we do have policies which don't allow API developers to
programatically circumvent the symbol limit in which a customer subscribes to.
Switching between portfolio's within the course of a trading day is perfectly
acceptible. Contantly watching and unwatching symbols in order to get around the
symbol limit is not. We have subscription options that go as high as 1800
symbols with DTN.IQ. "
So, theory is one thing, practice is another. This is NOT how you think it is. IN PRACTICE, backfill is NOT really separate from streaming, even though technically there are TWO separate API requests to two different servers. In practice, when you want to have CURRENT DATA for given symbol you must do BOTH. You must request backfill data (missing data from last update present in the database) AND you must ask for RT data stream, so backfilled data are kept up to date without needing to backfill them again in a second.
It would be pointless to backfill without keeping data current as backfills are lenghty and costly. They would NOT want you to backfill over and over again.
So when symbol is "old" and you want it to become "current", in PRACTICE you must do both: backfill (to fill missing data) AND 'watch' the symbol (which counts against symbol limit).
So there are official limits on streaming symbols (500 symbols by default) and there are "unofficial" thresholds on backfill. The backfill is "dynamic" meaning that they will throttle if you download too much too fast especially during trading hours. Limits/throttling is relaxed outside of trading hours. This is from hands-on real-life experience, not something written anywhere.
Running scan/exploration/backtest that exceeds subscription limit would qualify for the practices prohibited by IQFeed service mentioned above. If you want to ensure data-quality and no surprises, you need to stay within the subscription limits. Exceeding it is definitely going to cause problems.
I signed up to the free trial with IQFeed to test the backfilling process with the 4 days intraday data that they make available.
I'm reluctant to sign up to the full service and backfill 1 minute data of the entire history for US equities (which would be 12000+ symbols over 18 years) as they impose symbol limits which you have explained above. Here is my latest conversation with a sales rep at IQFeed on this:
Nobody offers 12000 symbols in real time at retail prices. There is no such thing available to retail customers. What you are trying is to get institutional feed for retail money. No such thing exists.
Some reality check is needed. Broadcast service that would bring you 12000 symbols in parallel means receiving upto one million events per second (bid/ask/trades). Even if you processed one even within one MICROsecond (1/millions part of the second), it would completely saturate your computer just to receive that much data.
Retail real-time data vendors don't sell "backfill alone". The closest thing to what you want is TickData offering. TickData.com is selling whole market historical intraday data without real time updates.
Alternative is: make sure to use 500K-900K bars per symbol in the database settings, use "Wait for backfill", "Local data storage" turned ON, use
#pragma maxthreads 1
Buy =0;
in your Analysis and run scan, AFTER trading hours (over the weekend). It will run one symbol at a time, waiting for backfill), because it would proceed slowly and not during trading hours, you would be able to get data for 12000 symbols, but it will take some time. Once it completes, press File->Save Database.
This way you will get intraday data in your local database (that you can even disconnect from IQFeed), without spending huge $$$ on TickData.
