Tick Data Backfill

I have a subscription to IQfeed and for some reason, I am not seeing the expected amount of data in a chart after backfilling. I am using tick data with the attached settings. As of 1/3/2024, the earliest tick data is on 12/1/2023, I am expecting the data to go back to July as IQfeed allows for 180 days of tick data. I talked to IQfeed support and they said that I am having an Amibroker problem. How do I fix my issue? Thanks!

image
image

Jeff

it depends on the symbol. You have to check how many bars (ticks or trades) there are in a day and compare this with your "number of bars" in the database settings. I use 3e6 bars (3000000) which gives about 10 days ES-mini tick data. Not sure if 10 days is exactly correct but more or less 10 days. But for instance QPA# (Palladium) has many more days for the same settings because there are fewer ticks (or trades) per day

I thought I replied to this but I guess not. The number of bars downloaded reaches at the highest 8,000,000. Even when I move the bars from 10 million to 20 million, nothing changed.

i know IQFeed has 6 months worth of tick data which can be called (the last six months). I am not sure if this can be called in streaming mode. If you go to the IQFeed Connection Manager logo (bottom right) and do a right click , then display Apps, Time&Sales it seems to suggest that 5.000.000 bars is the max. Not sure. Yes I think this 6 months should be able to be called if you can load enough bars. I think the data is a mix of streaming and historical data. So I am not sure what is the problem

Maybe someone else can answer

Thanks for the reply and your screen clip. From what I understand, backfill should fill the rest of the data. Here's the message I got from IQfeed regarding the number of days of tick data I should see:

"Dear Jeff,

Thank you for your contact.

We store 180 calendar days of TICK data for all the symbols we carry.
With that being said, Amibroker must have a setting to request that specific amount of days.

If you have issues with the Amibroker backfill function, our best advise it to contact their support team and I am sure they will be able to help.

Kind regards,
"

inside Amibroker there is also a max number of bars built in. You can increase this in the registry:

Typically to IQFeed staff, they don't know their own service. Plus they know absolutely NOTHING about AmiBroker. Their staff changes so often that every time you reach them someone new comes up apparently again "learning" and all you get is canned response having no connection to reality.

The truth is that they advertise something yet deliver something else. They have many limits that are NOT DOCUMENTED and their own 1st level support staff does NOT know about. Only when you actually try, you find out that there are "hidden" limits. And when you contact some API developer staff, they tell you "yeah we limit that", but they don't mention that in their advertising. For example backfills depend on TIME OF THE DAY. During market hours backfills are severely limited. Especially tick backfills are limited. You must backfill OUTSIDE TRADING HOURS. If you try during market hours you will get much less data. The way how IQFeed works depends on symbol, time of the day and your luck. You might also need to upgrade IQFeed Client because older versions have trouble with long backfills.

As you can see in the screenshot sent by @empottasch their OWN APPLICATION limits requests to 5 million, so there is absolutely NO WAY to verify their claims about data length. Maybe they should fix their own stuff first before they make claims about other apps.

AmiBroker by itself (and it WAS TESTED and verified in real-life) can have upto 50 000 000 bars per symbol (that is 2GB contiguous file). Please note that although it works, it is not really good idea to use that many bars for reasons explained here: Performance tuning tips (scroll down to the end).

Also note that actual number of bars totally depends on DATA SOURCE, not AmiBroker. All AmiBroker does is to MAKE REQUEST for certain amount of data. Whenever DATA SOURCE responds with that many or other amount TOTALLY depends on data source, nothing else.

It works like "hey data source please give me 100'000 bars, then after 2 seconds response comes and data source delivers 20'000 bars" and that's it. You can't force data source to do anything what it does not want to do.

1 Like

I appreciated your candid and thorough response Tomasz!

Thanks for all that. I did change my registry to include that and the problem still exists. I will be contacting IQfeed soon.

I called IQfeed and of course, the technician blames Amibroker or my settings in Amibroker and told me to try using the data outside business hours which I have already tried and now I'm stuck in the middle. Is there any other setting that I might have messed up that would be causing this issue? I'm pasting some more settings in case I missed something.

Thanks for all of your help!


I just tested. I have my Regedit setting at 5e6 (5 million), see figure. Then I had my database settings (inside Amibroker) set at 3e6 (3 million) bars.

using this code on top in the tick chart showed exactly 3.000.000 for the symbols @ES# and @NQ#.

SetBarsRequired( sbrAll, sbrAll );
"cum: " + LastValue( Cum( 1 ) );

Then I changed the database settings (inside Amibroker) to 5e6 (5 million, 5.000.000). Did an exit of Amibroker and restarted. And indeed I do not get the 5.000.000 bars for @ES# and @NQ#. I get a little over 3.000.000 bars. Then I went out Amibroker and restarted it again. For @ES# it shows 3.001.092 bars. So I basically can confirm what you are saying.

Just did a force backfill of QSI# and Amibroker says it backfilled 5.000.000 but I only see a little over 3.000.000 bars in the chart. So maybe this has to do with the recent change of the plugin that ticks are clustered together into trades. Ticks with the same tickID are put inside 1 single bar.

So for instance below some raw tick data for @ES#. They all have the same tickID and after the last update of Amibroker they are put inside a single bar because it is 1 trade. So maybe this is what is going on. Meaning, Amibroker backfills 5.000.000 ticks but after clustering the ticks inside single trades there are less bars left in the chart.

2024-01-04 10:20:32.620470,4754.25,1,468844,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,2,468846,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468847,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468848,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468849,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468850,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468851,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,3,468854,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,2,468856,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468857,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468858,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468859,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468860,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468861,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468862,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468863,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468864,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468865,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468866,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468867,4754.00,4754.25,27940013,C,43,01,1,4
2024-01-04 10:20:32.620470,4754.25,1,468868,4754.00,4754.25,27940013,C,43,01,1,4
1 Like

Thanks again for your help. I have my regedit value set to 25,000,000. Then I switched my "Number of bars" in my Database settings to 25,000,000, from 20,000,000. I forced a backfill and checked the beginning date on my chart and it stayed at 12/5/2023. I also videoed the backfill and it stopped at just over 10,000,000 bars.
image

Yes, IQFeed sends duplicate ticks and, as per USER REQUEST, the plugin combines "same tickID" ticks, see: IQFeed, CME and TickID

Therefore even though AmiBroker asks for say 5 million ticks, about 2 million of ticks have duplicate tickIDs and you would end up with 3 million actual non-duplicate ticks. Read above thread for details.

The work that is supposed to be done on IQFeed side is done on plugin side because IQFeed thinks it is good idea to send duplicates.

As I wrote, the plugin simply asks for whatever amount that is entered in the "number of bars" box, but it is totally OUTSIDE AmiBroker control how many ticks are actually returned and how many of them have duplicate tickIDs.

So if you ask for 20 million you might receive 15 million out of that 5 million would be duplicate tickIDs and you would end up with 10 million.

But that is ALL IQFeed problem that they
a) return duplicate TickIDs
b) return less history than advertised

And of course, if you change the "Number of bars" to higher value you MUST do "Force backfill" because by default, the plugin, to conserve bandwidth, will NEVER request old data that it already has in the database and would only fill missing data from "last present" tick to current time.

And please do send me the address, first and last name of IQFeed support representative because I need to tell them to stop spreading the nonsense.
They are getting paid to deliver data in the way customers want yet they fail to deliver non-duplicate data.

What IQFeed plugin does is issue IQFeed API command

HTX,[Symbol],[MaxDatapoints],[DataDirection],[RequestID],[DatapointsPerSend]

MaxDataPoints is set to whatever you enter in File->Database Settings, "Number of bars".

And that is ALL what plugin can do. From then it is totally upto IQFeed to respond to HTX request with as many data points as specified by MaxDataPoints. Note that the name is "MaxDataPoints" which implicitly means that it is MAXIMUM, not actual number of ticks that you would receive. And many of ticks returned would have duplicate TickID.

2 Likes

Tomasz,

I don't know the name of the original guy that I talked to.

Today, I talked to another IQFeed technician and he mentioned that I could possibly speak to one of the developers on Monday and explain my issues to him.

Given, that I'm ignorant of the inner workings of Amibroker is there any other information that I could tell the developer to make the case that the issue is on IQfeed's side, other than the IQFeed API command that Amibroker uses? Ultimately I would want them to fix the underlying issue, but I doubt they will if they've gone this long with their data restrictions.

Thanks for your help!

Ask him to read this thread

Did you read that thread:

It is CRUCIAL to read that thread.

It explains clearly the fundamental problem that IQFeed sends duplicated unbundled ticks (ticks with SAME TickID), and they clearly said that they are not going to change. The thing is that even if you request 20 million bars, you might simply get less in response and even if you did get them all, even half of them might be duplicated. IQFeed plugin as per user request combines duplicate ticks with same TickID into single tick. As a result you never going to have as many non-duplicate ticks as you requested. @empottasch gave you actual sample of duplicate ticks here Tick Data Backfill - #12 by empottasch

Okay, I read the thread. I'm using Amibroker 6.40.4, so I believe I'm using the IQFeed plugin that does not consolidate. So, it seems that the backfill bar (pictured below) should reflect the number of ticks that IQFeed is sending me, and the number of ticks in Amibroker should also match.

Am I correct in that statement? With that being said, I'm still only getting ~11 million instead of the expected 25 million (ignore the backfill value on the bar below as this was not a forced backfill). From what I can tell, it seems my issue is different than the consolidation issue as mentioned in the link you posted. I just want to make sure I have my facts straight before talking with IQFeed again.

image

AmiBroker version does not dictate IB plugin version.
Plugins might be different versions with no relationship to AmiBroker version.
To find out what plugin do you use you need to go to Tools->Plugins.