IQFeed, CME and TickID

I am sorry, I had a 14 hours working day and at the evening when I did a quick posting of the problem, my brain was exhausted. OK, let me try again. :slight_smile:

First of all, this with missing Tick Volume (the screen capture is from a 1-Tick Chart) happens just after a backfill is starts.

I just started up AmiBroker at 6:50 AM Singapore Time, before the Futures Market has opened for today and it seems that the backfill now was OK with no missing Tick Volume, since there is no live streaming of data.

Then, I tried it again after the market opened (7:00 AM SG Time) and @ES# backfilled as it is my default Ticker- no problem this time. Then I switched to @MES$ at about 18:03:47 (Eastern Time) I got Zero Tick Volume at 18:03:37. I am using my Tape Reader code here, and it shows exactly the same if I do a simple 1-Tick Plot of the Volume as my earlier screen capture.

image

After a backfill I get this:

image

So it looks like at the startup I got one extra line at 19:03:37 with zero volume that disappeared after a manual backfill.

Here another test and this time three Zero Volumes and one tick with 1 Contract with the same time stamps was added:

image

image

The three Zero Volume shows up in both my Tape Reader and a simple Tick-Volume plot. But after a manual backfill, I now got this:

image

image

As we can see, the actual Volumes at 18:43:43 are four fills with 1 Contract each and with the same time stamps. Starting with 1 contract moves the price up one tick (dark green background)

So something strange is going on and as I mentioned this only happens a few seconds after the startup of AmiBroker and the backfills starts.

I hope this is enough information. If not, please let me know since it's easily replicated.

Thanks and Regards,

Jorgen

I did not have time to test it and did not reproduce it yet, but doing thought experiment, I don't think this is particularly strange. This is most probably the side effect of the fact that IQFeed unlike say QCharts, for example, has separate server and connection delivering real time stream and separate delivering backfill. At start you have to send TWO requests: one for real-time stream and one for backfill. The responses are NOT synchronous. Now depending which one responds faster you may have that one second with "missing" data due to inequality of response time. The other possibility is your computer real-time clock being slightly off from IQFeed clock. Neither of two things (discrepancy of response time between servers and/or computer clocks) are controlled by AmiBroker. Trouble does not exist in 1-minute mode, but in case of ticks there can be several dozens of ticks in a second and slight differences in time may be represented by couple of ticks.

I will look at it but the solution for now is to wait a moment and backfill then.

3 Likes

Noted Tomasz! :smiley:

Thanks and Regards,

Jorgen

As far as I see Tomasz, your thought experiment is correct. I don't see this problem when I start up AmiBroker when the exchange is closed! Only when there is a real-time stream active.

Regards,

Jorgen

Not sure what's going on with the IQFeed servers, but this morning one hour after the Exchange Opened, even a manual backfill couldn't remove the zero Volumes! Took many attempts to get a proper backfill.

Regards,

Jorgen

Could be related to this..

@Wallgren IQFeed, as any other data source or any other thing on this planet, is not flawless. Data errors are fact of life. Report data errors to IQFeed, as when something is present on THEIR servers no-one else can fix that.

Hi Tomasz,

The data from the IQfeed Server was correct, but as you suspect due to the responses are not synchronous the plugin gave the zeros. After several backfills, the zero Volumes where gone.

Regards,

Jorgen

No. Don’t make theories because you don’t know how plug-in works. A single forced backfill is enough. It deletes everything you have and what you get is data straight from IQFeed, so if you seen zeros, the zeros WERE in IQFeed servers. Subsequent forced backfills do the very same thing. If outcome is different, data were changed at the source. Receiving end did not change. I know it. It is completely different unrelated story from initial, partial backfill, that does not delete existing data. period. I know how it works. You don’t. Just don’t make assumptions, assumptions are not knowledge, they are fantasies and pretty much every time are wrong.

1 Like

OK, here is exact what happened. I started AmiBroker which load @ES# by default and I got several Zero Volume ticks. I waited it to finish the backfilling. Then seconds after it was done, I did a forced backfilling. But I still got a couple of Zero Volume Ticks! So like 10 seconds after the 2nd backfilling was done, I did a another forced backfilling and this time it was OK.

So what you are saying is that that IQFeed Server first gave me Zero Volumes and then minutes later updated their data and gave me the correct data? I am a simple man with limited knowledge and don't know much about this, so I just say OK then.

Regards,

Jorgen

AH- same thing happened today when I started up AmiBroker! ES and MES only required 2 forced backfills. But for NQ I had to do 6 forced backfills to get rid of the Zero Volume Ticks.

As I mentioned earlier, I don't know much about this- but when the Zero Volume ticks shows up at different times and in different amounts, with NQ I experienced 1 up to 8 Zero Volume ticks, and since I finally could get data with NO Zero Ticks, I personally seriously doubt the problem is on the IQFeed Server. But I don't know much, so who knows?

Jorgen

Hi @Wallgren
According to Tomasz explanation, it seems quite improbable that the problem is on Amibroker side: I doubt an experienced coder such as Tomasz would introduce a bug in such a straightforward processing (but it's an assumption :slight_smile:) It's hard to imagine that in a repeated processing the same action could lead to different outcomes. So it's probably due to a problem in IQConnect or feed (I suspect a cache problem in client). If this annoys you, I suggest you report it to IQFeed forum or technical support.

Regards

https://forums.dtn.com/index.cfm?page=forum&forumID=11

https://iqhelp.dtn.com/contact-us/

That is an interesting reply alligator, that in my opinion, adds absolutely no value to the TickID Consolidation IQFeed plugin Beta Testing! You are just showing up to collect brownie points, instead of actually help with the testing since you could easily replicate this! Since you obviously think that I am some newbie that do not know much, let me remind you of the fact that I found the problems that has so far been identified and fixed! OK?

Earlier I had to work hard to prove "the error that moved around", since no one believed me! But I did not give up and I continued until it was proven being a problem and it was fixed. The same thing I will do now and you can help by replicating the problem.

So if anyone here would be so kind and help, then you can use this code that empottasch wrote and used when he did a quick test yesterday. Put this code in a 1-tick chart:

SetBarsRequired( sbrAll , sbrAll );
Plot( Volume, _DEFAULT_NAME(), IIf( C > O, ParamColor("Up Color", colorGreen ), ParamColor("Down Color", colorRed ) ), ParamStyle( "Style", styleHistogram | styleThick, maskHistogram ) );
PlotShapes( IIf( Volume == 0, shapeUpArrow, shapeNone ), colorLightBlue, 0, V, 15 );
Title = "N zero volume: " + LastValue( Cum( V == 0 ) );

empottasch also got the initial zero volume ticks, but so far not the forced backfill zero volumes. But he did not yet do a comprehensive testing.

Details to keep in mind while testing:

  1. The initial zero tick volumes, and at forced backfill, does not happen all the time. You can have many backfills without any problem.

  2. It seems that this problem is about exact timing! Basically the AB request for backfilling has to come exactly at the same time streaming data is received.

  3. As I mentioned, you have to try many times to get what seems to be all about the exact timing! Yesterday, I tried forced backfill on NQ twenty times, and I got zero volume ticks 4 times but this was during ETH. So it's probably a good idea to try it during the US Open.

So I would highly appreciate if we could focus on the IQfeed Plugin testing, instead of trying to prove that I am some nitwit.

Thanks and Regards,

Jorgen

P.S. I was sarcastic in my previous mail about not knowing anything. :smiley:

1 Like

hi, in my case I tested the code Jorgen posted (in the previous post). So this code I put in a floating window with the "interval lock" ON in the TICK "timeframe" and the window also uses "symbol link" ON.

Initially I found no zero volume bars with all the IQFeed symbols I have loaded (18 in total) . I usually start up the IQFeed database without doing a forced backfill. I let the backfill go automatically when a symbol is selected.

After that I did a forced backfill for @NQ and it got 4 zero volume bars which were adjacent to each other. I tried a "refresh all" but that did not do nothing.

After that I did a forced backfill again for @NQ and these zero volume bars disappeared.

So when the market opens on Monday it could be easily tested.

1 Like

alligator- first of all I want to apologize for overreacting! I am sincerely sorry about that! Reporting problems is such struggle, that I didn't expect that type of a message when I am trying to show an actual problem and can't gain any traction.

Then thank you very much empottasch!!! :smiley:

So now I am not the only user that got Zero Tick Volumes after a Forced Backfill! At least that proves my point, so I am happy now.

If you do the backfill during RTH and especially during the US Open, you most probably will get Zero Tick Volumes in several of your 18 tickers. I know that speculations and theories is really not appreciated, but my observation is that during ETH it doesn't happen that often and I guess because it's not as much streaming data that comes in at a fast and furious speed. I will do some further testing during the RTH on Monday!

I also want to highlight that I am not saying it's a bug in the IQFeed Plugin, but as earlier it could be something with the IQFeed data that is causing this and hopefully it can be handled in one way or another.

Thanks and Regards,

Jorgen