IQFeed sends a tag with the CME tick data to allow the trades with the same 'Tick ID' to be consolidated, saving a ton of space in the database. Would something like this please be considered for a future release @Tomasz?
1 Like
hi, together with another Amibroker/IQfeed user we are working with IQFeed tick data. Data with the same tickID have the same timestamp (most of the time). In Amibroker "bundled orders", orders with the same timestamp and same tickID are serialized.
Problem is that sometimes (it happens quite often) separate trades/orders have the same timestamp. So inside Amibroker we can not see that there are 2 trades/orders. In Amibroker the timestamp will be serialized and therefore (when there is no price change) we will analyse this a 1 single trade while this is not the case.
Here (@ES# symbol data) you see we have an uptick from 273491110, to 273491111. The volume is on the ASK side so we will count this as an uptick order. However, order 273491112 has the same timestamp so our code will think it has the same TickID, or the same trade. So it adds 1 contract too much to this uptick order/trade.
2022-09-23 16:58:10.860026,3710.75,1,2815216,3710.75,3711.00,273491108,C,43,01
2022-09-23 16:58:10.860026,3710.75,1,2815217,3710.75,3711.00,273491108,C,43,01
2022-09-23 16:58:10.880431,3710.75,1,2815218,3710.75,3711.00,273491109,C,43,01
2022-09-23 16:58:10.880431,3710.75,1,2815219,3710.75,3711.00,273491109,C,43,01
2022-09-23 16:58:13.139789,3710.50,1,2815220,3710.50,3711.00,273491110,C,43,01
2022-09-23 16:58:16.168680,3711.00,1,2815221,3710.75,3711.00,273491111,C,43,01
2022-09-23 16:58:16.168680,3711.00,1,2815222,3710.75,3711.00,273491112,C,43,01
So in my opinion there is no other way we can separate orders 273491111 and 273491112 inside Amibroker without having access to the tickID.
So we request for an additional array containing the tickID. 1 thing I need to mention is that also tickID's are not unique since they seem to re-start counting at certain intervals or using some other logic. Not sure what they use since I did not investigate this yet. But per day the tickID seems to be unique so we could distinguish between identical tickID's by looking at the date.
Would it be possible to add the tickID data? And if not why not? Or would it be possible to add this data array via the IQFeed plugin? Or maybe some other solution.
thank you
That is basically flaw in IQFeed data. They report time down to microseconds but clearly their clocks do not offer 1us precision. No two trades should ever have same timestamp down to microsecond, because it is impossible in reality. They post duplicate time stamps only because they don’t measure time with proper resolution.
thanks for your reply. I will look into this. There is a similar question on the IQFeed forum I just found now. So will be looking into this first: DTN IQFeed, DTN.IQ, DTN ProphetX Support Forums
Since it's a flaw in the resolution of the timestamps, the implementation of access to the IQFeed TickID will solve the problem. So is that something that could be done?
Thanks and Regards,
Jorgen
I checked the source of the plugin and if TWO ticks with duplicate timestamps (exact same time down to microseconds), AmiBroker will still CORRECTLY add TWO records in TICK mode, so each trade is correctly represented in quote array.
To handle this situation that should NOT happen (duplicate timestamp down to microsecond level), the plugin makes artificially unique timestamp by adding 1 microsecond to timestamp (in second trade) so timestamps of two subsequent ticks are made unique.
Please also note that reliance on TickID does not solve the problem as it has been said on IQFeed forum, that TickID can be duplicate as well
i will respond a bit later with some examples of where this still would give a problem. I agree it is due to the resolution of the timestamp but according to the forum this is how IQFeed gets the data.
here is 1 example from last Friday:
These are the tick data:
2022-09-30 13:53:56.728893,3637.00,1,1822627,3636.75,3637.00,273918716,C,43,01
2022-09-30 13:53:56.812103,3637.25,1,1822628,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812103,3637.25,1,1822629,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812103,3637.25,1,1822630,3637.00,3637.25,273918718,C,43,01
I am interested in the last 3 lines. We see an uptick at the ASK price. The volume I should assign to this trade is 2. Because the trade consists of 2 order lines. But in Amibroker the data looks like this:
2022-09-30 13:53:56.728893,3637.00,1,1822627,3636.75,3637.00,273918716,C,43,01
2022-09-30 13:53:56.812103,3637.25,1,1822628,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812104,3637.25,1,1822629,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812105,3637.25,1,1822630,3637.00,3637.25,273918718,C,43,01
So I can not distinguish the line with tickID 273918718 from the previous 2 lines with tickID 273918717
So with the data in Amibroker I assume it is the same trade and then I add the last volume to that trade and therefore I will get a volume of 3 while it should be 2.
First how did you produce those listings ? (Formula?)
So I can not distinguish the line
You can, because (as described earlier) in the presence of duplicates, the plugin produced different UNIQUE timestamps for those trades with duplicates coming from IQFeed, and all 3 lines are present in quote array and can be uniquely identified.
i download the tick data files from IQFeed using Python code.
And yes you are correct to say the lines are unique because in Amibroker it will look like this (look at the timestamps of the last 3 lines)
2022-09-30 13:53:56.728893,3637.00,1,1822627,3636.75,3637.00,273918716,C,43,01
2022-09-30 13:53:56.812103 ,3637.25,1,1822628,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812104 ,3637.25,1,1822629,3637.00,3637.25,273918717,C,43,01
2022-09-30 13:53:56.812105 ,3637.25,1,1822630,3637.00,3637.25,273918718,C,43,01
But the 2nd and 3rd line have the same tickID. The 4-th line has a separate tickID. So how do I know in Amibroker that the 2nd and 3rd line belong together (they are 1 trade) while the 4th line is a separate trade. I agree they all have a unique timestamp in Amibroker but because they are all 1 microsecond apart I can not conclude that the 2nd and 3rd line belong to 1 trade and the 4-th line belongs to a separate trade.
Quote from DTN IQFeed, DTN.IQ, DTN ProphetX Support Forums
Gary Stephen DTN IQFeed Implementation Support Specialist wrote:
" there is no unique identifier anywhere in the IQFeed system"
In the light of the quote above, how do you know they "belong together". I am not so sure.
Also another quote from the same guy:
Furthermore, TickIDs are not necessarily unique by trade. Futures exchanges and OPRA can have the same TickID applied to multiple trades.
i will respond a bit later. Will ask some question to IQfeed but I had some interaction with them before and I am pretty sure that the tickID represents 1 buyer or seller. And indeed the tickID is not unique. Even for the symbol ES the same tickID can be used multiple times but not during the same trading session.
So I will try to clear these things up since if the same tickID is used multiple times per trading session and is unrelated to a single buyer or seller then it is pretty useless. But I think for a single trading session for the CME futures from 6PM to 5PM the next day the tickID is unique for a single future and is tied to a single trade.
Hi Tomasz and Ed,
A few important detailed is missed out on in this thread! It's correct what Gary Stephen from IQFeed said about the TickID and it is not necessarily identifying one specific Trade, it could be several trades of the same type in one bundle. But they are all of the same type- for example Buy or Sell. And the bundles are actually listed as one line item in the CME Time & Sales Data (more about it below)
Gary Stephen also said "- TickIDs are unique by ID, but also symbol and traded market. For example, there may exist a TickID 123456789 on NYSE and a TickID 123456789 on NASDAQ. It's also possible for two different symbols to have the same TickID. If you want to identify trades uniquely, you have to include the traded market and symbol as well. (However, see next item.)". After that he added that TickIDs are not necessarily unique by trade.
But please observe that the TickID is unique to a specific symbol, for a specific day. Actually seems to be unique for a whole trading week Sunday to Friday. I have 2 years, 2 months worth of ES and MES tick data that prove that for a fact.
Then I have actually used TickID from downloaded Tick Data and I can find each Bundle (same TickID) listed as one line item in the CME Time & Sales Data at CME Time & Sales Data
So the IQFeed TickID is actually confirmed to be a solution to many interesting indicators that can be designed using this data. This is data that can give us Retail Trader an edge against the "Big Boys"!
So if IQFeed TickID could be implemented in the IQFeed Plugin, that would be awesome and so very useful in not only Intraday Trading, but also longer term trading. Since it's possible to download up to 6 months worth of Tick Data from IQFeed.
Thanks and Regards,
Jorgen
And here is an example of the IQFeed TickID and how useful it is:
First here is a screen capture of some Sales & Time data from the CME website:
then followed by IQFeed Tick Data from the same time:
As you can see, the IQFeed TickID is showing the Tick Data that is bundled and it is a perfect match to the CME T&S data.
This is clearly data that the "Big Boys" don't want us Retail Traders to have! Maybe that's why the discussions you can find on the internet are confusing. I don't know. But I know that the TickID is the data that solves the problem and gives us AmiBroker users an extra edge!
Thanks and Regards,
Jorgen
yes I just checked and you are correct. They restart assigning tickIDs every Sunday at 18:00 ET. For the symbol @ES# they start with number 273 and as the number increases the first 2 digits always stay the same. So for instance 4-Sept-2022 the start tickID is 273 and the end of the week 9-Sept-2022 the last tickID is 272792863. The symbol @NQ# starts on 4-Sept-2022 with tickID 239 and ends on 9-Sept-2022 with 239322539.
Trade/Order lines with the same tickID represent 1 trade that is displayed unbundled. And this is what we need. If there is an uptick or downtick we want to count the volume of the entire trade, not just 1 order line. But we need the tickID for this because the timestamp is insufficient because the resolution is too low.
Sorry, but I don't get it why they are sending ONE trade as two or more lines. It is supposed to be tick by tick feed, where one line == one trade. IMHO the processing should be done by IQFeed so one line in tick history represent one trade.
Currently there is no room to store tickIDs as 32-bit numbers (the whole date time stamp is 64-bit and it stores year, month, day, hour, minute, second, microseconds), unless other field (such as Aux1/2) is re-purposed for that.
No, because the PLUGIN will generate UNIQUE timestamps if duplicate timestamps are found.
The only thing that I can do reasonably (without blowing up entire structure of the database) is to use the leftover 5 bits that I have in my 64 bit datetimestamps to store TickID % 32. These 5 leftover bits constitute "Submicrosecond" field that I left when designing the datetimestamp just in case I ever needed resolution finer than microsecond. I could store TickID % 32 (modulo 32), and you could use that to detect if TickID changes or not in subsequent "lines" or whatever we call what IQFeed sends.
I agree that what they are doing doesn't make any sense! Of some reason, the Exchanges bundle trades, so they can send several trades as once. At least that is what they claim they are doing. But I don't think that is 100% correct....
Then Gary says the following- "All bundled trades will have the same TickID. The IQFeed API "unbundles" them, so each trade has its own record, but they do have the same TickID. This is by design, so you can "rebundle" the trade data yourself if desired."
Saying that, I have hard to believe in what he is saying, especially "The IQFeed API "unbundles" them, so each trade has its own record,"! Because that would mean that each Tick Volume we get in AmiBroker from IQFeed is One Trade! And it is not! After lots of research and even by identifying my own trades of sizes 5 to 10 contracts in the tick data- I can see that Bundled data is at least One Trade! It could be 2 or more, that I don't know. But as long as we can bundle the trades and get identical data as CME is listing as Time & Sales Trades- we are good to go!
To what Gary said, my answer is: yes, our desire is to re-bundle the data and get exactly what CME is listing in their public Time and Sales Data listings. So Tomasz, your suggestion to store TickID % 32 should work fine! As long as we can identify the tick data with Individual TickID as "normal data" and a sequence of data with the same TickID as being a Bundle- So we can bundle it- that works for me. Works for you too Ed- right?
Thanks and Regards,
Jorgen
yes that would be great. I think that would work to identify trades. Thank you.
SUPER!!! Yes, you would be able to identify actual Filled Trade Orders. If it happens to be true that one "Bundle" can contain more than one Order (for which I have so far never seen any proof of) it will still be of huge advantage, since if that's the case- all orders are of the same type.
But that the CME Time & Sales Data show what it shows, kind of pointing towards one "Bundle" being one Filled Order.
Regards,
Jorgen
My understanding of the need for the TickID (CME) is to identify separate executions (partial fills) that all belong to a single trade.
I'd prefer to have a check box in the data plugin interface that allows the option of consolidating these executions into a single data point. IRT does this and it is described here Sierra has a similar feature.
It saves a TON of space in the database to do so for CME based futures contracts. A checkbox provides the ability for those who want this functionality to have it and for others to import the data from IQFeed as it is ...