Data Quality / Unfiltered Data

Does anybody know if IQ feed still provides unfiltered data or did they change that and the data quality is good now compared to eSignal? I am considering switching from eSignal.

Their web site still says unfiltered:

1 Like

The unfiltered feature of IQFEED causes two problems for me. First, trades of fewer than 100 shares are included in the feed. That can affect backtesting because the values are, I think, included in the open/high/low/close values even though they may not be indicative of trades you could actually execute. I filter those trades out by exporting only trades of 100 shares or more to a file and then importing them to a different database with only "qualifying" trades. For real time applications as opposed to backtesting, XLQ can filter these non-qualifying trades.

The second problem is, I think, more severe. I think some trades that are executed off the exchanges are reported to an exchange and then included on the IQFEED feed. Those trades, I think, have a time stamp of when they are reported to the exchange, not when they are actually executed. As a result, there are frequently large trades in which you cannot participate and which represent market values at a different point in time that are included in the IQFEED feed as if they are regular trades.

I am not completely certain that I am describing the issues correctly but I think I am. I would appreciate hearing if anyone has a solution to the second problem. If the solution can be implemented with IQFEED, that would be fine. If it involves another data source, that would be fine too but I would prefer that source to provide tick data and work with Amibroker.

1 Like

@mss Thank you for this info! Just curious given so many issues with the data, why would you use IQFEED instead of eSignal then?

@mss - Filtering or not filtering is only about bad ticks. Filtered feed attempts to remove bad ticks using algorithms. Not more.

There is official FINRA document http://www.finra.org/industry/trade-reporting-faq#101 that explains how trades are reported in "real time" and as you can read upto 10 second delay in reporting executed trades is considered "normal". These are not bad ticks and won't be filtered.

1 Like

I apologize in advance for what is going to be a long comment.

  1. Regarding ESignal.

I did not intend to complain about IQFEED or Amibroker. I am, in fact, quite pleased with both. I am just trying to deal with a few issues that appear to be relevant to my trading. I can see nothing in the ESignal documentation that suggests ESignal would be better for me. Since my subscription to IQFEED appears to be cheaper than data subscriptions to ESignal and since many posters on this and other forums appear to be at least as happy with IQFeed as with ESignal, I am not considering changing (although I would if I thought the change would solve my issues).

  1. Regarding Filtering and odd-lot data

Unquestionably Tomasz's description of filtering is the accepted one and that document from FINRA was quite interesting. Some trades that I am guessing are off-exchange trades are so far off from the surrounding trades that I was surprised to learn of the 10 second guideline. Dealing solely with odd-lot issue, however, in 2015 I noticed differences between the TDAmeritrade feed for last price and the IQFeed feed for last price. I did not think the differences could be accounted for by simply some delays in price quotes. I then found a setting in XLQ that allowed me to filter out non-qualifying trades (i.e., odd-lot trades) from IQFeed. Adjusting that setting caused the last price values in the two feeds to be substantially identical. That suggests to me that some feeds (apparently including TDAmeritrade at the time) "filter out" odd lot trades.

  1. Regarding my real problem -- Filtering off-exchange trades

I think IQFeed provides some sort of consolidated data feed. The sample spreadsheet that comes with the IQFeed software has fields relating to the quote called "Mkt Center" and "FSI." I could not find documentation on those fields but I wonder if it would be possible to "filter out" off-exchange trades by using them. There is a function in XLQ called "xlqExchangeofLastTrade" which suggests to me it might be possible to exclude off-exchange trades. If it is possible, I would appreciate advice on how to proceed.

  1. Filtering off-exchange trades and Tiingo

I had never heard of Tiingo until I started searching this site for leads on my issues. I see that it apparently reports trades from a single exchange and that exchange has meaningful volume. Is it possible that, since the data is apparently not a consolidated feed and would, I assume, not include off-exchange trades, I could use that data instead of IQFeed data? Does anybody have a feel for whether that data is appropriate for realistic modeling in Amibroker?

Thanks in advance for any advice on these items.

Tomasz:

I spoke to a fellow at DTN Iqfeed (Kevin) about some of the ticks that are brought over from more minor exchanges that create these distortions. For example, they are typically very low volume trades of 1 to 10 shares or so. He had me open Iqfeed "Time & Sales" , and ensure that "Enable Quote Updates" was not checked. I compared these prices to my symbol quotes (ticks) in Amibroker for the time periods that these "outlier ticks" occurred from other exchanges, and it seemed they had been eliminated in the Iqfeed listing.

I am not sure if Amibroker could be modified to accomplish the same result, but thought I would bring this to your attention as a possibility, or if there is another workaround.

Thank you for considering this information.

Mike