There are too many spikes with IQfeed data. Is there any way to prevent it. It's quite annoying .
I once had a long discussion with IQFeed tech support about this issue in 2017. What is happening is that IQFeed is sending some trade data for stocks, it only happens with stocks, quite late and that spike that you are seeing is that data point being charted. Ideally it should be ignored but I am not the one designing this app.
The message I received from IQFeed is quoted here verbatim along with the reference image: "We found the spike you are talking about. Please see the screenshot below. This was a legitmate trade and not totally uncommon in SPY data. IQ Feed is an unflitered data feed. You would need to use your charting platform to filter these type of trades maybe by INC Volume if you do not want to see them."
If you want to see what rules are used by the Consolidated Tape Association to update the last price, see here:
on page 63.
e.g. a Contingent Trade does NOT update the open, high, low or last price. It only updates the volume.
Cheers,
Richard.
That is the KEY sentence in IQFeed reply. IQFeed does NOT filter such trades. AmIBroker just displays what it receives from IQFeed.
We are NOT data vendor.
AmiBroker is transparent with regards to data feed, which means it displays quotes exactly and precisely as they arrive from the feed's API, without any modification.
1 Like
Hi Tomasz,
If I may add to this thread, IQFeed does the right thing with its datafeed - they provide all the trades along with the trade conditions defined in the UTP and CTA specifications.
As Richard (@NorgateData) has mentioned above, UTP and CTA specifications require that certain trade conditions not update the OHLC fields. The OHLC bars computed by Amibroker are incorrect since they do not conform to the UTP and CTA specifications. This nonconformance with exchange specifications manifests as spikes on OHLC bars. These spikes are hugely problemantic for intraday strategies (both realtime trading and backtesting).
The solution to these spikes is logic that needs to be embedded in the Amibroker IQFeed plugin (the Amibroker platform along with its IQFeed plugin needs to conform to the exchange OHLC specifications). This logic is not something for the IQFeed servers (they are sending the right data) and the logic is also not something for the Amibroker user (the user does not get to see the trade conditions sent by IQFeed).
As example, please see Amibroker M1 chart for MSFT today with two large spikes (14:39 bar and 15:22 bar). IQFeed Time & Sales tool suggests that the 14:39 spike resulted from a Qualified Contingent trade condition and the 15:22 spike resulted from an Average Price trade condition. Neither of these trade conditions should have been allowed to set the OHLC fields (pages 43 and 44 from the UTP specification attached below).
Amibroker is a fantastic platform with great speed and power. I am grateful to you for making such a platform available to us. If we could make the Amibroker IQFeed plugin conform to the exchange specifications, it would make Amibroker a platform par excellence for intraday stock trading and backtesting.
I have attached images explaining these issues below.
Best wishes,
@alpha1
MSFT M1 chart in Amibroker with two data spikes:
IQFeed Time & Sales data for 14:39 spike:
IQFeed Time & Sales data for 15:22 spike:
Page 43 of UTP spec:
Page 44 of UTP spec:
Page 64 of CTA specification (CTA is to NYSE as UTP is to Nasdaq):
Theory is simple. Practice isn't.
The practice is that only IQFeed has access to feed directly from exchange. Now two things may happen, one that is possible to handle on client site and one that doesn't.
Situation A (all conditions must be met at the same time):
1 If spike occurs NOW (in real time while new candle is forming)
AND
2. If plugin is subscribed already to the symbol
AND
3. the symbol is already backfilled
AND
4. plugin builds given candle from incoming ticks
AND
5. if incoming spike tick is really marked appropriately by IQFeed
then yes, the plugin may have a chance to filter it
Situation B:
- When NOT ALL conditions of Situation A are met
OR - When spike exists in IQFeed HISTORICAL BACKFILL DATA
Yes, my experience with IQFeed is that the spikes most often occur in HISTORICAL BACKFILL data that we don't control. We just receive say 1-minute OHLC bars without any extra info that were composed entirely on IQFeed side. To address that the plugin has option to enable ATR-based bad tick filter that filters those outliers. The filter is turned on, so you may not see all the spikes that you would see without it and those spikes exist on what IQFeed composed on their side. Historical 1-minute data downloaded from IQFeed does not have any "trade conditions" because these are already time compressed bars just OHLC from entire 1-minute representing possibly hundreds or thousands of individual ticks.
Hi Tomasz,
Thank you for looking into this.
I have some programming experience with the IQFeed API trade conditions. My experience has been that the trade conditions are accurately set in realtime. I have compared DTN trade conditions in realtime with another datafeed and they both matched with respect to the trade conditions.
Also, my experience is that the spikes do not occur when historical M1 data is downloaded by AB from IQFeed servers since IQFeed servers conform to the exchange rules when constructing the OHLC bars.
Attached below is a chart of MSFT for same time interval as the one in earlier post. The earlier one was M1 bars constructed in realtime by AB plugin while this new chart is based on historical M1 bars sent by IQFeed servers. Note that the spikes are gone.
Also, when implementing the trade condition logic inside the plugin, there is one trade condition for which I would deviate from exchange rules and that is ODDLOT condition. That exchange rule comes from the Jurassic Park era when ODDLOT trades were infrequent. But these days, 90% of the trades in some stocks like AMZN are ODDLOT trades due to high share prices.
Best wishes.
Some additional notes:
- If we use tick database, and then construct M1 OHLC using historical data download, the spikes reappear since all ticks are present in the historical download. Below is chart of same time interval for MFST with M1 bars constructed from tick history download.
- Any database interval other than ticks has no spikes for historical downloads. I already posted the chart in post above for database interval being M1. Below is chart with 1 second database interval. M1 bars constructed from historical 1 second has no spikes since DTN servers did the right thing when constructing the 1 second OHLC bars.
Well, last time I checked, there were spikes in historical 1-minute data backfilled straight from IQFeed. You are not looking far enough. I just did backfill today, there is a spike in MSFT 29.07.2021 16:37:00 in HISTORICAL 1-minute backfill:
Hi Tomasz,
Thank you for the reply. The spike in the IQFeed M1 historical chart you posted is "legitimate" (see caveat in paragraph below). IQFeed servers did the right thing.
The 16:37 bar is an outside market hours OHLC bar. Outside market hour trades have FormT trade condition. UTP and CTS rules say that FormT trades should not set OHLC fields. Implicitly, the exchange rules prohibit constructing OHLC bars with outside hour trades. All bets are off when constructing OHLC bars using outside market hours trades. In other words, spikes may result.
Attached is a Time & Sales from a different datafeed provider which shows that the spike in the IQFeed M1 chart resulted from a FormT trade:
Also attached is UTP page 63 with OHLC conditions for FormT:
Best wishes.
I know it is after hours, but we are speaking about 1-minute bars with extended trading sessions, not end-of-day bars.