IQFeed plugin 7.xx with bad tick filter

[Tomasz]

UPDATE: There is a newer version of the plugin announced here: IQFeed plugin 7.xx with bad tick filter - #24 by Tomasz

Here is a new version of IQFeed plugin that features experimental RT streaming bad tick filter that is based on "Trade conditions" reported by IQFeed.

DOWNLOAD

https://www.amibroker.com/bin/IQFeed7000.zip

(both 32-bit and 64-bit plugins are there in separate subfolders inside zip)

INSTALLATION INSTRUCTIONS:

Right click on link, "Save Link As...", download the ZIP, then UNZIP (right click on zip and "Extract All") and navigate to either 32-bit or 64-bit subfolder, then replace the IQFeed.dll in AmiBroker's "Plugins" directory

Then run AmiBroker. Go to Tools->Plugins and verify that listed version is 7.0.0

If you see version 7.0.0 it means that you have installed it correctly. If you see any earlier version, it means that you did not replace the plugin.

NOTE: please DO NOT keep old version of the plugin in the "Plugins" folder. If you want to keep it, you need to move it to other folder.

HOW TO USE

To enable filtering of bad ticks you need to go to File -> Database Settings -> Configure and
check the following in the "Ignore price, but accumulate volume" field:

• 13 - PRP Prior Reference Price
• 19 - SOLDOSEQ Sold Out of Sequence
• 3A - FORTMTSOLDOSEQ Form-T Sold Out of Sequence
• 3E - DERIVPRI Derivately Priced (update)
• 47 - TTEXEMPT Transaction is Trade Through Exempt

These trade conditions were found to cause "bad ticks" due to reporting trades that occurred out of sequence. By default the plugin will ignore price of such tick, but accumulate volume.

You can however configure the plugin (in File->Database Settings->Configure) to either completely ignore trades with given trade condition or to ignore price, but keep accumulating volume.

The complete list of trade conditions reported by IQFeed is here:

01 - REGULAR Normal Trade
02 - ACQ Acquisition
03 - CASHM Cash Only Market
04 - BUNCHED Bunched Trade
05 - AVGPRI Average Price Trade
06 - CASH Cash Trade (same day clearing)
07 - DIST Distribution
08 - NEXTDAY Next Day Market
09 - BURSTBSKT Burst Basket Execution
0A - BUNCHEDSOLD Bunched Sold Trade
0B - ORDETAIL Opening/Reopening Trade Detail
0C - INTERDAY Intraday Trade Detail
0D - BSKTONCLOSE Basket Index on Close
0E - RULE127 Rule - 127 Trade NYSE
0F - RULE155 Rule - 155 Trade AMEX
10 - SOLDLAST Sold Last (late reporting)
11 - NEXTDAYCLR Next Day Clearing
12 - LATEREP Opened - Late Report of Opening Trade (in or out of sequence)
13 - PRP Prior Reference Price
14 - SELLER Seller
15 - SPLIT Split Trade
16 - RSVD (Reserved)
17 - FORMT Form-T Trade
18 - CSTMBSKTX Custom Basket Cross
19 - SOLDOSEQ Sold Out of Sequence
1A - CANC Cancelled Previous Transaction
1B - CANCLAST Cancelled Last Transaction
1C - CANCOPEN Cancelled Open Transaction
1D - CANCONLY Cancelled Only Transaction
1E - OPEN Late Report of Opening Trade - out of sequence
1F - OPNL Late Report of Opening Trade - in correct sequence
20 - AUTO Transaction Executed Electronically
21 - HALT Halt
22 - DELAYED Delayed
23 - NON_BOARDLOT NON_BOARDLOT
24 - POSIT POSIT
25 - REOP Reopen After Halt
26 - AJST Contract Adjustment for Stock Dividend - Split - etc.
27 - SPRD Spread - Trade in Two Options in the Same Class (a buy and a sell in the same class)
28 - STDL Straddle - Trade in Two Options in the Same Class (a buy and a sell in a put and a call)
29 - STPD Follow a Non-stopped Trade
2A - CSTP Cancel Stopped Transaction
2B - BWRT Option Portion of a Buy/Write
2C - CMBO Combo - Trade in Two Options in the Same Options Class (a buy and a sell in the same class)
2D - UNSPEC Unspecified
2E - MC_OFCLCLOSE Market Center Official Closing Price
2F - STPD_REGULAR Stopped Stock - Regular Trade
30 - STPD_SOLDLAST Stopped Stock - Sold Last
31 - STPD_SOLDOSEQ Stopped Stock - Sold out of sequence
32 - BASIS Basis
33 - VWAP Volume-Weighted Average Price
34 - STS Special Trading Session
35 - STT Special Terms Trading
36 - CONTINGENT Contingent Order
37 - INTERNALX Internal Cross
38 - MOC Market On Close Trade
39 - MC_OFCLOPEN Market Center Official Opening Price
3A - FORTMTSOLDOSEQ Form-T Sold Out of Sequence
3B - YELLOWFLAG Yellow Flag
3C - AUTOEXEC Auto Execution
3D - INTRMRK_SWEEP Intramaket Sweep
3E - DERIVPRI Derivately Priced
3F - REOPNING Re-Opeing Prints
40 - CLSING Closing Prints
41 - CAP_ELCTN CAP (Conversion and Parity) election trade
42 - CROSS_TRADE Cross Trade
43 - PRICE_VAR Price Variation
44 - STKOPT_TRADE Stock-Option Trade
45 - SPIM stopped at price that did not constitute a Trade-Through
46 - BNMT Benchmark Trade
47 - TTEXEMPT Transaction is Trade Through Exempt
48 - LATE Late Market
49 - XCHG_PHYSICAL Exchange for Physical
4A - CABINET Cabinet
4B - DIFFERENTIAL Differential
4C - HIT Hit
4D - IMPLIED Implied
4E - LG_ORDER Large Order
4F - SM_ORDER Small Order
50 - MATCH Match/Cross Trade
51 - NOMINAL Nominal
52 - OPTION_EX Option Exercise
53 - PERCENTAGE Percentage
54 - AUTOQUOTE Auto Quotes
55 - INDICATIVE Indicative
56 - TAKE Take
57 - NOMINAL_CABINET Nominal Cabinet
58 - CHNG_TRANSACTION Changing Transaction
59 - CHNG_TRANS_CAB Changing Transaction Cabinet
5A - FAST Fast Market (ssfutures)
5B - NOMINAL_UPDATE Nominal Update
5C - INACTIVE Inactive - Nominal - No Trade
5D - DELTA Last Trade with Delta Exchange
5E - ERRATIC Erratic
5F - RISK_FACTOR Risk Factor
60 - OPT_ADDON Short Option Add-On
61 - VOLATILITY Volatility Trade
62 - SPD_RPT Spread Reporting
63 - VOL_ADJ Volume Adjustment
64 - BLANK Blank out associated price
65 - SOLDLATE Late report of transaction - in correct sequence
66 - BLKT Block Trade
67 - EXPH Exchange Future for Physical
68 - SPECIALIST_A Ask from specialist Book
69 - SPECIALIST_B Bid from specialist Book
6A - SPECIALIST_BA Both Bid and Ask from Specialist Book
6B - ROTATION Rotation
6C - HOLIDAY Holiday
6D - PREOPENING Pre Opening
6E - POST_FULL Post Full
6F - POST_RESTRICTED Post Restricted
70 - CLOSING_AUCTION Closing Auction
71 - BATCH Batch
72 - TRADING Trading
73 - OFFICIAL Official Bid/Ask price
74 - UNOFFICIAL Unofficial Bid/Ask price
75 - MIDPRICE Midprice last
76 - FLOOR Floor B/A price
77 - CLOSE Closing Price
78 - HIGH End of Session High Price
79 - LOW End of Session Low Price
7A - BACKWARDATION Backwardation - immediate delivery costing more than future delivery
7B - CONTANGO Contango - future delivery costing more than immediate delivery
7C - RF_SETTLEMENT Refined Fuel Spot Settlement
7D - RF_RESERVED1 Refined Fuel Spot Reserved - 1
7E - RF_RESERVED2 Refined Fuel Spot Reserved - 2
7F - RF_RESERVED3 Refined Fuel Spot Reserved - 3
80 - RF_RESERVED4 Refined Fuel Spot Reserved - 4
81 - YIELD Yield Price
82 - BASIS_HIGH Current Basis High Value
83 - BASIS_LOW Current Bases Low Value
84 - UNCLEAR bid or offer price is unclear
85 - OTC Over the counter trade
86 - MS Trade entered by Market Supervision
87 - ODDLOT Odd lot trade
88 - CORRCSLDLAST Corrected Consolidated last
89 - QUALCONT Qualified Contingent Trade
8A - MC_OPEN Market Center Opening Trade
8B - CONFIRMED Confirmed
8C - OUTAGE Outage
8D - SPRD_LEG CME spread leg trade
8E - BNDL_SPRD_LEG Final CME MDP3 trade from Trade Summary message that could not be Un-Bundled
8F - LATECORR LSE - Late Correction
90 - CONTRA LSE - Previous days contra
91 - IF_TRANSFER LSE - Inter-fund transfer
92 - IF_CROSS LSE - Inter-fund Cross
93 - NEG_TRADE LSE - Negotiated Trade
94 - OTC_CANC LSE - OTC Trade Cancellation
95 - OTC_TRADE LSE - OTC Trade
96 - SI_LATECORR LSE - SI Late Correction
97 - SI_TRADE LSE - SI Trade
98 - AUCT_TRADE LSE - Auctions (bulk;individual)
99 - LATE LSE - Late trade
9A - STRAT LSE - Strategy vs. Strategy Trade trade
9B - INDICATIVE_AUCT LSE - Indicative Auction Uncrossing Data

[Tomasz]

In File -> Database Settings -> Configure

More codes to consider ignoring are (on-going list):

• 3E - DERIVPRI Derivately Priced

[PeterD]

My plugin didn't seem to be pre-configured to have any bad tick categories checkmarked after installation, so I checked the boxes manually.

[Tomasz]

Yes. It seems that on clean, new installations the defaults are indeed empty and one has to add those manually.

[PeterD]

Thank you for creating this new plugin version! As I was testing it, I was still seeing price spikes in SPY even when I checked all the boxes. This is what I was seeing: The low of 742.0805 at 3:46 doesn't fit with the rest of the lows around that time.

Then there were more bars at 3:57 and 3:58 with the exact same low. How strange!

[Tomasz]

By "all the boxes" you mean that you checked these boxes:

• 13 - PRP Prior Reference Price
• 19 - SOLDOSEQ Sold Out of Sequence
• 3A - FORTMTSOLDOSEQ Form-T Sold Out of Sequence
• 3E - DERIVPRI Derivately Priced (update)
• 47 - TTEXEMPT Transaction is Trade Through Exempt

Correct?

Or do you mean really ALL boxes? If so, it is not desired because many trade conditions are perfectly valid trades that should not be ignored.

If you marked those 5 boxes mentioned above, chances are that there are other trade conditions that should be filtered out. Unfortunatelly, IQFeed support is not particularly co-operative. Clearly they filter out some of the codes from the historical intraday (because backfill removes those spikes) and yet when I asked them they say that they don't filter anything. Either the person answering does not know, or I don’t understand what is the problem with documenting stuff.

These "same low" ticks from my observation are typical to out-of-sequence trades. There are other possible trade conditions that might need to be ignored. Possible candidates are

05  AVGPRI
08  NEXTDAY
10  SOLDLAST
12  LATEREP
13  PRP
17  FORMT
19  SOLDOSEQ
1A  CANC
1B  CANCLAST
1C  CANCOPEN
1D  CANCONLY
1E  OPEN
1F  OPNL
22  DELAYED
2A  CSTP
33  VWAP
36  CONTINGENT
3A  FORTMTSOLDOSEQ
3E  DERIVPRI
43  PRICE_VAR
44  STOCK OPT TRADE /* new */
47 - TTEXEMPT Transaction is Trade Through Exempt
48  LATE
52  OPTION_EX
58  CHNG_TRANSACTION
59  CHNG_TRANS_CAB
63  VOL_ADJ
64  BLANK
65  SOLDLATE
88  CORRCSLDLAST
89  QUALCONT
8F  LATECORR
90  CONTRA
94  OTC_CANC
96  SI_LATECORR
99  LATE

[PeterD]

I kept seeing the "same low" ticks, so I literally checked every single box just for testing, and I kept seeing the same lows again.

[Tomasz]

Did you see that lows on new bars created AFTER you turned on all the boxes?

Keep on mind that the filtering occurs in RT stream and only newly arriving ticks are subject to filtering. Changing the filter does NOT affect the history at all.

If so it means that these lows were created by "regular" ticks that don't have any special conditions attached to them.

The plugin solely depends on whatever is sent from the source/IQFeed/exchange.

If that was indeed a case, the only way to filter those remaining bad ticks is to make arbitrary assumptions (guessing) as to what tick is "bad" and which isn't. Arbitrary assumptions may lead to filtering out legitimate trades.

[PeterD]

I was seeing the lows on new bars after turning on all the boxes, so they must have come from the regular ticks, which is unfortunate.

Somehow IQFeed's chart app is able to filter out the bad ticks and their customer support people just shrug their shoulders when API clients can't filter the same way.

Thank you for working on this filter, hopefully it will lead to a feed that doesn't have these spikes.

[Tomasz]

Their "chart app" simply uses backfill. And the same you will get using backfill in AmiBroker ("force backfill"). The DO FILTER the historical data (backfill), but they for some reason say that they don't, when asked specifically.
The only way to filter regular trades that are not marked with any trade conditions is artificial processing (for example you drop trades that occur further than certain % from bid-ask midpoint). I can do that, but that is ARBITRARY processing. Unless IQFeed reveals what ARBITRARY processing they do with historical data, you can't replicate what they are doing reliably. I simply don't get why they don't document stuff.

[BJW]

I think it's worth mentioning that for CME Futures, I have never seen a bad tick and that's over 10 years! But with stocks, like AAPL, plenty of it!

Some time back, I did a quick check and for example ES and NQ always stick to the specified 0.25 Tick Size. While stocks, which as far as I know, have a Tick Size of 0.01, do not stick to it! I have seen stock prices with three/four decimals. Not sure if this has anything to do with bad ticks, but I tought it was worth mentioning.

Regards,

Jorgen

[johnlauve]

Thanks, Tomasz! I am testing now and will give feedback when I can.

[alligator]

Unlike derivatives markets such as CME or EUREX, off‑exchange trades on U.S. equities (NYSE, NASDAQ) do show up in the quote feeds. The reason is a regulatory transparency requirement imposed by the SEC. The regulation mandates full price transparency for any transaction on the underlying asset, regardless of where it is executed. The goal is to maintain a single national price (NBBO).** **OTC trades on derivatives, on the other hand, are often bespoke contracts (swaps, forwards) that are not standardized like exchange‑traded futures. As a result, they are not reported in the real‑time price feed, even though they may be submitted to clearing houses or trade repositories (such as DTCC or LCH), but with a delay and without appearing in the real‑time quote feed.

[alligator]

There are specific cases where price precision goes beyond 2 decimals (up to 4 decimals). If you see a price quoted with 3 or 4 decimals, it’s either a penny stock or an average price computed from a consolidated trade (VWAP of multiple executions). By the way, the TWS API lets you query the tick‑size / precision depending on the price level and the exchange.

[alligator]

I’m not sure, but I think we can basically restrict ourselves to four condition types: REGULAR, AUTO, OPEN, and CLOSE. OPEN/CLOSE could be discussed though. Everything else seems to just pollute the quote feed. For bad ticks, you probably need to filter out ticks coming from certain exchanges, and keep only the major ones for U.S. equities: NYSE, NASDAQ, ARCA, BATS, IEX. Alas there are several codes for each exchange (possibly XNYS, NYSE, XNAS, NASDAQ, BATS, BATY, BZX, EDGX, EDGA, IEX, IEXG, N, Q, P, Y, Z, K, J). I suspect faulty quotes come from secondary exchange(s). Would it be possible to test this combination and, if it works, have a one‑click ‘Restrict US Stock Feeds’ mode that combines both filters ?

[Tomasz]

Some of those special codes are legitimate good trades for example intermarket sweep. FormT is also good outside regular hours.

[johnlauve]

It's hard to quantify, but today I saw lots of bad tails with the first 5 recommended boxes checked. I estimate no real improvement. I am going to try tomorrow with the next 25 or so candidates you listed.

[johnlauve]

With all candidate boxes checked (28 of them) I am seeing excellent agreement between IQ RT feed and Schwab charts.

I visually compared 40 or so symbols and could find no differences. Prior to the new plugin, half of the symbols were having obvious issues every day. It appears the filter is preventing the bad ticks without causing any important missed data. No significant volume differences either.

It needs more rigorous validation obviously, but I'd say you are on to something here, Tomasz! This makes me happy.

[PeterD]

@johnlauve, would you mind taking a look at SPY on a low timeframe, say 1-min? When I leave the chart open I see the tails, even with all 28 checked. Curious if you will the same thing.

[johnlauve]

I did not have any ETF's on my RT list today, but I would be happy to do some comparison's next week.