I'm seeing lots of bad ticks (long wicks) causing my system to send bad signals to IB API. I know this has been discussed and is not an Ami problem, but I would love to keep my backtests and trading both on this platform for ease of reconciliation.
Can anyone suggest a reliable way to do this? Or if not, please chime in with your preferred trading tools. I am a newb and am looking for options from this knowledgeable community!
In the subject line you say IQFeed but in the post you say IB (Interactive Brokers). So which one you are after?
Discovering that IQF RT data is unusable has ruined my plans. I have been paper trading using IQF->Ami Explore->CSV->Python Trade Client->IB API. I like the tools I developed, but need to rethink now. I do not have any backup ideas on how to automate day trading strategies.
I am hoping the community will share their tools of choice and provide some avenues for research.
No, IQFeed is not "unusable". The code you write should handle bad ticks because bad ticks just happen and may happen with any source. They happen because they are sent from the exchange. Recommended reading on the subject:
Thanks for the reply. Interesting paper, but very discouraging.
There is a section in this paper that describes TickData filtering:
The basis of the filter is a moving transform of price. The number of data points used in calculat-
ing the transform is a function of tick frequency. This is the first step in adapting the filter to the
unique activity levels of various issues. Next, we measure each tick’s distance from this moving
transform and convert that difference into units that scale across securities. This makes the filter
adaptable to securities with different price scales. Ticks that exceed a user-defined threshold are
deemed bad. Allowing the threshold to be defined by the user enables the filter to adapt to the base unit of analysis of the trader and manage the overscrub/underscrub tradeoff. Lastly, ticks that are deem bad are replaced with the value of the transform and assigned the volume of the bad tick.
They clearly state that they don't want to fully disclose their method, but from chart in the article and the description it looks like they are using a kind of short-term median as "transform" and distance from median. Median is relatively compute-heavy algorithm, but with short periods it is not as bad. You could use Median() built-in AFL function and distance from median to easily detect which tick is bad (if you were using TICK data).
For 1-minute data things are more difficult.
I may consider adding median based filter in the plugin, but such filter requires additional computation (CPU cost) and obviously, it won't cure all bad ticks, but only those that are distant enough (above threshold distance)
Filtering does sound doable, but adds alot of complexity. This stuff is frankly complex enough!
Would love to know how other people are trading.
I will do some experiments with median filter on top of IQFeed stream and will see how much it helps. One other thing that should be considered if ticks arrive slowly (or after pause), I think there is no resonable way to apply filter because with large time gaps between ticks, the price gaps can be large too in practice.
While not directly related to the topic, here's an interesting article on how intraday data from different providers can yield significantly different results:
I have always believed in our own custom "Bad tick" algorithms for this very reason. In essence, it should make the back-test and real-time match closer, if not exact (if the same source). Therefore, your bad or questionable price algorithm is in the back-test also. If you look at the TAQ documents, sometimes it is simply what "flag codes" are allowed into the feed, like late or out-of-order trades is just one example. Just looking for a confirmation print or confirmation bar will take 98 percent of these "bad" prices out IMHO. These mostly look like "Island" spikes as well.
beppe, I was hoping for an open-ended discussion here, so this is definitely not off topic. Very interesting paper.
I noticed that IQF corrected all the bad ticks I had identified. When deleting and backfilling, the new database was brought into agreement with my broker's charts. Per the paper, small differences may be left that can make a backtest questionable, but as long as you keep some skepticism, that should be okay.
But how to make the system even tradeable in realtime? Are all daytraders self-correcting their stream?
One simple way to remove out-of-order trades is to check ticks against bid and ask prices. Depending on the exchange, it can also be useful to verify that ticks come with their corresponding bid and ask quotes. If they don’t, you can be certain they are late orders.
This bad tick issue is currently being observed and investigated. Expect improvements in the form of post-filtering soon.
I was watching the tape as reported by IQFeed and after all day long tracing it I found out that majority of those "bad ticks" have the following "Trade conditions" set:
47 - TTEXEMPT - Transaction is Trade Through Exempt
19 - SOLDOSEQ - Sold Out of Sequence
Many of them are odd lot trades. It seems like filtering them out would remove majority of really bad looking ticks. But be aware that reported cumulative volume after filtering then would not match official figures.
That would be great if a filter that simple could work. I suspect the volume would not be significant, but maybe you have numbers.
You mentioned to "expect improvements in post filtering." Does that mean it would be incorporated into the plugin? That would be amazing. If so, would it affect aggregated feeds?
Yes it will be integrated inside the plugin.
A new plugin (experimental) is available: