I just noticed that many of my tickers have a price spike on october 28 at 15.55. The feed is from IQfeed and I'm also contacting them about it. However, the strange thing is that I cannot find the corresponding value in the quote editor?
I don't see that spike on mine. Right click on the green "OK" light and Force Backfill. The spike will probably vanish. This happens on many of the symbols that I watch because the data feed is "uncorrected" until you Force Backfill.
Another thing to do is look at IQFeed's chart and see if the spike is there. IQFeed's chart uses the corrected data if I'm not mistaken.
But I don't know why you see that spike on the chart but not in the quotes.
Of course this represents a major problem, because bad ticks/prices can cause errant signals and order generation. Fixing it afterwards at a later time doesn't fix the issue, because it's a matter of trying to have back-tests match as close as possible real-time results.
The only way to perhaps address this is to write your own bad tick/price flags and then to wait and delay your system/algo actions until more "validating" or "invaliding" price prints come in. Problem of course is when the market runs you will be late and incur more skid.
They do show up (unless you force backfill and remove spike). Everything what you see on chart is ALSO displayed in the quote editor. Quote editor displays data in BASE time interval (as selected in File->Database Settings), not in chart interval.
Then this is strange.. If you look at my two screenshots above, the "Base time interval" is set to 5 min. The chart is also 5 min. The spike appears in the chart, but not in quote editor. This is true for all my spikes.
Could it be that IQfeed somehow sends quotes with smaller resolution than 5 min and they go undetected in the editor?
PeterD, your trick of clicking the green light and force backfill solves the problem with the IQFeed unfiltered quote messing up my chart, and also seems to download more accurate volume data, so thanks for that!
Do you know if there a way to force backfill the entire database this way to rinse all the spikes I got?
Unfortunately, there is no way that I know of to automate the process of forcing backfills on all the symbols. I've brought it up before to no avail so I am stuck going through manually backfilling 67 futures symbols once a week to ensure that I don't miss any contract rollovers, because I want the old data to be replaced with the new adjusted data after rollover.
My TF is 5 min. The problem, I think, is that AmiBroker does not detect any missing bars. The bars are there, but with real time "spike-quotes". The manual "force refill" re-downloads the quotes and IQFeed has cleaned their data by them.
I tried the "wait for backfill", but that didn't work for me. And it makes sense that AmiBroker wouldn't re-download the quotes that are obviously already present.
IQFeed plugin (as any other data plugin) is meant to be TRANSPARENT. With the exception of recently requested re-bunding of tick data, the plugin DOES NOT DO any processing. Everything you see comes straight from IQFeed, including spikes. IQFeed is known as "UNFILTERED" feed.
Yes, I think both me and PeterD fully understands that, and we also understand the rationale behind it, and we are not suggesting there is any bugs or flawed design here. But what he noticed is that an easy way to "clean" the unfiltered data is to force a backfill of that particular symbol. And I also noticed that the volume seems to update to a more correct number in my case.
So if there was a way to, for example, select multiple symbols and do a force backfill of them, both our problems would be solved.
I guess if there was a way to purge the last few days for every symbol in the db, that would trigger a backfill, and that would also do the trick.
I was planning something along these lines long ago, but few obstacles exist:
some feeds prohibit automated backfills for large symbol lists
some feeds put limits on how many symbols / how often you can send requests
some feeds specifically say that backfill requests need to be triggered MANUALLY (no automated scans)
So it is not an issue from AmiBroker side but from data vendor side limits. It is a case of "fair usage", how much you can do without raising red flag of "abuser".
Alternatively, I could provide more "lean" method: instead of forcing backfill of entire thing, allow the user to specify "overlap" zone - how many bars / days to backfill when software is run and symbol is displayed as compared to last available date. Currently it uses 3 bars overlap (so last 3 available bars are re-downloaded each time symbols starts streaming). If plugin allowed user-selectable 'overlap' of more than 3 bars, it would fix them (assuming the spikes appeared only recently).
Third solution - is to implement bad tick filter for streaming data from IQFeed (so plugin is no longer "transparent"). But then some would complain that data "don't agree" with IQFeed (because they were filtered).
One "easy" solution could be that you implment the functionality that users are allowed to select multiple symbols and force backfill them, and exclude that option from feeds that doesn't allow it. I would say that that procedure should count as "manual". Also, adding a delay of maybe up to a few seconds between symbols would be fair. This isn't supposed to be something users should do often, more like a last resort solution you would do at the end of the week for the entire db. To be more gentle with the providers, maybe users could specify how many days back a "force backfill" should fill also. In my case, 4-5 days would be more than sufficient.
About the "lean" method, I see some trouble with that.. If a user would fiddle around with that setting and leave it at a high number, it may put a lot of pressure on the providers and maybe giving AmiBroker a bad reputation. Also, that would trigger the process of re-download every time, even when the data is fine.
The third solution is appealing, but I agree that you will get loads of complaints about data inconsistency. I guess the way forward would be to have an on/off filtering option for the plugin, where you can specify a few basic criteras when the quote should be dropped.
That, however, would not solve the problem with the volume appears to be more correct after a forced backfill, but it would save traders acting on real time data from headache.
I guess both a way to manually force backfill of multiple symbols and a realtime tick filter would be on top of my wishlist
Thank you for your insight on the limitations of reasonable use of the data feed, as well as your continued support of all of us!
I like your idea of a "lean" method, and would also like to be able to replace the entire history if needed. Here's an example of how a contract backadjusts after rollover. All the prior data changes as you can see from the animated chart.