How to Manage Streaming vs Backfill Discrepancies (5-second charts)

Does anyone here have experience algorithmically trading lower timeframes? I'm currently shooting for 5 to 15 second timeframes, but I am seeing sizable differences in historical (backfill) data and that which I received via streaming, or live / real time.

Does anyone know of any "tricks" or have tips to working in these lower timeframes where precision is more important? Is there a way for me to backfill data every ~second? I may have some delay, but at least my signals will be correct.

I recognize a lot of people here use IQFeed, is this the reason why? Do you see less discrepancy between live and backfilled data?

Thanks in advance

Hi
i hope you know you must set in the preference window Realtime chart refresh interval to one or zero?


this is from the the manual

Realtime chart refresh interval - defines interval between automatic chart refreshes in real-time mode. By default charts are refreshed every 3 seconds but in very volatile market you may prefer to set it to 1, so charts are refreshed every second in real-time mode.

1 Like

IB uses local computer timestamp to determine where given real-time tick falls into because (at least at the time of the plugin development) IB did NOT send timestamps with each tick. The reason for that is that IB does not really send real trades one by one. They send "accumulated" ticks every 0.2...0.3 seconds that contain accumulated information about all trades that happend over this small period of time. This may mean that if your computer is not in perfect sync with exchange boundary ticks may fall into different candle. When you backfill, you get HISTORICAL data that were created on other computer (IB server you are currently connected to) with likely slightly different clock. And no, you can not backfill every second. IB backfills are slow take a lot more than one second.

IB is actually well aware about this problem and they recently added some "workaround" so they provided alternative to receive 5 second bars that are created on their end instead of 0.2 second accumulated ticks. These 5 second bars are aligned with their clocks, but IB plugin does not use that because this means that giving up resolution below 5 seconds.

1 Like

@PanoS Thank you for the reply, and for the reminder. I did know about that setting and thought I had it set to zero, but alas! That was a different database. It will likely be a bit closer now, but I'm sure not perfect as suggested by Tomasz.

@Tomasz Thank you for your insight. Last night I thought about my question again and realized the idea of pulling from the historical bars every second wouldn't work on many levels, the first being rate limiting (and then obviously speed as you mentioned. It was a dumb idea :grin:

I did not however know about the 5 second bars. All of my systems take trades on the next bar after signal, so this wouldn't be a terrible option for me. But I'm also not going to ask that you update the plugin for this very specific use-case. :wink:

If anyone else has tips for working in short timeframes, please do add to the discussion!

If they extended this functionality not only to 5 second bars, but to any user-definable bar compression, it would be useful and in fact in the style of QCharts API that was considered most "developer friendly".
On the other note, I rechecked TWS API today and they added quite a few things, for example tickString that can hold timestamp (in theory it should solve clock sync problem).
I don't make any promises, but will look at the changes for possible enhancements of the plugin.

4 Likes

Thanks much @Tomasz for the insight. I know you are a busy man, and thank you for whatever you are able to facilitate.

I'm really starting to see a need for this settled(?) 5 second data feed. Today my system took three trades due to bad streaming data, after backfilling the signal was not true.

I'll reiterate my original question: Does anyone trade these sub-minute timeframes with the current IB plugin? If so, do you have any tips?

All I can think of is attempting to smooth the data in some way to minimize the impact of bad streaming data (and hoping I can still get valid signals). I'm not sure if this information applies since IB wont deliver raw ticks.

2 Likes

Until we get to use new TWS supplied timestamps, you can try syncing your computer clock better. Some ideas can be found here

1 Like

@Pinecone - I am seeing this issue as well on minute bars, and it has been very frustrating. Have you been able to find a suitable solution? My system clock is synced every 15 mins but I still have problems with historical trades appearing/disappearing whenever the data is reloaded.

It does not happen everyday but frequently enough where it is causing problems.

1 Like

No solution unfortunately. I've focused my energy on eod for the time being.
I suspect it would be a simple change if the data plugin were open sourced. Perhaps nudge Tomasz to do so, as I have done :wink: :wink:

Going to try a better NTP solution than what I was using for my system clock @Pinecone . Will report back with results.

I use the following network time software https://www.timesynctool.com/
It's free and automatic update time can be configured (every second/minute but 15 min is recommended)

By the way it seems to me that if a system is so sensitive to timestamp it's probably too fragile for using it. Even if quotes were perfectly timestamped, there's a short lag introduced by streaming and this may influence your system (hint: it shouldn't !) Also remember that a real-time stream can include reported block quotes at any time...finally although it may be interesting to switch to another data provider (ms precision for some) it won't change a system sensitivity to a few wrong/laggish quotes.

The problem I was having was that the backfilled "settled" data was drastically different from the live streaming data. It's possible the system was too fragile, but then I did test it with various lag amounts and it still held up.

For me, I was missing a majority of my signals all together. It was that different. Then I would backfill and all my missed signals would appear, just to laugh at me :grin:

I didn't do any elaborate clock syncing. I would really love to trade settled data though, that to me is ideal, and if we can get settled data every 5 seconds, that sounds great.

1 Like

You need to be aware that those 5 seconds bars are not necessarily "set in stone".
Backfill done later can be different anyway because regulations allow late reporting.
Trades can be reported later (upto 10 seconds after they happened) see:

https://www.finra.org/filing-reporting/market-transparency-reporting/trade-reporting-faq#102

1 Like

I had a suspicion this would also be true. But regardless, IB has implemented these "settled" 5 second bars for a reason. Likely because a good number of clients were having syncing issues. It's definitely a topic I see in these forums a lot, and we are just a small community.

If we can get closer to the goal with a line of code, and without elaborate syncing schemes that still don't seem to deliver for everyone, again, that would be great.

3 Likes

In my mind it is not possible to come closer to the IB RT ticks because

  1. It seems IB is not direct connected to the exchange and use GFIS as data broker
    https://gfis.info/
    ... this is reported lower right corner in my TWS client
  2. I have a latency around 160 ms to the IB hub and they are synced worldwide between their "IB cloud infrastructure"
    https://www.interactivebrokers.com/cgi-bin/conn_test.pl
  3. I don't know in which way RT ticks are delivered/delayed to IB and in which way IB transfers these ticks to the "access data hubs" and then push to TWS
  4. "Some kind of HFT" is difficult in such a timeframe and i will take slippage on every scalp trade, because to seems i don't trade on real RT ticks
  5. I checked out IQFeed, Lenz&Partner for a test and think i got some more better quality, but also a latency of arount 140 ms
  6. I use NTP as timesource
    https://uhr.ptb.de/

Yes you are right, "IB Backfill" is really different from IB RT Datafeed. So i did some data export for daily RT ticks on the DAX without IB Backfill and setup my own historic database and tried to backtest and devekop signals on this data ... In my result i mostly got slippage of around 1-3 points via Paper Account and also in my approach to trade the index life.

Finally my research is ended up to use range bars, i trail this IB RT Datafeed in the defined range and let me kick out from them ... so i have to life with the situation or have to change the Data Provider

Sounds like we have similar experiences.

I want to clarify, I am in no way doing any sort of HFT thing.
I was trading the 5m chart, but certain signals were generated on the 5s timeframe. Due to discrepancy in data, those signals never came.

The true goal here is to find out whether or not "settled" 5s bars are any closer to backfill reality.

@alligator - I agree with your sentiments. I'm not looking for perfection, but just to tighten things up a bit. There is no way to get rid of the issue with the late reporting quotes that will need to get backfilled that Tomasz referred to.

I was using NetTime but have found more success with the Meinberg NTP package which is a true NPT solution. This has seemed to get rid of the major issues with timestamp issues between my trading computer and TWS and made things much more stable and consistent.

1 Like

Glad to see you found a solution to your problem @vmonkey .
However note there's nothing more in Meinberg compared to NetTime except file size :wink: Improvement could be due to a predefined list of servers, maybe closer to your location or less busy. pool.ntp.org is a good start for everybody (and the basic Windows Time service can be configured to use it).

By the way as you mention Meinberg, I forgot to mention that if for some reason you really need precision it's possible to use a time receiver, either based on RF or GPS but it's a bit pricey and overkill imo (and in practice everything considered precision is no better than 1ms on Windows if it's connected via ethernet, a card is better though).
For this kind of problem internet time servers are good enough as precision is dominated by the datafeed's lag. On my system logs show that time is rarely adjusted for more than 10ms.

PS: I don't see any practical use to IB 5s historical bars, as it's not realistic it can't be used for backtesting.

Windows Timers don't offer 1ms by default. 1ms resolution is only for demanding multi-media applications (such as Digital Audio Workstation) that specifically request multi-media high resolution timer.
Under normal circumstances the default resolution of Windows clock depends on hardware interrupt that can occur anywhere from every 10ms (Windows NT, Windows XP, 7, 8 and Windows 10 under some circumstances), every 15.6ms (Windows 10), to every 55ms (the worst case in Windows 9x). Windows changes interrupt frequency depending on power saving settings and whenever other applications are calling multimedia APIs (timeBeginPeriod).

1 Like