How to speedup candle formation from the RT plugin streaming data

[karthikeyanp]

Running Amibroker 6.30.5 Professional Edition 64 bit licensed version.

I trade on 5 min interval and I have set my database Base time interval as 1 minute.

In order to ensure tick data is fetched from the plugin for all the 81 symbols and updated into the database, I have automated a simple screener (using Analysis explore OLE object) to be run separately every 30 Secs. I have enabled - wait for Backfill for this explore.

The screener (Analysis explore on 1 minute timeframe) runs at 1st Second of a minute and 30th Second of the minute. I have presumed here that frequent scan/explore will enable periodic fetching of RT data and updating into database instead of fetching/updating it in bulk at the start of the 5 minute timeframe.

In order to measure how many seconds (after the start of the minute) it takes to update the database fully for all 81 symbols, I ran scans (1s 3s, 5s, 7s) and found out it takes a minimum of 3 secs and maximum of 7 secs to fully update past candle information for all symbols and the time delay varies depending on the data volume/market timings.

Screenshots below depicts scan state at 1 second, 3 second 5 second and 7 second after the minute. Notice the symbols that show wrong previous bar timestamp at the top and this list gradually reduces and the scan at 7th second shows all symbols with the last ‘minute candle’ fully updated/completed and the previous bar and current bar timestamp aligns with the current time.

Exploration at 1st second (run @ 15:25:01:007): Almost all symbols are still showing close price of last but one minute (15:23:59). I’ve highlighted the timestamp (15:25:01:007)on the auto screener batch command window for reference.

Exploration at 3rd second (run @ 15:25:03:571): Almost 10% of the symbols are still showing close price of last but one minute (15:23:59).

Exploration at 5th second (run @ 15:25:05:180): Atleast 2 ( to 4) symbols are still showing close price of last but one minute (15:23:59).

At the completion of 7th second (after the minute) all the symbols get updated and show the close price of the last completed candle (in this case (15:24:59).

I have referred to this post, which addresses most of the questions related RT data feed, thanks to @term, @milosz, @sean and of course @Tomasz.

Architecture for taking Amibroker /IB live to large universe of stocks on intraday data?

I completely agree! This is great stuff and I wish there was more of it frankly. The knowledge of how the “Gears” work aid in going from playing with AB and conducting research, to real-time application. I would even add to that, will help keep strategies researched with the practicality of how it might actually be implemented et al. My personal angle with AB as it were, was to maybe use it as a from end graphics and data management, storage (historical and real-time and give that access to…

I have referred to other posts on this subject as well but couldn’t find a direct solution. Maybe I missed something.

I have RT subscription for 225 Symbols.

I have added the 81 Symbols in RT quote window and I can see it updates the data/quote columns for every tick that it receives for every symbol.

My screener exploration runs well within 0.15 seconds on average.

I don’t open any charts during market hours.

I'm running Amibroker on dedicated internet with almost no packet drops and minimum latency.

My 1500 line inefficient code (Strategies + Trading system) takes less than 3 seconds for backtesting for almost 1 year data of 81 symbols on 5 min timeframe. Thanks to superfast AmiBroker.

But the tick data append to the database from the plugin for 1 recent bar takes 3 to 6 secs for the same 81 symbols.

I believe I’m missing some setting that's causing this delay. Or maybe the plugin is not feeding the data fast enough for the data append call from Amibroker.

(And the reason for me posting this post is my earlier post which was the actual impact I faced, ie., delayed Buy Signals, because of the candle formation issue.

Buy signal missed/delay because of candle formation delay )

I guess I dint ask the right question in that post and just simply detailed my issue and understanding.

So I have attempted to ask the right question.

How to speedup candle formation/completion from RT plugin streaming data?

Has anyone faced such an issue? Any settings/solution that could help speed up the process?

Any other thoughts/inputs are welcome.

Thanks,
Karthik P

[Tomasz]

What DATA SOURCE exactly is it?

Unfortunately your question isn't clear enough and does not provide all necessary details to give you an answer. Please follow this advice: How to ask a good question

I guess this is some 3rd party (unknown) unsupported data source.

When you use AmiBroker supplied plugins, you DO NOT HAVE to do anything. All candles are updated automatically as new ticks come, without need to do ANYTHING user-side.

If you are using 3rd party vendor, complain to data vendor/ plugin vendor. They have to fix their plugin. It is plugin vendor task, not yours.

[karthikeyanp]

Hello Tomasz,

Thanks for your time and response.

The Plugin I use is Nimble DataPro from Global Data Feed for India-NSE.

I’ll attempt to explain again crisply.

There is a delay of atleast 3 to 7 secs for a 5 min candle to be completely formed after the end of the previous 5 timeframe.
For Example, for a given symbol, the 9.15 candle forms fully anytime between 9:20:03 and 9:20:07.
( I use ‘Start Time of Interval’ setting in Intraday Preferences)

The RT Plugin has the latest data (verified it manually for selected symbol using getRTData (“Last”) and the broker screen) , but the time it takes to append the latest data to the database is the reason for the delay.
In one of the forum post (When is a bar complete? - #2 by Tomasz) you had mentioned that the Last bar is incomplete until the new bar arrives. In this case the tick data for new bar is already in the RT quote window but the last bar is not yet complete as per the scan. And that is the delay I’m trying to highlight here.

The charts refreshes with each tick data promptly. So there is no issue there.
But I don’t use charts. I use only auto repeat scan for approx. 80 symbols.
Ans that's where I see the delay.
I understand there could be latency issues with the RT data. But the issue is consistent and doesn't seem to be caused by latency as GetRTdata ('Last") shows the correct last traded price.

I have collated the screenshots into a single frame with an example Symbol – POWERGRID – for which the last bar is not yet complete even after 5 secs post the last 5 min timeframe complete. When I run the scan at 7th second after the last 5 min time frame, last bar is complete for almost all symbols.
( For this example illustration I have used ‘END Time of Interval’ setting in Intraday Preferences so that the I can highlight the delay easily).

I have highlighted the issue to the data vendor and their claim is that the streaming tick data is available in the plugin for Amibroker to append to the database.

Please let me know what more information would help to get to the root cause. Would any back-end trace or log help here.

Thanks,
Karthik P

[Tomasz]

Complain to DATA VENDOR / PLUGIN VENDOR.

Data are delivered by data plugin, NOT AmiBroker. The PLUGIN controls when data are refreshed.

All those Indian vendors have NO CLUE on proper programming.
Properly written plugin MUST SEND proper notifications described in this post

Chart is not updating in GetQuotesEx

Let me reiterate what was written already a couple of times. As written in ADK docs and numerous posts you have to send proper message: SendMessage( g_hAmiBrokerWnd, WM_USER_STREAMING_UPDATE, (WPRAM) Ticker, (LPARAM) &recentInfoStructureForGivenTicker ); You must send it the way it is shown above, with proper arguments in WPARAM and LPARAM fields. Also RecentInfo structure (which must be separate for every symbol) must be populated properly as shown in a number of examples in the ADK. Study Q…

[karthikeyanp]

Hello Tomasz,

Thanks for the response.
I have a open ticket with them. Will escalate further.

Thanks,
Karthik P

[karthikeyanp]

Hello Tomasz,

As my current RT vendor, Global Data Feeds is slow to respond, I'm checking out alternatives.
I explored eSignal, but their subscription cost looks way out of my budget.
Is there any other Amibroker Certified 3rd party plugin for RT streaming data for Indian market?

Thanks,
Karthik P

[Tomasz]

Did you try Interactive Brokers? If you trade with them you would get data free or very cheaply.

[karthikeyanp]

Hello Tomasz,

I had a detailed discussion with one of the founder/developer of Global Data Feeds on this issue.

As NSE provides provides Tick by Tick Data via UDP multicast (MTBT) method, the data vendor enumerates all ticks at the close of the minute, package it and send it out.

Though this process of enumeration and packaging takes only milliseconds past the minute, for some Symbols some of the ticks arrive late from the exchange and hence, the last ‘minute’ timeframe data for those symbols are delayed and so the backfill is delayed for those symbols.

In order to overcome this issue, they have advised me to change my base time-frame to Tick, but in which case, I believe I’ll be running the risk of missing ticks at the time of decision making at the start of each minute.

So I have decided to retain 1 minute has the base timeframe and wait till back-fill / last-bar completes for all symbols and then generate the signals.

Please advise if there is any different/better approach.

(I’ll checkout IB as well, thanks)

Best Wishes for the New Year..!

Thanks,

Karthik P

[Tomasz]

Regardless of whenever NSE sends using UDP or TCP/IP, each tick should come with timestamp and they should use timestamp to correctly update minute bars. Also there is no obstacle in updating already existing minute bar if tick arrives late. That is all the duty of data vendor.

[Sean]

Curious, seems like a late quote, meaning timestamped earlier than other that have already arrived, could cause a bad tick (if not timestamped and thus added to current bar) or in the event it has a timestamp and then is added with an older time stamp, thrown into the data series, could cause a signal to appear/disappear that had already been generated after the fact? If that can happen, it might be nice to have a switch to not allow any late (out of order) quotes.

Or I have this wrong with regards to how it is handled? Just a thought.

[karthikeyanp]

Hello Sean,

I believe you are right. It is indeed late quote which the RT provider is claiming to be beyond their control.
Hence I have decided to accept the couple of seconds time delay instead of deciding on potential repainting signals.

Thanks,
Karthik P

[Tomasz]

@Sean you can't fight with facts of life. By rule there is 10 second window to report trades 10-Second Rule Adopted for Reporting Transactions

Depending on timestamping method it may be accounted for past or current bar. Timestamping of incoming ticks in RT feed is not controlled by client software.

[karthikeyanp]

Thanks Tomasz, That was informative.

[Sean]

@Tomasz Not an all-out brawl with the facts of life, just a debate/shoving match.

Wishing you a healthy, safe, happy and prosperous New Year!

[RPS]

Under Database settings have you selected Exchange time or Local Time, select thats recommended.