Optimizing Real-Time Data Plugin for Multiple Tickers

I'm developing a data plugin where the incoming data is received as a stream. On the C++ side, a separate thread handles the data reception. To update the GUI in Amibroker, I send a WM_USER_STREAMING_UPDATE message from my DLL.

The GUI and chart updates work flawlessly when new data arrives. However, the issue arises when I need to monitor multiple tickers, say 10 instead of just one. As new data comes in for each ticker, each one sends its own WM_USER_STREAMING_UPDATE message to the MFC side. This results in Amibroker calling GetQuotesEx multiple times unnecessarily. Consequently, the GUI updates become more resource-intensive, especially when market updates are coming in quickly.

According to Thomas' posts and the ADK manual on GitHub, the recommended approach is to use ::SendMessage(g_hAmiBrokerWnd, WM_USER_STREAMING_UPDATE, 0, 0);, which is what I've implemented. I considered using ::SendMessage(g_hAmiBrokerWnd, WM_USER_STREAMING_UPDATE, (WPARAM)ticker_name, 0); in a manner similar to real-time quotes, hoping that Amibroker would then only call GetQuotesEx for the specific ticker that triggered the WM_USER_STREAMING_UPDATE. However, it appears that this is not the case, as Amibroker still calls GetQuotesEx every time the WM_USER_STREAMING_UPDATE is sent.

I'm looking for a way to make Amibroker call GetQuotesEx only for the ticker that has received bar updates. Is there a method to achieve this? Thank you

You are getting calls for all symbols because you are NOT telling AmiBroker which symbol has updated data. Sending WM_USER_STREAMING_UPDATE with both params set to zero means "hey something is updated, but I won't tell you what".

You should be sending WM_USER_STREAMING_UPDATE with both fields set to NON-ZERO

  • wParam set to (LPCTSTR) pszSymbol (pointer to symbol name), and
  • lParam set to (RecentInfo *) &RecentInfoForGivenSymbol

This is essential to fill BOTH params, not to leave them at 0 and not just one as you did.

If you set those params to zero you don't tell AmiBroker which symbol received data and this has implication that neither RealTime Quote nor Time&Sales can display any update.

What docs say is that you can use WM_USER_STREAMING_UPDATE with wParam and lParam set to zero only as "lazy" notification (in times where NO streaming data really occur, such as weekend). This will tell AmiBroker to perform "lazy" update (not more than 1 per second).

GetQuotesEx might be called in different occasions. These calls are async and done on AmiBroker discretion. They are not sent in direct immediate response to your WM_USER_STREAMING_UPDATE. You are NOT controlling AmiBroker from the plugin. It is the other way round, AmiBroker controls what and when it needs something from the plugin. GetQuotesEx is called when AmiBroker decides it needs data for particular symbol.. When AmiBroker does not have any need to display or analysis running particular symbol, you might even not get GetQuotesEx at all. It is totally upto AmiBroker to decide.

All you need to do from the plugin is to send a hint that you have fresh data for particular symbol. That's all. Don't overthink it. AmiBroker handles the rest.

You really need to go back to GitLab, scroll down the docs and see the workflow diagrams I prepared recently, GO TO: README.md · master · AmiBroker / AmiBroker Development Kit · GitLab

The diagram there shows all communication in detail. Implement everything as it is shown there and you will be good.

Final note, it is really needed to read the docs (especially programming docs) very very carefully. Every sentence counts. There are single sentence advices like this:

The plugin should examine the nLastValid bar as passed from AmiBroker and only request missing data from external data source (from last valid bar until current date/time)

that are super crucial. Using nLastValid to fill ONLY missing data is absolutely crucial for performance. Your plugin MUST NOT fill entire data array without reason. It should only fill missing data. That's the key to performance. Properly written plugin spends not more than single-digit milliseconds in GetQuotesEx.

6 Likes

Thank you, Thomas, for the thorough and detailed answer to my question. The manual (README.md) is indeed very detailed, and the sample codes provide ample guidance.

What you mentioned about setting both wParam and lParam to non-zero values is crucial; I was sure AmiBroker must have that feature. I had reviewed the source code of all the plugins and went through the manual multiple times without finding success in this aspect.

Regarding efficient data request handling, all your points are valid, and that’s precisely how I manage it—I only update the necessary values using nLastValid and nSize and I only send WM_USER_STREAMING_UPDATE when plugin receives updates.

In my specific use case, I use WebSocket connections, so new data arrives whenever there is an update. I have several background threads depending on the number of connections I have, and they work together to keep the data ready locally for AmiBroker to be requested. The workflow is as follows:

  1. WebSocket receives data for a ticker.
  2. My plugin updates its own candle data locally (still no connection to AmiBroker).
  3. Then, ::SendMessage(g_hAmiBrokerWnd, WM_USER_STREAMING_UPDATE, ...) is called each time new data arrives and is different to last time
  4. AmiBroker immediately calls back GetQuotesEx.
  5. My plugin then passes the new data to AmiBroker.

The markets are closed right now; I will modify the code to pass both wparam and lparam and test it first thing on Monday.

For anyone interested in writing data plugins, I heavily use Boost Asio, Boost Json, ixwebsocket libraries, and the new feature of C++ coroutines, which makes asynchronous programming extremely easy. Using these libraries, or similar ones, along with a solid understanding of how AmiBroker operates, one can write efficient data plugins.

To be honest, it took me 2-3 years to reach this level of understanding of how things really work in AmiBroker—things like arrays, stop losses, backtesters, custom backtesting, plugins, etc. I went through many cycles of thinking I understood something, only to come to a new understanding of the same concept later. Things aren't complicated—they're just different compared to Thinkscript or Pinescript; compared to the other options, AFL gives you superpowers.

4 Likes

Congrats to your efforts.
It all boils down to how much proficient one becomes eventually in c++
I have often pulled my hair, almost cried and decided to abandon it.

Currently, I have a minimal prototype which is work in progress.

My approach has been contrary to yours. Plugin is lightweight websocket client only. The other stuff is in python because that was the origin of the problem.
So, there is

  1. Relay server always running (python based)
  2. Clients connected with a code, either SENDER or RECEIVER. So all brokers provide a python side client and therefore I need not rewrite much.
  3. Only AB RTD plugin for now connects to relay server as RECEIVER
  4. So SENDER clients connect when required and dump to stream, if data is broadcasted, AB just consumes it.
  5. For now the only thing not worked on is Backfill / intraday historical data. To fix this, one needs to store quotes inside the plugin during all intraday ticks. For backfill, another mechanism. My C++ skill now is just average.

AB doesn't request data if Ticker is not in any context. So for example I need minimum 1min bars. To keep GetQuoteEx() running, i put a 1min Exploration so the AB DB keeps bars updated.
If you have a chart with 20 symbols and no exploration, then EVEN if you send WM_USER_STREAMING_UPDATE, symbols not in context will NOT call GetQuoteEX().
(This is my observation, AB is behaving efficiently as Tomasz wrote)

So none of the heavy BOOST/ixwebsocket libraries. The client websocket is C++11 on STL with windows headers.
I initially started with nlohmann json but using RapidJSON which is again only C++11 and STL. Socket code runs on just 2 threads, and consuming with timer like the OnTimerProc()

RapidJSON is extremely fast in parsing Benchmarks (cppalliance.org)

Coming to your problem of WM_USER_STREAMING_UPDATE

Like in QT example, there is a global Array g_aInfos so when I pass its reference it works, but if i use ri because i think it is "out of scope" it doesnt work.

//here i is the index of that ticker
::SendMessage(..., (WPARAM)&g_aInfos[i].Name, (LPARAM)&g_aInfos[i] ); //works

::SendMessage(..., (WPARAM)&ri->Name, (LPARAM)&ri); // does not

If you get this right, then you get realtime window ticking nicely.

image

2 Likes

You can this thread.
I use the py server script to generate dummy quotes for testing.

Request for Generic Websocket Data Plugin - Plug-ins - AmiBroker Community Forum

I have been there - C++ resembles an onion, with layers upon layers of complexities and issues—until one day, it magically transforms into an apple.

Regarding nlohmann, I chose not to use it because I try to stick with Boost whenever it provides the same functionality. However, I am aware that nlohmann is faster when it comes to parsing, so it's a valid choice depending on the requirements. My approach is to address problems using the tools I already have - thanks for mentioning that.

Overall, it appears that our solutions are quite similar; the primary difference is that my relay server is implemented entirely in C++. I employ multiple threads to manage communication with my data provider, and the data plugin interacts directly with these threads. In contrast to your approach, I opted to avoid Python entirely, as the Global Interpreter Lock (GIL) has caused issues for me in the past - numerous times, with no way around it. Despite the availability of coroutines and multithreading in Python, the GIL limits true multithreading. Given my need for multiple threads to handle communications, I found C++ to be a more suitable choice. With this setup, I have managed to keep the entire S&P 500 tickers updated in real-time on my system.

Regarding your observation:

AB doesn't request data if Ticker is not in any context. So for example I need minimum 1min bars. To keep GetQuoteEx() running, I put a 1min Exploration so the AB DB keeps bars updated.
If you have a chart with 20 symbols and no exploration, then EVEN if you send WM_USER_STREAMING_UPDATE, symbols not in context will NOT call GetQuoteEx().
(This is my observation, AB is behaving efficiently as Tomasz wrote)

I do not believe that is entirely accurate. While AmiBroker is efficient, it is not necessary to use exploration to keep the bars updated. If you are sending WM_USER_STREAMING_UPDATE and the bars are not updating, it might be due to the "Real-time chart refresh interval" setting. This option controls how often GetQuoteEx is called. If you set it to 0, the chart will update every time you send WM_USER_STREAMING_UPDATE. Otherwise, if this setting is, for example, 5 seconds, then even if you send the message 100 times, the chart will only update once every 5 seconds.

You can find this setting here: Tools -> Preferences -> Intraday -> Real-time chart refresh interval.

As for your issue with SendMessage:

::SendMessage(..., (WPARAM)&g_aInfos[i].Name, (LPARAM)&g_aInfos[i]); // works
::SendMessage(..., (WPARAM)&ri->Name, (LPARAM)&ri); // does not

SendMessage operates synchronously; the DLL code is blocked until WM_USER_STREAMING_UPDATE is consumed on the MFC or AmiBroker side. Therefore, as long as ri is defined within the same scope when SendMessage is called, it should function correctly. If it does not, I would recommend verifying that ri is valid and accessible at the time of the call.

If you download AmiBroker version 6.0 or 6.1 and set up Visual Studio for debugging, you can set a breakpoint where you send your WM_USER_STREAMING_UPDATE to see what is happening with ri. This can provide more insight into why the reference may not be working as expected.

1 Like

It is accurate. If symbol isn't used anywhere (no chart, no RT quote window, no Analysis, nothing) then AmiBroker will not request data for such symbol.

1 Like

You’re right. If it's not used in any context, then AB does not request it. I misunderstood his point because I assumed that you have at least one active chart, and that due to this chart, when you send WM_USER_STREAMING_UPDATE , which then triggers the updates I mentioned in the previous post.

  • My apologies for any confusion — replied and edited my post using a touchscreen device, so it turned out a bit messy

yes, I read all the posts. Testing with AB 6.1 x64

Just another question, how big is your DLL? The AB sample QT from Gitlab builds to 280KB for 64bit DLL. This is with VS2022 and V143 for MFC/ATL/other components. It works properly but have 64bit become so big?

I opened it as CMAKE project and all my other iterations also work fine but with RapidJSON and a small Websocket header its 550KB

edit: @Tomasz your opinion is also welcome

Long gone are the days when Visual C++ produced executables as small as 5KB (Example Candle.dll plugin size in 32-bit) . Welcome to a new world.

1 Like

My plugin is 1,089 kB. In my case, the size is mainly due to the use of libraries like Boost.Json and Boost.Asio, which involve extensive template programming and inlining.

1 Like

Maybe the next version of 'candle.dll' plugin should be named 'full-on-bonfire.dll'

I tested the plugin after making the changes, and I can confirm that the issue was not using both WPARAM and LPARAM - RecentInfo must be used as well. The GetQuoteEx is now called only when it should be.

Ok, a small update. I don't how many pitfalls one must go through.

Sometimes maybe we dont know the switches given in CMAKE work or not as the hard working MSVC team left many bugs around.
One such is Zc:__cplusplus and so on.

Unrelated to that, in VS2022, the better thing that worked is to use the VS UI which configures CMakeSettings.json. So instead of playing around with files directly, using its UI tool enforced config.
So all the while the debug version DLL was large in size. (obvious for debug)
CMAKE release version is quite lean at 112KB vs 700KB debug.
Hope it helps some noob like me :slight_smile: