I'm using the ODBC plugin to connect to MySQL and fetch quotes. When I run an exploration with multiple symbols, I can see 12 threads spin up on the desktop but see only one active query at a time on the MySQL server. DB queries are serviced from RAM/cache on the server, so it could be the queries are completing very quickly. But, I'd still expect to see 3-5 concurrent queries on server once in a while.
Does Amibroker use multiple DB connections for the threads in an exploration and across explorations? Am I missing something basic?
Limits of multithreading seems to indicate the data stage is shared before "setup -> execute -> process" can fan out on multiple threads. Does that mean ODBC queries for multiple symbols in an exploration are serialized?
Related app settings and context -
I'm using ODBC plugin, data storage is enabled. Quick AFL is enabled, symbol padding is disabled. I've gone over the "Efficient use of multithreading" & "ODBC" related guides. The DB is running on server grade hardware, serving quotes from RAM, network is far from saturation. Desktop CPU and memory are at 20-50% utilization.
Thanks for confirming @Tomasz. Could I request you to please consider allowing multi-threaded access for just the native ODBC plug-in?
I'm finding ODBC quite fast with Amibroker for my use cases, getting 25ms per symbol during real-time scans. ODBC provides a lot of flexibility and safety especially in regions where the most reliable real-time data sources are provided via web socket. E.g in one of the two countries I trade in, the Amibroker unapproved real-time data vendor requires root + firewall access on my PC. I'd also think this would open up a lot of possibilities for folks with an interest in programing.
If you had data in native database the access time per symbol would be zero, because data in AmiBroker database are directly accessible without any processing via pointer in RAM.
ODBC is NOT guaranteed to be threading-friendly, and is likely to be serialized on driver level anyway , see Microsoft docs:
@Tomazs you're the expert on programming on the Microsoft stack so I'll defer to your inputs on this. I'm not familiar with desktop development.
I did check the MySQL ODBC driver has been multi-threading certified recently 8.0.30 (2022-07-26). Current Oracle ODBC driver is also multi-threading ready. Common thread seems to be the recommendation to not use asynchronous (non-blocking/polling) style IO and have threads block on their individual connection. Amibroker already does spin up threads that block on data, I don't know if that's enough though. I'm sure there're more nuances to this.
I'd love to have the data in the native database, but am not able to figure out the method. I have a real-time data feed via websockets. Is there a way to push that into Amibroker with a 15s-30s latency. Websocket feed has 3000 symbol quotes in real-time, I can aggregate them to any time interval, say 10 seconds per candle. I dont have enough knowledge with the Microsoft platform to write a data plugin though. Would import called every 30s work? When I tried calling import earlier, the UI seemed to freeze for a few seconds every 30 seconds as the small file contained a few latest quotes for 3000 symbols, which is not the structure Amibroker's import seemed to be optimized for. 1 symbol with a even 100K candles completes in a blink of an eye though.
@Tomasz request you to please take a look at the previous message when you get the time. Current MySQL (post 2022) & Oracle drivers no longer serialize queries and are multi-threading capable. Thanks