Already asked on the Q&A website. http://ask.amibroker.org/index.php
I’m comparing the experience by posting the same question here. I must say, Discourse has a slick user interface, but I’m unable to create a tag here, or maybe I’m not doing it properly.
Can someone answer this? I want to try marking the question solved.
The number of threads that actually are launched depends on your CPU and the version of AmiBroker you are using. Standard Edition has a limit of 2 (two) threads per Analysis window. Professional Edition has a limit of 32 threads per Analysis window. In addition to this limit, AmiBroker will detect how many logical processors are reported by Windows (for example a single Intel i7 920 CPU is recognized as 8 logical processors (4 cores x 2 hyperthreading)) and will not run more threads per single Analysis window than the number of logical processors.
@pushkan I'm not sure to understand what you mean by "check a change", but in recent versions of AmiBroker you can use the following directive:
#pragma maxthreads 1
(change the number as needed)
More, in general, under Windows this kind of task can also be done independently of the application using the "Processor Affinity" settings. But many applications developers recommend to avoid such procedures.
(Regarding AmiBroker I think it will be better to ask directly to @Tomasz).
I did a very limited test with AmiBroker, and it seems to work (even in the GUI AB still prints the number of all the available cores).
I left enabled only one core and running an exploration I got the same result as when I use the pragma with maxtreads = 1.
I mean, that in this test, tracing all the tickers in an exploration I got a list sorted alphabetically.
I do not know, nor tested what happens when you mix the two approaches (#pragma and processor affinity)....
Just to clarify on this thread with current processors such as AMD Ryzen 9 3950X, 16 Cores, 32 Threads. If using professional the analysis would only open 16 threads, one per logical core? And if you have a lot of cash and are very impatient, and have this Ryzen Threadripper 3970X Processor 32 Core/64 Threads.... it would max out Amibroker capability at 32 threads for analysis?
I use a computer with AMD Threadripper, 32 cores.
Amibroker I use version 6.35.
Version 6.39 can handle up to 64 threads as I've read.
On a computer with "only" 32 cores, hyperthreading would have to be used to process 64 threads.
Hyperthreading is slower than processing only through physical cores if I understand it correctly.
Is there still an advantage when using version 6.39 (32 cores, hyperthreading) compared to version 6.35 (32 cores, no hyperthreading)
But it is important to me. I only test on American indices with historic constituents from Norgate and only use Walk Forward. Russel 1000, for example, then has over 3000 members. And that is exactly what requires a lot of computing power. Best regards
If you are using Norgate or any external database, the limiting factor is NOT CPU speed, but data speed. CPUs are orders of magnitude faster than data delivery capacity. Modern CPUs can only execute at full speed if they run entirely from on-chip cache. Even RAM access slows them down.
I just installed update on Version 6.39.1. Without problems. Has run without errors so far. Speed on Threadripper 3970 X with 32 cores, 64 threads about 120% faster than version 6.30 on the same processor with 32 threads.
Norgate performs a lot of calculations on-the-fly because we offer multiple price adjustment and date padding mechanisms. This extends to the indicators we offer too, introducing further complexities. As we understand it, right now in v6.39 and below, data access is limited to single-threaded operations.
Our data plugin would certainly benefit from multi-threaded data access mechanisms. Our routines are re-entrant and can handle many simulataneous multithreaded calls as far as we've tested internally. Norgate would certainly welcome higher performance data access mechanisms in AmiBroker.
Throwing growing number of threads does not increase data speed on single shared resource
You are free to use as many threads as you wish in your data plugin. Our own IQFeed plugin uses 5 threads for example. But it uses threads not because it is magically "faster" but to handle multiple asynchronous socket requests.
AmiBroker also uses multiple threads to access data from many places including its own in-memory cache and static variables storage. AFL functions are also called from multiple threads. Plugin-exposed functions are also called from multiple threads but protected by critical sections because 3rd party plugins don't understand threading or perform worse with threading (OLE based APIs are one of many examples) or won't work at all. It is not only about re-entrancy, for the record, there is a whole lot of other things.
You read data from single physical device, i.e. DISK DRIVE. No matter how many cores your CPU has, your disk is still ONE PHYSICAL resource that does not scale with threads. SINGLE CPU core FULLY saturates ANY today's disk drive (yes even fastest M.2 SSDs). Even if you already have data in RAM, things are not as one may naively expect. Single CPU core saturates RAM access. This is as simple as that. Not CPU but RAM and I/O bandwidth is a limiting factor. Throwing more threads make things worse when they compete for single hardware resource. And no "adjustments" (multiplies and additions) are NOT enough to keep CPU busy and slow it down. You would need to do whole bunch of transcendental functions to keep CPU busy enough so RAM can keep up.
Memory-intensive threaded applications can suffer from memory bandwidth saturation as more threads are introduced. In such cases, the threaded application won’t scale as expected, and performance can be REDUCED.
To get better performance you may take a look at speeding up existing plugin functions that are reported to be bottleneck:
As users reported simply avoiding them using super simple caching gives 14x improvement.
One more small note ... I estimate that at least about 50% of the computing power in my Code is required single threaded in StockNum == 0. Among other things, he puts together the buy signals consisting of 2 variables (from walk forward run) for each title. With an OOS period of 26 years with annually changing 2 variables, a lot of time is consumed here. All the more astonishing is the immense increase in speed with 64 threads.