What is the maximum number of CPU cores AmiBroker supports for multi-threaded optimization?

You can add tags at the bottom of the edit field, see this:

The "Solved" button (checkmark) becomes visible when you click on (...) button left to the "Reply" button

@Tomasz I saw your post in another thread explaining why I could not add tags to my post. I’ve not yet earned the privilege.

Just wanted to check whether we may give preferences to use number of threads...
Wish to know how to check a change between Celeron Dual Core, i3 & i7 processors

@pushkan I'm not sure to understand what you mean by "check a change", but in recent versions of AmiBroker you can use the following directive:

#pragma maxthreads 1

(change the number as needed)

More, in general, under Windows this kind of task can also be done independently of the application using the "Processor Affinity" settings. But many applications developers recommend to avoid such procedures.
(Regarding AmiBroker I think it will be better to ask directly to @Tomasz).

Anyway, if you are interested, please see this article How to Restrict Apps to Specific CPU Cores with Processor Affinity. There are also some utilities, like ProcessLasso (not a recommendation) to permanently set some specific settings each time a process is started.

I did a very limited test with AmiBroker, and it seems to work (even in the GUI AB still prints the number of all the available cores).
I left enabled only one core and running an exploration I got the same result as when I use the pragma with maxtreads = 1.
I mean, that in this test, tracing all the tickers in an exploration I got a list sorted alphabetically.

I do not know, nor tested what happens when you mix the two approaches (#pragma and processor affinity)....

OK let's put it other way round, Pentium Quad core @ 1.1 GHz better as compared to i3 Dual core @ 2 GHz?

oops, :flushed: I got it completely wrong!
While you wait for a precise answer maybe you should take a look at this other thread.

Not all Pentiums and not all i3 are created equal. Different families have different IPC and use different RAM. RAM is important. Clock speed isn't everything. Generally there is NO "general" answer. This is equation with dozens of variables. http://www.amibroker.com/kb/2017/10/06/limits-of-multithreading/

Thanks a lot @beppe & @Tomasz. Indeed a good step for me on the learning curve...

Just to clarify on this thread with current processors such as AMD Ryzen 9 3950X, 16 Cores, 32 Threads. If using professional the analysis would only open 16 threads, one per logical core? And if you have a lot of cash and are very impatient, and have this Ryzen Threadripper 3970X Processor 32 Core/64 Threads.... it would max out Amibroker capability at 32 threads for analysis?

You are reviving old thread (from 2017)

Currently (in 2021) AmiBroker 6.39 allows upto 64 threads per single analysis window. And your Ryzen 9 would use 32 threads and Threadripper would use 64 threads.

1 Like

Hi Tomasz,

I use a computer with AMD Threadripper, 32 cores.
Amibroker I use version 6.35.
Version 6.39 can handle up to 64 threads as I've read.
On a computer with "only" 32 cores, hyperthreading would have to be used to process 64 threads.
Hyperthreading is slower than processing only through physical cores if I understand it correctly.
Is there still an advantage when using version 6.39 (32 cores, hyperthreading) compared to version 6.35 (32 cores, no hyperthreading)

Sorry for this simple question. I've read a lot of posts here as well as your article
-> AmiBroker Knowledge Base » Limits of multithreading

But it is important to me. I only test on American indices with historic constituents from Norgate and only use Walk Forward. Russel 1000, for example, then has over 3000 members. And that is exactly what requires a lot of computing power. Best regards

1 Like

No that is incorrect understanding. Processing using 4x2 hyperthreaded cores is faster than 4 singlethreaded cores. The article AmiBroker Knowledge Base » Limits of multithreading includes results from hyperthreading and they are NOT slower.

If you are using Norgate or any external database, the limiting factor is NOT CPU speed, but data speed. CPUs are orders of magnitude faster than data delivery capacity. Modern CPUs can only execute at full speed if they run entirely from on-chip cache. Even RAM access slows them down.

4 Likes

I just installed update on Version 6.39.1. Without problems. Has run without errors so far. Speed on Threadripper 3970 X with 32 cores, 64 threads about 120% faster than version 6.30 on the same processor with 32 threads.

Perfect Tomasz .... many, many thanks .....

4 Likes

Norgate performs a lot of calculations on-the-fly because we offer multiple price adjustment and date padding mechanisms. This extends to the indicators we offer too, introducing further complexities. As we understand it, right now in v6.39 and below, data access is limited to single-threaded operations.

Our data plugin would certainly benefit from multi-threaded data access mechanisms. Our routines are re-entrant and can handle many simulataneous multithreaded calls as far as we've tested internally. Norgate would certainly welcome higher performance data access mechanisms in AmiBroker.

Cheers,
Richard.

1 Like

Two points:

  1. Data access is NOT limited to single thread.
  2. Throwing growing number of threads does not increase data speed on single shared resource

Ad 1.
You are free to use as many threads as you wish in your data plugin. Our own IQFeed plugin uses 5 threads for example. But it uses threads not because it is magically "faster" but to handle multiple asynchronous socket requests.
AmiBroker also uses multiple threads to access data from many places including its own in-memory cache and static variables storage. AFL functions are also called from multiple threads. Plugin-exposed functions are also called from multiple threads but protected by critical sections because 3rd party plugins don't understand threading or perform worse with threading (OLE based APIs are one of many examples) or won't work at all. It is not only about re-entrancy, for the record, there is a whole lot of other things.

Ad 2.
You read data from single physical device, i.e. DISK DRIVE. No matter how many cores your CPU has, your disk is still ONE PHYSICAL resource that does not scale with threads. SINGLE CPU core FULLY saturates ANY today's disk drive (yes even fastest M.2 SSDs). Even if you already have data in RAM, things are not as one may naively expect. Single CPU core saturates RAM access. This is as simple as that. Not CPU but RAM and I/O bandwidth is a limiting factor. Throwing more threads make things worse when they compete for single hardware resource. And no "adjustments" (multiplies and additions) are NOT enough to keep CPU busy and slow it down. You would need to do whole bunch of transcendental functions to keep CPU busy enough so RAM can keep up.

Quote:

Memory-intensive threaded applications can suffer from memory bandwidth saturation as more threads are introduced. In such cases, the threaded application won’t scale as expected, and performance can be REDUCED.

See also:

and

To get better performance you may take a look at speeding up existing plugin functions that are reported to be bottleneck:

As users reported simply avoiding them using super simple caching gives 14x improvement.

2 Likes

Thank you for reporting Threadripper results. Very nice indeed.

One more small note ... I estimate that at least about 50% of the computing power in my Code is required single threaded in StockNum == 0. Among other things, he puts together the buy signals consisting of 2 variables (from walk forward run) for each title. With an OOS period of 26 years with annually changing 2 variables, a lot of time is consumed here. All the more astonishing is the immense increase in speed with 64 threads.

You can measure the time spend in multi-threaded and single-threaded code with this snippet: Measure phase 1 and phase 2 backtest durations with this code

It is not needed at all. The information is natively included in the "Info" tab, in fact there is a whole lot more information there. Figures are explained in the Knowledge Base:

1 Like

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.