Data access speed, not CPU, is often limiting factor, was: AmiBroker perf

[topmarx]

Hello,

So I decided to move forward with the previous thread discussions and upgraded my desktop, I moved from a 4th Gen i7 (8 Core no threading) to the Ryzen (16 Core 32 Thread). In addition, to support the new CPU added all the latest parts, a top-end motherboard, new RAM, and M.2 SSD HDs.

And ... disappointingly, I barely note any performance improvements, in fact, might even be slower. I have a number of strategies that I am familiar with back-testing and optimization and comparing the speed across each machine. During the test, it registers the 8 threads vs 32 threads in the analysis window.

I notice CPU usage barely goes above 5%, see stats below:

image|665x500

Anyone has anyone seen similar or contradictory results?

Understand there is a decreasing performance with cores vs threads, however, even ignoring the threads, the processor is 5 years newer and has double cores. I would expect at a min it should half the time is taken with the old CPU.

I noticed decent improvement when I overclock the clock speeds but really should not need to do this.The only part I can think of is RAM, as both machines have 16GB but doubt 32GB would make any difference in this use case.

Any thoughts on how I could improve raw hardware performance (non-code options)?

[Tomasz]

Unfortunately your post does not include crucial information required to help you. Please follow this advice: How to ask a good question

CPU speed is irrelevant if your data can't be delivered any faster. If you don't get improvement over previous setup it just means that in the OLD SETUP you were already waiting for data.

Are you using 3rd party data source? For example Norgate?

If you observe what you described it typically means that you are WAITING FOR DATA.

Slow optimizations with Norgate Data

Hello, I have subscribed to Norgate US Stocks Diamond Package for the very first time, everything works great except for the speed of optimizations. I realize that using a third party plugin is going to slow down the process, but sometimes it's so slow that it makes the whole task quite unpractical. I would like to know if there is somethingo to be done to reduce time. In regards to Amibroker, I am not sure if it is possible to calculate less metrics in the backtest and if that is going to sp…

FIRST THING is to read existing documentation
http://www.amibroker.com/guide/x_multithreading.html

And then CAREFULLY READ THIS:

Limits of multithreading

The screenshot that you have included is NOT ENOUGH to give you the answer. It just shows that only small portion of symbols (1021 out of 33686) in in the cache, which may indicate wrong settings.

As I wrote many many times on this forum CPU can only be busy IF your formulas are complex. If your formulas are trivial, CPU will WAIT FOR DATA, especially if they are delivered by 3rd party source that are typically much slower than AmiBroker native database.

[Ector]

This year I’ve tried four different CPUs: 10900K, 5800X, 5950X and 11900K in quest for better Excel performance. And surprisingly only 11900K was able to produce speed what I was looking for

Ryzen 9 5950X was the main disappointment because most Windows applications are using only few cores and CPU load is about 5-10% in most scenarios. Also I didn’t like high temperatures in light work

I am still working on database before transferring it to AmiBroker, so can not give you feedback in AB experience with this processors. But seems like something bottlenecking your performance. You can try with local database for example and compare results

[topmarx]

Thanks Tomasz, this makes sense, I have read the info on threading.

I am using Norgate Premium data. The waiting for data option, appears likely, as the CPU utilization is so low. Is there any steps I can take to improve this, or is this Norgate limits?

Per other threads I have set the cache to half RAM, so 8GB, didn't make much difference, and also put settings to load all 32000 symbols to cache.

Thanks Ector, I was wondering whether there might be something in the AMD vs Intel. I do some testing with a seperate local DB vs Norgate, and update with performance.

[Tomasz]

Regarding AMD CPUs they are good:

What is the maximum number of CPU cores AmiBroker supports for multi-threaded optimization?

I just installed update on Version 6.39.1. Without problems. Has run without errors so far. Speed on Threadripper 3970 X with 32 cores, 64 threads about 120% faster than version 6.30 on the same processor with 32 threads. Perfect Tomasz .... many, many thanks .....

Regarding Norgate, see previously mentioned thread:

Slow optimizations with Norgate Data

Tomasz, Thanks for your additional comments. I tested both proposals, but I got similar results. I guess I'm doing something wrong, but I can't figure out what 1) Data source: [Image 1] [Image 2] 2) In-Memory Cache (before this test, the values were 10.000 symbols and 526 MebaBytes) [Image 3] [Image 4] I tested the same code: #include_once "Formulas\Norgate Data\Norgate Data Functions.afl" function IndexConstituentTimeSeries(indexName) { staticVarKey = "IndexConstituentTimeSeries_" …

Regarding helping you, I need to re-iterate this again, please follow this advice: How to ask a good question

without FORMULA, without description on WHAT you are doing (precisely, not vaguely - tell exactly if you are running backtest or optimization, on how many symbols, how many data bars, how many optimization steps, etc, ALL details), without screenshot from "INFO" tab, one can not give you any answers.

[beppe]

@topmarx, looking at your post's linked image, I see that you also have many static variables.

As you surely know (and is also clearly explained in the multithreading section of the manual), if in your formula you have a section of code included in the following conditional:

if (Status("stocknum") == 0)
{
  // StaticVarSet(...)
}

all the code inside it is executed in single-thread mode, so this too could be a factor to take care of when comparing performance.