I have a new installation of Amibroker on Windows 10 and Parallels on an IMac Pro. This machine is usually about 3x faster than a desktop i7 under Windows 10. So far all backtests on Amibroker were running fast, since this morning a backtest takes about 90 seconds on an I7 notebook, about 20 minutes on this Apple machine instead of the expected less than one minute. Script (20 lines. wellknown and fairly simple), settings, etc. are all the same. The Task Manager signals 50% load at extremely high power consumption, which is also the same on my I7.
Datasource is a Taipan-EoD database.
Does anyone have any advice on what else can be done here to restore speed?
It has nothing to do with AmiBroker.
You need to find out what happened to your machine. Maybe Windows is running update in the background, maybe your hard disk / SSD develops errors, maybe you have other hardware malfunction, maybe your data provider (Taipan) uses internet connection and your internet connection is slow today. Or maybe you changed some settings or installed 3rd party software (some antiviruses degrade system performance badly)
I would first disconnect from 3rd party data source and try local.
It is your computer and you can see the screen and check things.
One thing to consider: all virtual machines (including Parallels) are always slower than running native. Sometimes the difference is substantial (for example when it comes to graphics).
Thanks for your fast response. Unfortunately the problem lies neither in the hardware nor in the virtual machine. And believe me - before I asked for support I made sure that the devices and tools on my side are working properly ...
Nevertheless just now for performance testing purposes I recompiled a very big ermbedded software project under Windows 10 on both computers. The result: the IMac/Parallels/Windows 10 combination is 2.5 times as fast as the I7 machine with exactly the same tools running.
No antivirus stuff is running in the background. The IMac Pro is a 14 core XEON machine with high speed graphics and 64 GB Ram, where 8 hardware cores and 32 GB RAM are used for the mentioned Parallels instance.
The database is local - as I wrote: same setup on both computers. But to make sure that it is no problem with the data provider I switched to my Amiquote database - the problem still exists.
And most important: Amibroker ran pretty fast already on the IMac. I will now give a reinstallation a try.
You are answering by email and ALL your screenshots are broken because they are left on your google mail account (private), not public. So use WEB interface to access forum and edit your post to include real screenshots.
You are right, Tomasz, my mistake! And my apology ... here the message with images (hopefully!)
Hello Milosz,
thanks for your email from yesterday.
By listing the performance data of my IMac, I didn't want to brag but to answer the claim that the problem was in my computer and that virtual machines were always slower than "real computers". This assertion came from you - and didn't help me. But now I have a new goal: I save my money for the Amibroker 2022 Quantum Edition and the Cyberpunk 2077 ...
To the items of your email:
Checking the "Quickafl"-Box decreased the time for the backtest by 60% - but this is the case on both computers.
Local datasource is selected - but obviously there is the problem!!! As I told I reinstalled Amibroker from scratch (I cleaned even the registry before ;-)). In the first attempt I used the small Dow Jones database supplied by you. The "optimization time" therefore is below 2 seconds for 30 symbols. Then I installed the same data from Taipan (Lenz&Partner) locally and optimized over the same period: First this procedure lasted 51 seconds, after checking "QuickAFL" this time was reduced to 20 seconds. (To avoid further misinterpretations: these times are only reflecting my test case, in practice an optimization should cover more complicated scripts. I would never mind about 10 seconds more or less ...)
This my test script:
=================================
SetPositionSize(5, spsPercentOfEquity);
SetOption("MaxOpenPositions", 20);
SetOption("InitialEquity", 250000);
SetTradeDelays(1,1,1,1);
//IndexSmooth = Param("Index MA Periods", 120 , 10 , 250, 5);
//Index = Foreign("969950","C",True);
//IndexMA = Index >= MA(Index,IndexSmooth);
slow = Optimize("Slow", 90, 20, 420, 10);
SlowRes = HHV(H,slow);
SlowSup = LLV(L,slow);
uptrend = Flip(H > Ref(SlowRes,-1),Ref(SlowSup,-1) > L);
trend = IIf(uptrend,1,-1);
indexMA = True;
Buy = uptrend AND indexMA;
Sell = NOT uptrend;
ExRem(Buy, Sell);
ExRem(Sell, Buy);
=================================
The script has the following "finger print":
Performance monitor:
I read the mentioned articles. As valuable as they are, none of them touches the problem I have to deal with.
Yes, my IMac is good - but as I told the quantum computer together with Amibroker will change everything for me ... I highly anticipate this ...
Maybe you have an idea what could be the problem with the database access (my item 2). I myself have absolutely no imagination what could be the difference between the one and the other database.
I write short posts, but really they are filled with essence, so please don't skip the information I provide. When I quote relevant part from Knowledge Base, it is relevant and should not be ignored.
Now switch to “Info” tab in the Analysis window and you will see this output (this example comes from 4 core / 8 thread Intel i7), all times are in seconds:
So, you should be looking at the "Info" tab after running Analysis
These data are what you should be looking at, not the other stuff that you sent.
Now comes important stuff, don't skip: look at the data time. Most probably it is dominating the time (I was telling that from the very beginning). The explanation of all those figures are in the Knowledge Base article I given.
On a side note, your last two lines (with ExRem) do nothing and can be removed.
PS: As I wrote before, virtual machines are all slower than native. Parallels is no exception. See https://www.tekrevue.com/parallels-11-benchmarks/ Virtualization always adds an extra layer and everything that is extra costs CPU cycles. And for the record, I think that Parallels is great virtualization solution. It is just laws of physics, if you add something it weights more.
I am little bit irritated about the undertone of your message. The problem is obvious: everything is slower in my iMac than on my I7 notebook. Nothing else is shown by the info tab of the Analysis window. The problem is definitely not the VM. Of course a VM is always slower than a ... what?? If I install Windows on an iMac this installation will run much faster than a VM installation. D'accord. But if you compare the iMac/Parallels VM with an I7 notebook the VM seems to be a rocket against a fly. I am software developper like you and therefore I attached two screenshots. Both show the info tab and a compilation window of an already mentioned embedded software project I am currently working on. As you can see the iMac/Parallels/Windows machine compiles the project in 55 seconds, the I7 compiles the same project in 131 seconds. As you can see the iMac optimizes the already known script in 45 seconds, the I7 needs only 5 seconds. Do you really believe that this is a VM problem???
So it is exactly what I predicted from the very beginning.
The source of slowness is data access on your Mac.
Can't you see - this is obvious from the Timings data you posted. Your Mac setup shows that out of 45 seconds needed to optimize it waited for 43.83 seconds for data. That is 97.4% of optimization time spent waiting for data. And this is where you should be looking at.
Your non-Mac machine only needed 3.47 seconds for data access. That is 12 times faster than your Mac.
Data access time is NOT really AmiBroker dependent. The impact of CPU speed on data access time is also negligible. It mainly depends on:
data PLUGIN (3rd party) - turn if OFF and re-check
speed of your disk and location of data (see 3)
speed of virtual machine disk emulation - if data are located across virtual machine file system ("shared" drive)
whenever your data fit into in-memory cache or not (you can try increasing the cache in Tools->Preferences, "Data" so it is larger than number of symbols under test) - but this only applies to second and subsequent runs when data are in the cache. RAM is orders of magnitude faster than disk, so proper cache settings do miracles.
Sorry to say that but software developers are often "difficult" as they are shooting themselves in the foot... why? because they often assume they know everything better already and don't listen to genuinely good advice.
If you don't want to hear the advice - don't ask. If you know everything better already - help yourself. But if you can not help yourself and need somebody to help you, show some respect to person who actually wrote that program and by slight chance knows something about his own creation and spends time on your local computer problem (not my duty) and tries to help you.
Believe or not I have helped lots of people with performance and it was always question of local problem. Either caused by incorrect settings or 3rd party software or (rarely) hardware.
And by the way: comparing to GNU ARM C++ compiler (part of Keil embedded IDE) is not indicative as compiler is simple command line program that is single threaded. So you are comparing apples to oranges (single threaded GNU command line compiler vs multithreaded AB)
This story has brought one idea for improvement (mainly to save time answering similar "issues" in the future) - I think that I can add some kind of "warning" when too much time is spent on data access and/or when people run optimizations on large watch lists with in-memory cache settings being too small and preventing caching.
On the right side of this reply box I found the following text:
Be kind to your fellow community members.
Constructive criticism is welcome, but criticize ideas , not people.
=================================================
I agree completely!
My understanding of kindness, particularly towards paying customers, somehow doesn't seem compatible with the tone in Amibroker Support.
I don't think I could have given the impression that I knew everything better. I'm simply not convinced that 10x slower data access as normal in a specific application is a computer problem when other applications doesn't show similar phenomenons. I can't help but the following benchmark comparison between the two machine environments doesn't suggest in my understanding that the iMac might have a data access problem. During the tests I accessed the same ssd (drive C:) where the Amibroker databases are stored. The &CPU load was approx. 800% during this test, that indicates that 8 cpu cores were busy. I hope that I am not comparing apples and oranges in this case ...
CrystalDiskMark result for i7 notebook:
CrystalDiskMark result for iMac/Parallels:
Second item: did I ever denied that the problem lies in the data access??? Of course it lies in the data access. But this is not the discrepancy between us. The discrepancy lies in the answer on the question where to look for the cause of the effect. You told me that this is a machine problem. But why do other applications do not have the same data access problem? What is so special with Amibroker's way of accessing data?? If you can help me answering this question we certainly will someday drink a beer together ...
My response to the factual points of your message:
What impact could the data plugin have on the runtime if the provided data are stored locally? But beyond that question: a Amiquote supplied database shows the same problem.
/3. Look at the two benchmark results. As I said my conclusion is: the iMac/Parallels combination is no data access bottle neck - at least not a physical one.
I increased the /Tools/Preferences/Data/Cache size to 4096 MB before I contacted the forum. This had no significant influence on the processing time.
I will now contact Parallels support. May be they will have an idea what can be done to locate the root of the problem. If they can give me a valuable hint I will inform the forum about it.
... another proof (coming from your Performance Monitor) that you are still connected to a plugin (not using a local database - as adviced):
Switch to a local database, do the Backtest and then compare the timings. Unless you do that, pasting CrystalDisk screenshots, make no sense at all, because the plugin's speed is the limiting factor here ...
as I already wrote: Taipan or Amiquote based data supply makes no difference.
Amiquote:
Taipan:
Beyond that I ask myself why there is a local data storage for Taipan data The TPW database consumes 378 MB disk space - for what??
For my idiotic brain this seems not necessary if the supply during backtests comes from the plugin (namely the COM interface). And I continue asking myself why the data supplied by the Taipan db are not up-to-date during backtests unless I restart Amibroker after the daily update from the Lenz & Partner sources. The synchronicity of the backtest data and the TPW database would IMHO always be guaranteed if the backtest function would be supplied with data by accessing the native TPW database via COM at runtime. But synchronicity-at-any-time does definitely not exist! - Please explain me what is wrong in my limited understanding.
I advice you spending some time learning the software - it will pay off. Contrary to today's standards, AmiBroker is ultra light (EXE and DLLs are less than 5MB! ) and effectively written (C++ and assembly). For this reason it's hyper fast - easily the fastest on the market. Properly written AFL can be as fast as machine code: http://www.amibroker.com/kb/2008/08/12/afl-execution-speed/
That is why 99,9% of the performance issues are the result of misconfiguration or a poorly written code or stem from different things not related to the program itself.
As you can see below, current versions of AB, is running perfectly fine on computers from two decades ago:
) and effectively written (C++ and assembly). For this reason it's hyper fast - easily the fastest on the market. Properly written AFL can be as fast as machine code: