Amibroker waits When Norgate Tries To Update Data

Is there away around this?

I'm using Norgate's data feed and I find that whenever I'm using Amibroker's backtesting function, and Norgate attempts to update, Amibroker freezes and needs a restart, after I wait for norgate to download the data.

A potential solution I've found is to delay Norgate's data update schedule until a time when I'm most unlikely to be doing testing, but this sounds like a band-aid fix that avoids the problem rather than fixing it. What other solutions do you have?

Yeh, I have the same issue here as well. However luckily, my AB just freezes and I don’t need to restart it after Norgate has finished updating. It’s darn annoying, probably made worse if you have the optimiser going. Work around would be to turn off auto - updates in NDU, but as you mention it’s not fixing the issue.

As @Tomasz has said, it is not AB, it is the 3rd party data source.

You need to tell that to Norgate. The plugin should do their actions either in "nice" way (not interferring with normal operation of program) or off-line when program is NOT in use.

And disabling auto-updates in NDU is recommended to avoid sudden updates.

There is nothing AmiBroker can do when plugin takes over the control over CPU. AmiBroker must wait for plugin to return control.

It does not freeze. It WAITS for plugin. Properly written plugin should never spend more than 0.1 second in any call. Longer workloads should be divided in not-so-time-consuming chunks.

If heavy lifting is required, it should start external separate process to do the work without interfering AmiBroker

Definitely Norgate has done something in the recent days. I am having issues to run backtests on the recent data. Since colleagues from Norgate visit this thread, appreciate any help in this regard

Norgate OK

The above backtest when end date is 09/11 works, but when changed to 10/11, it does not work anymore as shown in below screenshot

Norgate NOT OK

(In both cases green OK was visible)
Filter used is Nasdaq current and past.

Thanks

It is just % sign used in symbol names that causes this error. % is a marker for special formatting sequence for printf/StrFormat, should not be used in symbol names because then

printf( Name() );

would cause errors

You can fix the problem by editing offending formula and changing the call to

printf( "%s", Name() );
1 Like

thanks for hint Tomasz. That was fast!
I am not using printf in "my" AFL, but am including the norgate functions within my AFL. I will check if include files has any printfs

It is likely Profit Table.afl in Report Charts folder.

1 Like

I will reach out to norgate about this. In the meantime, here's a more descriptive issue of my problem.

AmiBroker seems to disrupt the Norgate download. You're right in the fact that AmiBroker, technically, is waiting for the download to finish, but Norgate will not download the update while AmiBroker is waiting. The two programs do not interact nicely together.

I was hoping there was a simple fix to this on the AmiBroker side but that doesn't seem to be the case. I'm going to reach our to Norgate on this one.

In the past I've had to close AmiBroker to let Norgate do its thing, then relaunch it. Next time this happens, I'm going to close NDU and see if that works. That might be less disruptive to my workflow.

I'll update if I hear anything useful from Norgate.

1 Like

How many times I need to repeat that when you configure AmIBroker with DATA PLUGIN, the plugin TAKES CONTROL - ie. AmiBroker calls say GetQuotes() function inside the plugin DLL, at this point DLL "owns" your CPU and plugin should return in fraction of second (may below 0.1s).

If plugin does not return quickly (even if it should), then the PLUGIN is the problem.

You don't know how program works. Calls to DLLs are synchronous. They are BLOCKING calls. At given time when DLL is called,the calling program CAN NOT do anything else that just being at the mercy of DLL to return control.

Imagine faulty display driver. If display driver hangs you will NOT see anything on screen and Windows can not do anything "on Windows side".

Disable Norgate Auto-updates. That is a fix. The only other fix is not using plugins. And complain to Norgate, not me.
I am not responsible for 3rd-party-created problems.

2 Likes

In general most systems can achieve a few to several hundreds of GetQuotes(ex) calls/second, which is well under 0.1s, even though this is limited to a single thread.

We have careful locking database procedures to ensure that incomplete/stale information is not provided to AmiBroker during a background update.

However, Norgate Data Updater cannot determine what any/all instances of AmiBroker is doing at any given point in time. For example, if a background update is underway in Norgate Data Updater, and you're 30% through an Optimize run, and a significant update is running, then you'll have 30% of your Optimize run on the prior data and 70% on the new data.

A way of avoiding such issues is to run any updates via AmiBroker's batch capabilities, which updates and closes thereafter:
https://norgatedata.com/amibroker-usage.php#batchmaint

Alternatvely, just shut down NDU prior to any Optimize/Backtest runs.

With regards to "AmiBroker seems to disrupt the Norgate download. You're right in the fact that AmiBroker, technically, is waiting for the download to finish, but Norgate will not download the update while AmiBroker is waiting. The two programs do not interact nicely together." - this is not an issue we've seen before or had reported to us previously, so will need some careful consideration as to diagnosis to determine the cause, replicate and fix as required, and we encourage users to report this to us directly with carefully constructed examples of how to replicate the issue.

Quite clearly AmiBroker should not interfere with Norgate Data Updater's operations and vice versa.

Follow-up to this issue:

We reached out to the user that reported the issue and he responded with: "I am not sure exactly what exactly was the cause. After rebooting the pc, it seemed fine.If it happens again, I will let you know"

So, we're not sure why it happened, neither us or the user can reproduce the issue but it is confirmed that a reboot fixed things.

4 Likes

From my experience, Windows is getting "old and tired" by itself after weeks of uninterrupted use and reboot makes it alive like a newborn again. "Getting old" manifests by slow response to user input. Sometimes when you use Process Explorer you can see more time spent handling interrupts (System/Interrupts) than usual. Sometimes this may be due to driver https://www.makeuseof.com/tag/fix-high-cpu-usage-caused-system-interrupts/ sometimes it happens with no clear reason (some possibilities: https://www.makeuseof.com/tag/fix-high-cpu-usage-windows/).

So if your system runs somewhat slow it is good idea to reboot periodically. In many cases it is quick solution to mysterious slowdowns.

5 Likes

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.