Optimization CPU Utilization with Norgate Data

I'm running an optimization on a large watchlist with 3974 symbols using the Norgate Data Plugin. The 64-bit Broker.exe CPU utilization is less than I expect, often using less CPU than the system idle process:

image

The info panel shows the following. If I'm reading this correctly, it looks like 29% of the time is spent waiting for data access.
image

Performance Monitor and data preferences:
image

image

Any ideas how I can tune the settings to spend less time accessing data and more time utilizing the CPU during the optimization?

You can:

  1. Make sure that "Local data storage" is enabled in File->Database Settings
    then
  2. temporarily switch Data source to "(local database)" so it does not talk to plugin at all
    and then run Optimization.

Thank you for the suggestions @Tomasz. Local Data Storage is enabled. The local database is also a good idea, but my formula is using some proprietary functions exposed by the Norgate plugin.

One more thing, could you clarify, does the symbol cache work with 3rd party data plugins?

UPDATE: It turns out, some of the Norgate plugin functions can cause backtests and optimizations to run slowly. I was able to almost double the optimization speed and increase CPU utilization to 90% during phase-1 backtest using memoization as shown in the example function below.

function ExpensiveFunction()
{
  staticVarKey = Name() + ":ExpensiveFunction";
  result = StaticVarGet(staticVarKey);
  if (typeof(result) != "array")
  {
    originalClose = NorgateOriginalCloseTimeSeries();
    originalVolume = NorgateOriginalVolumeTimeSeries();
    result = originalClose * originalVolume;
    StaticVarSet(staticVarKey, result);
  }
  return result;
}

Just an observation about this particular example: originalClose * orginalVolume will always be equal to Close * Volume. So you can really save some time by not calling either Norgate function. :wink:

3 Likes

Quotation cache works with plugins but... plugin can forcefully invalidate cache simply by sending WM_STREAMING_UPDATE message. I don't know what Norgate is doing or not doing.

Anyway calling ANYTHING external is always slower than staying within AmiBroker native code.

1 Like

@mradtke, you are mistaken. Close and Volume return the adjusted price and volume after capital events and dividends. @Steve seems to be after a result of close * volume as would have been calculated on that day before any future adjustments.

Yes, Volume and Price would get adjusted in a way that cancels out for stock splits or consolidations, but when price alone gets adjusted for dividends the C*V value changes from what the result would have been on the day.

https://norgatedata.com/amibroker-usage.php states:

Referencing the Original Unadjusted Close and Volume

Depending on your plugin configuration settings for Price & Volume Adjustment, the price and volume data may be adjusted to account for the effect of capital events and dividends.

If your trading system needs to reference the original unadjusted closing price of a security, use the function shown below.

oc = NorgateOriginalCloseTimeSeries();

Likewise, if you need to reference the unadjusted volume, use:

ov = NorgateOriginalVolumeTimeSeries();

Frankly speaking these shound not be delivered by functions but by aux1/2 and/or artificial tickers that would be much faster and would benefit from caching

1 Like

@jani, I understand what Steve is looking for and I do not believe that I'm mistaken. I ran the exploration below against Norgate's S&P 500 Current and Past watchlist from Jan 1 2000 to present and it found no instances where the product of adjusted close and volume was different (within AB's numerical precision) than the product of orginal (as traded) close and volume.

#include_once "Formulas\Norgate Data\Norgate Data Functions.afl"

pUnadj = NorgateOriginalCloseTimeSeries();
vUnadj = NorgateOriginalVolumeTimeSeries();
toUnadj = pUnadj * vUnadj;

toAdj = C * V;

Filter = !AlmostEqual(toUnadj,toAdj);
AddColumn(pUnadj, "pUnadj");
AddColumn(vUnadj, "vUnadj");
AddColumn(toUnadj, "toUnadj");
AddColumn(toAdj, "toAdj");
AddColumn(C, "C");
AddColumn(V, "V"); 

If you change the Filter line to:

Filter = toUnadj != toAdj;

you will come up with a lot of hits, but it will quickly become obvious that the "inequality" is beyond AB's 7 significant digits as described here: http://www.amibroker.com/kb/2010/07/20/about-floating-point-arithmetic/

1 Like

Will this code work properly during a walk forward test? I'm concerned the length of the arrays will change as the test moves forward and the static variables captured at the start of the test may not have all the data at the end of the array.

function ExpensiveFunction()
{
  staticVarKey = Name() + ":ExpensiveFunction";
  result = StaticVarGet(staticVarKey);
  if (typeof(result) != "array")
  {
    originalClose = NorgateOriginalCloseTimeSeries();
    originalVolume = NorgateOriginalVolumeTimeSeries();
    result = originalClose * originalVolume;
    StaticVarSet(staticVarKey, result);
  }
  return result;
}

Matt's understanding is correct here.

Unadjusted close * Unadjusted volume = Adjusted Close * Adjusted Volume.

The reason for this is the adjustment mechanism applied to the close is an exact inverse of the one applied to volume.

Consider a 2:1 stock split. Prices are multiplied by 0.5. Volume is multiplied by 2.

This methodology is used for all types of adjustments, not just for capital events such as splits.

4 Likes

Wow, I learn something new every day. The support documentation on how the price/volume adjustments are carried out didn't make it clear that dividend distributions also adjusted the volume data.

In which case, so long as the indicators you use are reversible and do not change based on price or volume levels, there is no need to access the original close or volume.

Most indicators would probably then be fine, but back-testing of anything that references absolute values would not be accurate - I have several systems which use limit orders to enter trades, and my backtesting calculates the level for the order and rounds to price steps that are actually available on the ASX ( https://www.asx.com.au/services/trading-services/price.htm ) This transformation would not be reversible after price adjustments.

Some users have absolute rules about prices (i.e. price must be above $2 etc.) so to handle that in a backtest you need to reference the original unadjusted close.

If you want to get pedantic about trading prices/price steps in a backtest you'd also need to reference this too and perform your own proportional adjustments against the adjusted close. I'm not sure the effort would be worth it though - I think the trading system backtest result would be highly correlated.

Also, I vaguely recall that the ASX price steps changed around the turn of the century too. The US ones certainly changed, when they moved from 1/8ths and 1/16ths to decimal pricing.

Cheers,
Richard.

1 Like

Steve,

Can you share, if your wrote, a memoization function & usage for NorgateIndexConstituentTimeSeries() ?

@DingoCC The pattern is the same. You can try this code. Disclaimer: this is completely untested.

function IndexConstituentTimeSeries(indexName)
{
  staticVarKey = "IndexConstituentTimeSeries_" + Name();
  result = StaticVarGet(staticVarKey);
  if (typeof(result) != "array")
  {
    result = NorgateIndexConstituentTimeSeries(indexName);
    StaticVarSet(staticVarKey, result);
  }
  return result;
}
2 Likes

Thanks @Steve . My 24 thread workstation now back tests in 1 second with the cpu at 30% , compared to 60 seconds (1% cpu), before this code change.

Based on numbers posted here, it seems that Norgate* functions themselves need some performance optimizations. The solution posted by @Steve is good but: a) it takes lots of RAM to cache all values for all symbols, b) when changing padding settings and interval selected the static variables for those values may need to be deleted.

Future versions will measure performance of each plugin-exposed function and report time spent inside 3rd party DLLs in Analysis.