How much RAM does the database consume? Mixed EOD/Intraday database settings

Hello @Tomasz. One of the changes in ver. 6.28 is this one:

IQFeed: in mixed mode EOD data got preference if there is not enough room to fit all intraday and all eod in defined “number of bars”

Does this change also apply to other plugins that support Mixed EOD / Intraday Data mode (especially Statica)? I'm aware of the fact, that Statica plugin is not officially recommended, but it is enlisted among other dedicated RT plugins for AB here: http://www.amibroker.com/guide/h_quotes.html and described as supporting mixed mode (which is confirmed by the plugin's provider). It is also the most popular RT plugin for WSE (Warsaw Stock Exchange).

I can't check how AB database managed by IQFeed plugin behaves now (after the change), because I don't use IQFeed, but in case of Statica (in AB 6.30) if the database is full each new intraday bar added to the database (in mix mode), still replaces one oldest EOD bar - so it seems to be working as it used to. Until the database is not full or after increasing the number of bars in DB settings, there is also additional long EOD history available (significantly surpassing intraday data) but gradually all historical EOD bars located at the end of the database are replaced by intraday quotes. So it seems, that EOD bars still have no preference (which would be very desirable).

3


I was also wondering what is the difference between Database settings --> Number of bars:

1

... and Preferences --> Data --> Local database save options --> Limit number of saved quotations:

2

Thank you.

1 Like

Do not mind my attempt to answering this part

I was also wondering what is the difference between Database settings

TJ's answer will be absolute, but since i've upped my learning curve thought i'd try :slight_smile:
Always happy to be corrected, I learn more.

In 1st case, "Number of Bars" field has a bigger role as in it would instruct the Data Plugin to fetch so many bars also.
A brief understanding of Data plugin shows that the last update tick DateTime ('Timestamp) is known and therefore only fresh data is imported and correspondingly downloaded by Plugin.
This reduces unnecessary overhead of fetching existing data.

Basically this field is capable of instructing the Source or Data plugin.

In 2nd Case, this is specific to the Number of quotes saved in Local DB, as a plugin might keep fetching fresh data everyday and it will eventually bloat the DB.

If the Above is correct, then ideally, one would like the Local DB bars to be slightly more than the DB number of bars.
I forgot whether default Limit number of quotes is checked, but I think its checked off (unchecked) for that reason and therefore the DB number of bars will play its role.

What I don't know is, say we have a very small number of bars to SAVE set in preferences, say 10k and number of bars in DB is 100k,
in this scenario, AB may fetch all 100k bars and we may work on it whole day from the cache, but if you exit the program only 10k will be saved.
Most likely it could be this way.

2 Likes

No it doesn't. This is totally controlled by the plugin. And any change requires changes to each and every plugin separately. Statica plugin is 3rd party. I did not write it, can't comment how it is internally done.

1 Like

Tomasz, thank you for the information.

Maybe it will be useful for other users. I found a way how to take advantage of the extended EOD history when using mixed EOD/Intraday database (like Statica) with limited number of bars. When I see, that the EOD history is becoming too short (because there is not enough room to fit all intraday and all eod data in defined “number of bars”), I decrease the Number of bars in DB settings by let's say 20-30%, run an exploration or scan on selected symbols to let AB adjust (limit) the number of bars and save the database. Next step is to revert to the previous number of bars and (if needed) run scan or exploration again. After this routine (performed once in a while) I have 20-30% shorter intraday history, but full EOD history and some spare room for the next intraday data.


Tomasz, could you comment on the second part of my above query?

What's the difference between Database settings:

Number of bars - defines how many bars should be loaded from external data source and kept in AmiBroker. Examples: 10-years EOD: 2600, 60-days intraday 1-minute: 30000 (approx). This setting has no effect if data source is set to (local).

and Preferences window local DB settings:

  • Limit number of saved quotations - if this option is ON AmiBroker will save database with limited number of quotations. This prevents the database from growing too much
  • Max. number of saved quotations - this is the limit itself. Preferable 300 or higher for EOD databases, 3000 or higher for intraday

Do these settings overlap and gets overwritten or maybe they have different purposes? Would enabling any limit in Preferences window ->data base settings -> Limit number of saved quotations, have any advantages over the limit (Number of bars) set in database settings in this or other cases?

1 Like

@Milosz You have stroked a very interesting topic, see this in exact words from adk.html

There is no Line Number, but it is the entire paragraph before Heading 3.5

Third parameter nLastValid requires some more description. When AmiBroker calls GetQuotesEx() function it may already have some data for given symbol (stored for example in its own files when local data storage is ON). However, before GetQuotesEx() is called AmiBroker always allocates quotation array of size defined in File->Database Settings: default number of bars. This size is passed to the plugin as nSize argument. If AmiBroker has already some data for given symbol it fills initial elements of quotation array passed to GetQuotesEx() function. The index of last filled element of quotation array is passed as nLastValid argument. This allows to update just a few new bars without the need to fill entire array inside the plugin. This technique is used for example in QuotesPlus plugin that does not update entire array if it finds that last valid quote is the same as last quote available from Quotes Plus database.

Its interesting how much memory can be blocked if user is not careful with number of bars DB setting :smiley:

2 Likes

@travick thanks for your replies :slight_smile:

For this reason I limit my current database to about 50K bars. It is because I periodically scan/explore about 1000 symbols during the session and my Plugin is available only in 32bit version. If I was using let's say 100K bars database I would easily run out of memory available for 32bit AmiBroker :wink:

A quote from https://www.amibroker.com/guide/x_performance.html

This is the most often mistake people make. For example I have seen 50000 entered into "Number of bars" entered for end-of-day database. This does not make sense because 50000 daily bars is 192 years!!! There is no data source that offers such history and entering that amount is waste of memory. You need to keep in mind that what ever you enter here forces AmiBroker to actually allocate memory for that many bars. Each data bar is 40 bytes. So if you enter 100000 bars, you will force AmiBroker to actually allocate 4MB of memory per symbol. It may not sound too much but AmiBroker keeps a cache of recently used data and if you have defined the cache that is 500 symbols large you will be able to hit 2GB of memory that way. So always look at what File->Database Settings dialog displays next to "Number of bars" field. It will display the number of days equivalent to entered number of bars and choosen base time interval. Make sure that you only enter resonable values and be aware that whatever you enter here has performance consequences. The consequences do not end with memory consumption alone, but also in speed. Processing more bars means more CPU time, especially considering the fact that while small amounts of data can be kept in on-chip CPU cache, large amounts can not. Now considering that on-chip cache is usually 10 (ten) times faster than regular memory you immediately realize that specifying too much bars here result in performance drop. Again: one bar is 40 bytes. For best performance, make sure that you don't exceed CPU on-chip cache size, so if your CPU has 4MB cache, for best performance it is strongly advised not to use more than 100000 bars in File->Database Settings.

Other interesting plugin related matters were also discussed in this thread:

2 Likes

100K bars is not a problem. It is just 4000KB per symbol in the cache (4MB and 0.004GB). You won't run out of memory with that because 32-bit addressable space is 2GB and this means if your data cache size is 100 symbols you are consuming 400MB which is 20% of addressable space. With smaller cache you would consume proportionally lower amount.
Symbols not in cache don't consume memory at all. Cache is maintained so most recently used symbols are in the cache and least recently used symbols beyond cache size limit are removed from cache. So regardless of how large database is the memory consumption is bounded by cache size which is user definable. You can have cache as small as 11 symbols.
This of course assumes that the PLUGIN does not eat too much memory. Our plugin do NOT keep all history inside. Once data are retrieved by AmiBroker, our plugins trim all internal tables and only keep few (something like 5 most recent) bars.
Can't tell what 3rd party plugins are doing. The plugin unfortunately is able to do anything, including all good AND bad things.

1 Like

@Tomasz frankly I don't understand that. Maybe I'm wrong, but it seems contradictory to what you wrote here: https://www.amibroker.com/guide/x_performance.html

You need to keep in mind that what ever you enter here forces AmiBroker to actually allocate memory for that many bars. Each data bar is 40 bytes. So if you enter 100 000 bars, you will force AmiBroker to actually allocate 4MB of memory per symbol. It may not sound too much but AmiBroker keeps a cache of recently used data and if you have defined the cache that is 500 symbols large you will be able to hit 2GB of memory that way.

According to your article 100K bars consume 4MB not (0.4 MB of memory).

Maybe it is because Statica plugin was not written properly, but I know, that when I was using larger number of bars (but still definitely under 100K) I was running out of memory ...


@travick I was playing with different settings of Preferences --> Data --> Local database save options --> Limit number of saved quotations :

2

and observed, that when this limit is below Number of Bars in DB settings, it truncates the number of quotes, when above it doesn't make AB override DB settings. I suppose it just imposes a limit on all databases in AmiBroker in one go. But because I'm not sure and I see no point in using it, I won't.

1 Like

Sorry, in a hurry I made mistake. Yes Knowledge Base numbers are correct. 40 bytes per quote, so 100K bars is 40*100K = 4MB. So I meant 100 symbols in cache @ 4MB per symbol = 400MB. I corrected the post.

The "limit number of saved quotations" has NO effect on plugins or database settings. It is just for "save" operation (i.e. writing to disk file), nothing else. It is intended to trim local files to specified amount.

1 Like

Tomasz, thank you for the reply.

In the meantime I performed some tests and confirmed my prior observations. As you suggested, I tried to decrease In-memory cache size to 11 symbols (and it's size to 64MB) and it didn't prevent AM memory allocation from growing when subscribing to new symbols. After AB restart and subscribing to less than 200 symbols (50K database) AmiBroker was consuming about 500MB of RAM (comparing to 70MB after the restart). It probably indicates, that this Plugin (although very responsive - Plug-in time per symbol --> 0.5 - 1 ms) was not written according to your above guidlines and doesn't conserve the resources.

The good side is, that as I can see, periodical decreasing and restoring prior number of bars (as I described above) allows to make use of a very long EOD history in the mixed mode Intraday database even when the number of bars is very limited and EOD quotes are not prioritized. So there's always a remedy :wink:


BTW is it possible not to display this info when switching from local database to Plugin and vice versa?

1

Regards

2 Likes

What I wrote is about AmiBroker, not about 3rd party plugins.

You are testing Statica plugin - this 3rd party plugin, NOT written by me. Plugins are free to allocate memory on their own all the way they like - the application does NOT control allocations made by DLL. DLLs just ask for memory directly from OS and but the OS accounts memory space per process and since DLL are loaded into process and use process address space the memory they allocated is displayed as consumed by the process. Depending on what allocation function the plugin is actually calling (malloc()/new/HeapAlloc/VirtualAlloc) the memory once allocated may or may not be immediately returned to OS when they release memory block. Since applications can't really control what plugins (DLLs) are doing there is a tendency recently (at least in web browsers) to run plugins as separate processes to isolate them from applications. This approach however creates another bunch of (new) problems and generally increases (not decreases) memory consumption - see how much memory Chrome browser uses for its multiple processes.

If you want to see memory that AmiBroker alone allocated use Tools->Performance Monitor:

image

1 Like

Thank you once again for the detailed explanation.

Yes I was using also the Performance Monitor. I understand that in this case the plugin (not AmiBroker) is to blame for the high memory consumption. I only provided above statistics to prove that if I was using 100K database I wouldn't be able to subscribe to 1000 symbols :wink:

Regards

I would not use the word "blame". They probably did their best. Like many specialized fields, virtual memory systems and allocators are not as simple as people sometimes tend to think. Often you don't control all factors. There are many Ph.D theses written on that subject of memory management: https://www.google.com/search?q=ph.d+thesis+memory+allocation

And you should be able to subscribe to 1000 symbols, just NOT to keep them in RAM cache at once in 32 bit version. Subscription is not equal to being in the data cache. Subscription merely means that plugin asked remote source to send trade/bid/ask in realtime. Assuming that plugin does not keep entire history (it should not) but only few recent bars it would be OK.

In general I'm very happy with their plugin. I have no reasons to complain - I've been using it for many years and additionally I was told they are (probably) going to release a 64 bit version of AB plugin as soon as they migrate to 64 bit with their main platform.

BTW since I started using AmiBroker I haven't faced a single obstacle which couldn't have been overcomed somehow even when I was told that something can't be done :slight_smile:

1 Like