if Data source setting of File > Database settings is set to local database then number of bars is not limited to default number of bars limit. There is no need to set number of bars in that case.
Number of bars setting of File > Database settings is relevant only if using data plugins.
Default 500k bar limit (AB versions 6.17+ have 1Million bars default limit) of number of bars setting is relevant only for data plugins too.
But also keep in mind that there is no hard coded 500k bar limit at all since it can be changed in registry any time.
Quote by AmiBroker:
The limit is not hard coded... it is here for performance reasons but if you have hyper-fast machine you can adjust the limit by creating the following (DWORD) key in the registry HKEY_CURRENT_USER\Software\TJP\Broker\Settings\MaximumNumberOfBars
If you use local database as data source then there is no need to set registry value to a limit higher than 500k (see 1.)! So once again it's related to DBs using data plugins as data source only!
5.it makes no sense to use multiple years of 1min history in real-time data bases -> (recommended) conclusion: use separate data bases (realtime (trading) <-> non realtime DBs (analysis)).
Now in case you still want to change the (real-time) plugin 500k default limit to a different number of bars here is how to... (in the example the limit is set to 1 Million instead of 500k):
Go to Windows Start > Run and insert regedit and hit enter
in the registry editor browse to HKEY_CURRENT_USER\Software\TJP\Broker\Settings
Right click Settings folder and choose New -> DWORD-value
insert MaximumNumberOfBars as DWORD name and hit enter
double click that newly created value and in settings window choose decimal and insert your custom value (new bar limit), in example below I have set to 1000000 (you can set to any reasonable value).
Click OK and close Regedit
Set your new number of bars value in AmiBroker
(Note: the registry editor below is not Windows's Regedit but 3rd party freeware called Regmagik. You may use Windows Registry editor instead via Start - Run - Regedit)
I am not very keen to use “unlimited” word. Everything has its limits. You are certainly limited by RAM size. The array has to be allocated in RAM, so there must be enough RAM. With 40 bytes per data bar, 1 million bars takes 40MB. It may not seem too much nowadays. With 50 million of bars you reach 2GB limit on 32-bit Windows. Keep in mind that data must be actually read from somewhere. Downloading 2GB of data per symbol takes time.
It is really a pity that in todays world that relies so much on technology, nobody teaches technology in schools. But at least everyone had physics in school so they may remember that speed of light is limited too.
Thanks to fxshrat and TJ to point out the errors of my ways. Yes, I added the key as suggested in the doc and set the value to 5MM. Now I have 1-min bars starting on 9/7/2005 in Amibroker.
Don’t worry. This was not meant as any kind of critique. Just general remark (2 cents worth). “Unlimited” was definitely better sounding than “not limited by limit”. If I were to suggest something I would say “Limited by computer’s memory”
Dear Tomasz,
I've currently read the same pages on database and the number of bars and other relations.
I get RTD data which I have set to base interval to 1 min and stored locally.
What I have realized is that when explore runs, it loads each symbols data file in the RAM.
When I start AB, its like 10MB, and for every symbol added to explore it increases by about 1MB per symbol because currently that seems to be the size of each flat file (file of each symbol in local DB).
Q1. Is there a way to limit the number of bars stored in the RAM? My Explore is very vbasic and computes only on past 10 days data.
Q2. 200MB isn't a big deal but the daily rise seems to be alarming over a period, can this be handled or one needs to dump, trim and re-import the data?
What is this limit about? If I set the Number of bars to 1M, does it mean that it is not possible to download more than 1M quotes with the plugin? Or is it just the limit for memory allocation?
It is just "number of bars" setting for the plugin-driven database.
It is the size of array to be allocated for plugin to fill data. Having that set makes sure that allocations are equal size and that prevents memory fragmentation (which otherwise would be a problem on Windows due to its poor memory subsystem).
Should you be systems programmer for 20+ years you would know what I am talking about.
Every plugin vendor can do whatever he/she wants inside their plugin, but why would it make any sense?
OUR plugins do NOT download more because it would be WASTE of resources (bandwidth, electricity, oxygen on this planet!) to download more than it would be used. What's more our plugins are even more efficient. They only download MISSING data (only the chunk missing from last valid quote).
So general advice is never download more than absolutely necessary.
I'd like to download 2 million quotes (couple of years of 1 minute data). Can the eSignal plugin do that even if I set the number of bars to 1 million? Or will it cut the data to 1 million?
If you want 2 million bars, you have to set number of bars to 2 million. It is that easy. I don't see the problem. Did you even read the thread from beginning?
Thank you to fxshrat and Tomasz for the suggestions in this thread.
I'm looking to do backtests with a large amount of M1 historical data and also do real-time trading, both with DTN data. I like the suggestion above to use a realtime database and a non-realtime database. My question is: does AmiBroker allow the same datasource (in this case DTN IQFeed) to populate two databases? I would like to set up the historical database with N=5,000,000 and the realtime database with N=5,000 where N is number of bars in the database settings dialogue.
My second question is whether using N=5,000,000 to download M1 historical data for 2000 stocks will result in memory exhaustion issues (since I understand that Amibroker allocates memory arrays of 5,000,000 elements for each symbol)? Please note that I'm not planning to use N=5,000,000 for realtime trading. It would be used purely for one-time download of DTN data.
My question is: does AmiBroker allow the same datasource (in this case DTN IQFeed) to populate two databases?
Yes
My second question is whether using N=5,000,000 to download M1 historical data for 2000 stocks will result in memory exhaustion issues
Use this: Data size calculator to learn how much memory you need for your data alone. You can do the math yourself. Each bar is 40 bytes. 5 million bars in single symbol is 200MB. 11 symbols * 200MB = 2.2GB. 11 symbols is minimum size of in-memory cache. Keep in mind that you have to use 64-bit version if you want to utilise more than 4GB of RAM.
