I use the continuous contract for Fdax from IQfeed, and have the database base time interval set to tic, i have the number of bars set to 10m which gives me less then a years worth in the database, id like to save every bit of data rather then limiting it to 1 year.
Currently the load factor sits around 400-800%, I’ve been dealing with it for a few years now but id really like to keep as much data as possible.
For most of the day i really only need 30 days of data for formulas to work, except when i want to check a few things.
so i suppose what i'm asking is it possible to have 5 or 10 +years of tic data in my database, but dynamically change the amount displayed/used over all chart panes and windows at the same time so everything runs smooth? Like a slide bar in the picture below that changes how much of the database is being used?
Yep i have that on a few of my window panes, limited to 1m bars, but i have to manually change each chart everytime i want to quickly look back, maybe a global variable or similar ? if it's possible?.
then there is still the memory allocation with a large database
Actually just made a static var for this and it seems to change them all at the same time. but hasn't made a great deal of difference. as the "total data access time" is the bulk of the delay
As I wrote in other thread, realistically from "responsiveness" point of view you should not be using more than 16 million bars per symbol that is 640 megabytes per symbol and that is large enough data for current hardware. If on top of that you are using any AFL it means possibly gigabytes worth of data per single symbol to be processed. Read the comment " Doing the math & resonable expectations" of Efficient use of multithreading
If you are looking at 10 years worth of data, then 1-minute or even hourly interval is MORE than perfectly adequate, simply because neither your monitor has one million pixels horizontally nor your eyes, not to mention 100 million data points that 10 years of tick data would easily exceed.
I only have 2 active symbols in the database continuous and active month.
the data is not just for charting,i like the tic data because it has trade data (bid/ask/tradeprice) and some of my back tests/explorations use this, and keeping raw data is always better from my point of view.
i was just hoping there might be a way to work this, but i'm not a computer engineer i don't know how the database and memory interact, i just thought that being able to adjust on the fly would work.
like using a shadow database and only loading N amount into amibroker?
As @fxshrat told you everything is already done, QuickAFL is just that it processes only as much data as needed with currently selected viewing range and/or analysis range.
Just holding data in memory (RAM) does not consume CPU cycles. There is no benefit from not keeping data in RAM. PROCESSING the data (i.e. display / analysis) is consuming CPU. And QuickAFL addresses just that. The same way as charting engine that does not plot same pixel twice.
I understand that people without programming background constantly come up with brilliant programming ideas that in their own opinion are best things since sliced bread but believe me, after 20 years this program under the hood is doing things that you would not imagine in your wildest dreams and everything is already pretty much as fast as it only could possibly be. The room for improvement is pretty much limited to some edge cases that usually are not worth the effort.
Alright, cheers for the explanations.
ill upgrade the PC soon to help a little bit too.
So is there anyway for me to maintain a large database automatically somewhere else as the old data drops off the end?
or i could just do that with a python script i can run monthly i suppose
