I am experiencing some database issues an am hoping that someone here can help point me in the right direction and explain what could be happening.
I have two issues, and I am not sure if they are related or not so I have placed both in this post.
Criteria: To complete some volume delta studies, I'd like a tick database that will hold all of the last 120 days of tick data that IQFeed provides for the @ES#.
a) Does not download all available data from IQFeed during after/off hours. I am only getting a month or so back. My database is set for 100,000,000 bars. I am only getting around 18,000,000 or so. If I force a backfill it will just stop at around that number and not go further back.
b) What appears to be a full download attempt occurs after I open the database - every time. This does not just stop at back filling any missing data. This occurs during off hours or during RTH - it does not seem to make a difference.
I am using the following:
AmiBroker 6.29 x64.
IQFeed Client Version 184.108.40.206
DTN IQFeed data Plug-in 6.0.5
Thank you for any direction or assistance that can be provided.
My database settings are as follows:
First read this:
(especially the last paragraph)
Then do the math 100 000 000 bars times 40 bytes (that is what single bar takes) is 4GB per symbol for data alone PLUS about 0.5GB for EVERY AFL array that you would have in your code
Does your machine have 128GB of RAM? That would be realistic amount of RAM required to do what you are asking for. I guess you don't have that. And even if you had 128GB of RAM just reading (not processing) 4GB file from disk to RAM would take half a minute or so on very fast SSD. So, you need reality check first.
DTN sales dept makes various claims but in todays hardware it does not make sense to use more than 10M bars. There is huge difference between being able to just "download" data (in the sense of storing them on disk - this is what they do) when you don't care if it takes an hour or three and using data for actual processing/computation/calculation when you want data to be processed every second or even more frequently. Hardware has its limits in RAM speed, CPU clock, memory bandwidth, and so on.
Thank you, I have reviewed that document in the past and have done so again.
I have a good system with 32GB ram but but what you are saying makes sense.
This high bar database was for backtesting - not for live trading, I was planning on getting the tick data and then converting it to second or some other periodicy to make it easier/faster to work with.
The actual database that the system would run is much smaller.
I was confused because I was prevented from even downloading the data with out any charts open. Is there no solution currently to be able to download the data and process it prior to running tests? I could get second data with the Volume Delta from Sierra but I was wanting to do everything through AmiBroker.
After completing several tests, I am still left with the problem of the data plugin re downloading appearing to perform a complete back fill upon re-opening the database. it is not just updating missing data, but replacing everything. This only occurs with tick databases. Second and minute databases operate as expected. To replicate, perform the following steps:
Select New --> Database
Type in a new folder name into the path and select "Create" button.
Select "DTN IQFeed data Plug-in" as your data source.
Select "Enable" for "Local data storage"
Input 1,000,000 bars.
Select "Tick" as the base time interval.
Select ok. The database will now be created.
Select "Symbol -- > New"
Type in "@ES#" or "@ES#C"
Click on the new symbol.
View the backfilling window and observe the data being downloaded from IQFeed.
Open the database that was just created if it does not open automatically.
View the backfilling window. The plugin will download all of the bars (1,000,000) data every time the database is opened.
If anyone has any ideas, I'd really appreciate it.
This is perfectly normal operation once you think about it. The plugin uses IQFeed API command that requires giving number of bars to download.
In time-based intervals you can actually calculate how many bars (i.e minutes or seconds) are missing from last update until now. So it can download how many bars are missing and request only that number (actually it downloads few bars more)
In tick mode, you can't calculate that because you don't know how many trades occurred during that time (this varies a lot depending on time and market activity), so you need to download as much as user has defined in the Settings. 1 million ticks is not that much on as active symbol as ES. Also there are tick corrections posted later than need to be downloaded too.
Yes, I used the 1,000,000 value just so that others could verify the behavior with out having to wait very long for the download to finish.
I was not expecting the issues as I can easily and relatively quickly, download and generate second bars in something like Sierra or IRT from the full 120 days and retain the bid/ask aggregate.
The AmiBroker IQFeed Plugin appears to limit the number of ticks to just under 20,000,000. If that could be increased to allow for the full capture that would be appreciated. I understand that a user would not be able to run a live system off of this, but for processing data, it is perfectly fine I think?
I'll probably now just download the data directly and convert in SQL Server to solve the problem, which will be ok I guess as I can handle rollover there as well.
Thank you for the clarification - it helps me to refocus my efforts.
I do appreciate AmiBroker - it is pretty neat and capable software - thank you for your hard work.