Mixed Intraday/EOD Databases, Size, Speed

Hi There,

Please forgive my non-technical understanding of Amibroker and databases, but perhaps I could get some practical answers to a few questions? I feel like some of the answers may be in this post "How much RAM does the database consume", but I'm not savvy enough to understand it sorry.

Specifically using IQ Feed plugin and AB 6.20.1

Note also that I am happy to move to 'local database' if that helps.

  1. I have created an IQ database with mixed EOD/Intraday data. I used a very high number of bars in order to download a FULL history of 1 minute data for around 5,000 stocks. I can run backtests on intra-day data fine, but not on daily bars. I believe it is a memory issue related to the massive number of bars configured in the db.
    Can I not decrease the number of bars for running on Daily Data without losing historic minute data? It seems that if I decrease the number of bars, the historic minute data disappears. I think that when I go back and increase the number of bars, AB has to backfill / download this data once again. I would like to avoid that.
    Basically, is there a way to run a BT on Daily Bars that is not using memory based on the high number of bars in the db? I don't understand why these need to be linked sorry? Can't I have 10 years of historic minute data saved in my database, and then change settings for efficient backtests, without losing historic data?

  2. With mixed eod/intraday databases - is the EOD data literally separate? e.g. with IQ Feed, the EOD data is back-adjusted but the intraday is not, therefore they are actually different data sets right? Does AB keep them separate? Or, if I build a separate IQ database with EOD data only will it be different to the mixed db? If they are separate, can I not have a full history of minute and eod data, without the intraday settings constraining the eod backtests?

  3. When running a backtest with IQ feed plugin, backfilling for all symbols in my backtests (thousands) happens automatically. Can I stop this from happening? Or do I have to set to local database to stop that?

Thanks for any help, again, apologies for my lack of computer science understanding here.

Regards,

i am a relatively new AB user, but recently learnt quite a lot because i was also trying to figure out what happens.
Others please correct me if i don't get some stuff right.

Anyway, you have to check the quote editor. AB is very versatile, so it is capable of building Daily bars from intraday data but it can use EOD data also if available.
now this depends on your vendor, i have not used IQfeed, but do you see bars with only the Date portion? Those rows in quote editor are called EOD data and AB uses them for daily and higher timeframes.

My suggestion is why not have multiple Databases and treat them accordingly for intraday and daily.
I also read some posts by Milosz, and he also says you can switch that in DB settings from Plugin to Local but for purposes of speed, it is best to backtest using Local DB.

Therefore having mixed DB is ok, but if you have so many bars and are facing issue maybe also because of hardware limitation then you need to sort it from scratch, how will plan to separate the data.

1 Like

@ezekiel Exactly for this reason you should rather post your questions in that (mentioned) thread instead of creating a new one. As you wrote - most of your problems were already addressed there...

Have you setup you database correctly? Have you checked Allow mixed EOD/Intraday data?

obraz

If yes, then your quotes (Menu --> Symbol --> Quote editor) should look similar to this ( one EOD bar after each session):

obraz

A quote from: http://www.amibroker.com/guide/w_dbsettings.html

Allow mixed EOD/Intraday data - it allows to work with database that has a mixture of intraday and EOD data in one data file. If this is turned on then in intraday modes EOD bars are removed on-the-fly and in daily mode EOD bars are displayed instead of time compressed intraday or if there is no EOD bar for corresponding day then intraday bars are compressed as usual. Mixed mode allows intraday plus very long daily histories in one database.

If everything is set-up correctly your backtest on daily bars should be performed much quicker that using intra-day bars (because there are much less daily bars) and for the same reason you shouldn't have any memory related issues. Have you really selected Daily Periodicity in AA settings?

obraz

As it was already written, to achieve the best performance and to avoid possible problems (like updating hundreds or thousands symbols during backtest), you should always switch to a local database before doing a backtest. So update/backfill all symbols before, save the database, switch to a local database and then run your backtests, optimizations, scans, explorations etc ...

You might consider updating AB to a newer version, because (as I wrote at the beginning of my first post in the mentioned thread) one of the changes in ver. 6.28 is this one:

IQFeed: in mixed mode EOD data got preference if there is not enough room to fit all intraday and all eod in defined “number of bars”

https://www.amibroker.com/iqfeed.html
https://www.amibroker.com/guide/h_iqfeed.html
https://www.amibroker.com/guide/h_rtsource.html

2 Likes

Thanks very much @nsm51 and @Milosz !!

Yes, I think two separate databases are a good idea.

@Milosz, yes is the answer to all of your questions - everything was setup correctly.

I am happy to upgrade AB, although I'm not sure I want "preference for EOD" data (?). I feel like what I want is for EOD Data tests to not be slowed down by a large number of bars. Which you say should be the case... So, I looked at the quote editor and perhaps this is the problem:

For some reason my Daily Data only kicks in at the point where the Intra-Day data stops (i.e. before 2007, and after that there is only Intra-Day data)!! (could this be solved with upgrade? It doesn't feel like it, because even when I increased bar count and forced backfill I can't get more EOD data)...

Annotation%202019-06-02%20130355

Annotation%202019-06-02%20130408

Another issue I'm having that I feel is related - (also seems to show intra-day / eod data not playing well together) is that if I import a symbol that I need in BT which is not in IQ db (such as $SPX) in EOD time frame, when I attempt to run a BT (in intra-day mode) it says this symbol "has no quotes", yet it does have EOD quotes. If I try to run in EOD mode, it does not produce this same error.

I was hoping importing EOD data would be sufficient for intra-day tests - I presumed it would take the EOD price and apply it to each minute of that day. Is that not correct?

Thanks!

"No quotes" means no quotes in selected interval. You can create EOD quotes out of Intraday but you can't do the reverse (create intraday quotes from EOD quotes) http://www.amibroker.com/kb/2006/03/19/how-does-the-daily-time-compression-work/

In similar fashion you can create 5-minute data from 1-minute data but you can't do the reverse.

Thanks @Tomasz,

So no ideas why IQ isn't downloading EOD data for the same period where it has Intra-day data?

I don't think this is just your data vendor, I get the same thing.
Probably they expect the charting software to extrapolate all the higher time frames.

The way the charting software [AB is much more... ] also use the data is different because there are so many around, there is no consensus between the two.

Either you can download your own EOD separately using Amiquote or other methods. I have seen some topics on exporting data from AB, then you can export Daily data from the intraday and use it in the daily DB.
I'm still working on all of this to streamline the process for all scenarios that i bump into.

Exporting works just fine and then re-importing it, i do it to keep the RT DB size down as i dont need such low res historical data for that DB.
You have to make sure you delete the quotes of the low res in that range first if you dont want them. (another pointer, thats all)
There is a script for that and is described here too: http://www.amibroker.com/kb/2015/01/13/how-to-delete-quotes/
You can use any suitable language provided you get the OLE part right.
Also, many a times intraday data that accumulates in realtime is often corrected so you will see backfill being different as it gets adjusted/corrected when done after the session or a day later.
The same occurs with daily adjusted data as well.

Cheers @nsm51, interesting that you get the same thing. I'll let you know what IQ Say.

It's not right, because IQ Feed EOD data is back-adjusted for splits, whereas their Intra-Day data isn't, so they need to be two distinct data sets. Also, I have proven this: if I create an IQ Feed EOD Data database only (not 'mixed eod/intraday) then I get full EOD history as expected.

2 Likes