Database management best practice?

AmiBroker
1

HI All, I purchased a historic data bundle from firstratedata.com that contains historic data for stocks, etfs, crypto, futures, indices at several different timeframes: 1m, 5m, 30m 1h, and daily. Futures data also comes in unadjusted and 2 flavors of adjusted data for different time frames. Unzipped this is about 400G of data. I am trying to figure out how to map this data to one or more AB databases. I have gone through the docs and forum questions to find a best way to do this. Some questions that i have: - How to store data for different time frames for the same ticker symbol in the same database? Lets say AAPL 5m and 30m data? - How do i store adjusted and unadjusted data for the same futures contract in the database? I would appreciate it if someone could point me in the right direction.

Thanks,
Eddy

2

In regards to timeframe AB recreates timeframes from the smallest ones. That means you just import 1m data and you will get 5m,1h,1D etc out of it.
If you import 1D you can have 3D,1W etc..

The databease has a base timeframe setting that you need to set to be as the data you put into the db.
Be sure to only import 1m data in the 1m database.
Clearly, the smaller the timeframe, the bigger the db will get

1 Like
3

@DBV . Thanks for replying. I very much appreciate that. I understand everything you say. I will create a database with a 1m base timeframe and let AB derive higher timeframes from that ( i thought i was limited by 1million bars in the database but that is not true for a local database). A few more Q's: 1) can i store 1min and 1day data in the same database? 2) By deriving lets say 1h candles from 1m data will there be a noticeable performance hit versus having a separate database with a base time of 1h?
At a more higher level i am still struggling with the following: Lets say i want create a strategy for stocks but i want to look at the VIX or any other index for sentiment. Does that mean all stocks and indexes (which VIX is part of need to be in the same database?

4

Don't be overwhelmed, if you can! :grin:
I know the feeling of starting with 'the best' practices/approach with swiss army knives like tolls like AB and before with some programming libraries. Don't worry about it for now.

For question one, I don't know if you can(you probably can phisically) but I for one would not do it.

If you can't fit all the data in one db, organize dbs by some criteria that suits you, markets, sectors, etc.

It all comes down to how you will be using amibroker. If you use daily bars and enter on the next day's open you will never need less than 1d bars. If you want to do intraday stuff say 1h and you enter on the open of the next bar you don't need 1m.

Decisions regarding db are all about usecase. But since you're new you don't know what's best for you yet, so take a rock, throw it in the air and see where it lands. The only rhing I would advise agains is mixing timeframes.

Performance wise AB is top shelf. Unless you have errors in the code or do funky stuff don't worry about it until it hits!

I think it's better to think the best thing is to get one strategy going. Once you have an AFL you cand play around with the databases and see what impacts performance or not.

For the foreign import I can't say. I have them in the same db when I need them.

1 Like
5

First remember, AB works fine in multi-instance in each Windows User Login. (Assuming its windows) Just don't load same DB in 2 instances, it will work, but who saves last will write all the stuff to that DB/ its settings.

Now, one AB instance loads one DB at a time. (you can switch DB though) While running that AB, think of that particular DB Tickers as the universe for this AB instance. So if you want to refer Foreign symbols, they should be in this DB.

The only mention of using multi-windows user login come if you need to use COM/OLE when it is run externally (ie. outside of AB). Invoking object from within AB again points to that instance(caveat). Just writing this if you need to use it, as few places mention multiple user logins. (otherwise, single user multi-instance AB is the way to go)

So, if you want to do alot of strategies, you just plan what you want to do and setup:

  1. DB with required Tickers and the base timeframe that you wish to use. AB on the fly builds higher TF (compatible TF) so 10min TF will not build 15min, use 5min in this case.

If you believe you want to do EoD testing without the need for intraday TF, then use a separate DB with EoD data that will just reduce bloat. Ofc, you can mix intra+EoD, check DB settings.
Since, you will import from local files, to local AB DB, this is primary scope of discussion. Data plugin is different animal.

Creating multiple DBs is no problem, have as many as want (just more disk space). Also, once DB is made you can add/remove symbols as you wish.
As @DBV wrote, plan what you want to do. That is the design phase. Also, it is not very clear what you want so I can't post specific but glad to help if you do.

Again, AB can be as complex as one can imagine, and happy to be corrected if I got something wrong.

1 Like
6

I would only add that AmiBroker can be run in multiple instances also from one user.

1 Like
7

But that was exactly what I wrote :wink: that is the best way to go

8

OK, I thought that you mean you need different Windows users. If that's not what you meant, then I misunderstood.

1 Like
9

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.