How do you construct your database?

AmiBroker
1

AmiBroker is such a versatile tool that we can go from backtesting historical data from 30 years ago to live trading the latest tick.

Data sources too are numerous and methods for importing the data into AmiBroker vary, from RTD plug ins like the awesome universal one that @nsm51 is currently developing, to ASCII imports via wizard, to the OLE API imports.

Each import method has its pros and cons and the need for various work arounds.

For example, importing via ASCII is slow (?due to IO file operations).

JSON imports are amazing for several (even 10’s of) thousands of rows, but when importing 1000’s of symbols on 5m timeframe for 8 years, then if the dictionary is too large it’s not imported, but if it is paginated (even slowly) it is written over. I suspect this is not a limitation of the plug in nor Amibroker, but of my understanding of database design principals.

I have had AmiBroker for 3 years and I haven’t finally settled on a design for my database that suits, and I’m looking for inspiration from other users on how they suggest configuring a database.

What would I like to be able to do?
-Backtest historic 5m data for crypto and forex.
-View historical data from low to high time frames, up to monthly and use Amibroker as my charting package.
-Run a scan every 5 minutes for entry and exit signals which I then parse.

I’m certain the obstacles I am facing are not insurmountable, but I suspect also I am looking at the problem the wrong way.

How would/do you construct and Amibroker database (or would you use 2 databases, one historic, ASCII, and one for real time data?)

2

It would take an afternoon to write a post but the quote is your answer.
Eventually, you would likely be limited by RAM with so many symbols and long data, assuming you have 5m bars for 10-30 years.

When you backtest, AB can sequentially load and unload from cache as it cycles through the entire BT for symbols in context.
In RT, when running an AA-explorations, you want all symbols in memory, otherwise for insufficient cache/RAM, AB will be burdened by reading files from local DB all the time.

Tomasz told me why he eventually chose to load all quotes to memory instead of partial loading.

As far as import goes, you can use ASCII import for rtd plugin, just ensure sufficient bars in DB setting to hold enough data.
Also, don't import new symbols directly with ASCII, if you did, change the local db only property to No from Yes.

Also, import data from oldest to newest.
If you are paginating using json, send the oldest page first and run explore to move the data to AB DB. Only after this, you repeat with 2nd oldest page > explore.

========
so here is what i have done.
I have 2 instances of AB, both with rtd plugin.
First instance has limited small numbers of bars, runs some explorations for RT every 5s.

Second instance, is like a charting tool. Has rtd plugin, and both fed from same websocket port.
Setup initially by importing all the data with ASCII ensuring same base time interval. This instance doesn't create much load but serves the purposes of having long historical data.
If you understand properly, you can use higher TF older data as well, AB justs works with it. So when i switch to Daily or higher, there is no issue.
I could BT with this DB.

Since both are plugin driven DB, I ensure sufficient bars otherwise even imported ASCII data will be over-written.

I might have missed few things, will add as i recall.
This approach avoids me going to an external DB. AB stores quotes which are in memory the same way to files. So performance wise, nothing else would be nearly as fast.

While I did experiment with partial loading in the plugin, its not worth it.
Better to have redundancy in SSD then in RAM.

4 Likes
3

My understanding was that if I populate database via ascii, I cannot then use RTD plugin for the same symbol. Am I incorrect?

4

create new db and set rtd plugin, select appropriate number of bars. Dont change these later.

AB Local DB grows automatically as data is imported.
Data Plugins will have fixed size by number of bars. ( DB Settings )

Now, backfill all your symbols so they are created and saved in AB.
( this step avoids new symbols being marked as local only, when imported by ASCII )

Now use ASCII imported to import quotes.
Check the integrity.

When your number of bars will reach max count, the earliest bars will begin to be overwritten.

2 Likes