Import wizard for huge files (5 million symbols,2.5GB+ into 230GB database)

May I further share my experience with Large Database in somewhat similar scenario?
Many years ago, I was involved in a research project involving Large Database (Not so large as the one being discussed here) - involving Importing Large number of Option Symbols.

Overall the experience was very good and we could test what we wanted to test - Per my experience AmiBroker was possibly the only one providing so much freedom and versatility. In the years after that the program has further evolved much.

One Practical thing that had made the things much managable was:

While working with Intraday Data we should take guidance from EOD data also -
Here in India we get EOD Tradefile from the exchanges - it is all in public domain. This file has details like

Number of Trades during the day in a particular security
Open Interest at the end of the day in a particular security
Total Value of Trades during the day in a particular security etc.

from this file we could find out securities that had very few Number of Trades, Open Interest, Total Value during its lifetime and/or had been deep out of money during its lifetime. - or we were not interested in the Underlying Security itself. For our testing we were not interested in these securities. The external application that was pre-processing the Intraday data filtered out such securities (We Simply did not import such securities) - and finally the Number of Securities in AmiBroker Database was reduced by around 70% (Total Database size was smaller by around 10%-20% only - but broker.master was lot smaller) - and overall the database started making better sense.

And again - it felt so good when we could do what we were looking forward to do.

With Regards

Sanjiv Bansal

You wrote a lot, but still you did NOT answer this basic question:

Please tell me what import TIME you are getting now @Sumangalam and how many symbols?

I don't need "life stories". I need numbers. Numbers are crucial. "Large" means nothing. It is subjective term without quantitative meaning. For one "large" means 100MB for the other 100GB. 1000x difference.

2 Likes

Dear Mr. Tomasz,

Thank you very much for guidance.
Details sent to support (#177048-R1612-61)

With Regards

Sanjiv Bansal

What is the standard percentage of memory available in Amibroker?, if have been using Amibroker for a long time. What need to clean the memory or cache it?. Thanks

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.