Using Metastock plugin could only retrieve 250 symbols

Hi,

I'm using Metastock data Plugin to get Data in Amibroker, but could only retrieve 250 symbols whereas the folder contains more than 250 securities. Is this per-directory symbol limit is with Amibroker? is there a way I can retrieve more symbols, it's not possible for me to split the directories. (tried with both Master and Emaster).

Second, I have the same data in ASCII format, i.e. a single directory with text files containing no headers and columns are Date, Time, O, H, L, C, V, and symbols name is there in the text file name. This data refresh every 15 minutes through an External program. I know I can import this data though ASCII Import and it works but I can't import every 15 minutes, and it contains 500 symbols it does take some time. Is there any way through a plugin I can get this data into Amibroker just like the Metastock plugin. its local data, there must be some way to get this data in.

Highly Appreciate any Help.

Thanks,

No, there are no limits on number of symbols in AmiBroker.

It is a limit of Metastock MASTER and EMASTER file format.

You should google for information: https://www.google.com/search?q=EMASTER+file+symbol+limit

and Google will tell you.

You have to use XMASTER instead to get more symbols. Or divide Metastock database into several subdirectories.

Note that Metastock XMASTER file format still has limits (65K). AmiBroker does NOT.

1 Like

Thanks for the help Tomsaz,
As i said I can't split the data in multiple directors. The data source don't allow me to do so and I think there is no way i can covert it to latest Xmaster format.

I've the same data in ASCII Text format, is there any way I can get that data in amibroker though any plugin? just like metastock plugin. (this ASCII Data update every 15 minutes)

Thanks,

You can use ASCII importer http://www.amibroker.com/guide/d_ascii.html and http://www.amibroker.com/guide/w_impwizard.html
and OLE interface to import data every 15 minutes http://www.amibroker.com/guide/objects.html

Thanks Again, A small problem I have 400 symbols with around 500 mb of data, its a bit time consuming to import every 15 minutes.

Every time we import data through ASCII it perform a complete import, is there a way to do only a incremental import like if the file have a data for yesterday, it only does a import for ticks made today in ASCII, is it possible?

I m Coming from Tradestation, I'm in love with Amibroker exploration speed...... it really fast.. !!

Thanks

It solely depends on contents of these files. If you want incremental imports, you should have incremental files (i.e. separate files for each day).

Currently I've these are consolidated files, each file is one symbol with data from 2008, and i have 400 of these, an option in wizard to select import range would be great to do incremental update to existing database.

I have to figure this out to make full use of amibroker and test my strategies on all symbol. can walk an extra mile and code stuff in python etc. in which direction i should head to look for a probable solution.

Thanks :slight_smile:

Hi Tomasz,

While using the import function, we have an option to skip the first specified lines, Can we get an option to import last specified line, It would be very much useful to get only required data imported in case some one has clubbed bulk files like me.

Thanks,

AmiBroker is fast and it does not require much time to add quotes once data are already in AB process space especially if quotes are in ASCENDING order. What takes time is file access by Windows, opening and READING the file, not importing. What is slow is:
a) antivirus scanning all data files over and over again with each read (yes it is absurd and you can disable certain directory from AV scan)
b) Windows file system limits reading speed

If your data are in ascending order you can simply add
$SKIPLINES 10000
and it will skip that many top lines. Reminder will be imported but it does not matter as I wrote as AmiBroker is fast.

Then true gains can be achieved if you have SEPARATE files for each day and for each symbol. It is much faster that way.

1 Like

Why don't you export just the most recent data through that external program?
Not possible?

1 Like

Hi Tomasz,

I 100% agree with you that amibroker is really fast and I love its speed, in fact, I've written I my earlier post "I m Coming from Tradestation, I'm in love with Amibroker exploration speed...... it really fast.. !!"

about skipping the first 10000, I'm already doing it. from there only I got a thought that an option to import last few lines could be useful. (though I know I'm missing newly added symbols)

As u suggested, I disabled the antivirus, now it takes 11 seconds for a full 400+ scrip import. Antivirus saved 5 Seconds.

I know amibroker can import in a blink of an eye if there had been incremental files...
I"m using a really good system, with SSD, its not that slow, but I like the easy useability of metastock plugin.

Hi fxshrat,

Yes, my external program doesn't provide data extract for last few days. It just add up the line to the existing ASCII files, every 15 minute. I was using Tradestation earlier, there i could just map the datasource once and tradestation used to read it from the source and no imports were required. just like how the metastock plugin work with amibroker.

I'm new to Ami, and I really don't know whether a plugin kind of stuff can be make which can acess data from the Ascii files, just like the metastock plugin does.

In the ADK (AmIBroker Development Kit) there is a sample plugin that does that (reads ASCII files directly) but:
a) it has fixed format for data files
b) it still reads entire file

Tradestation also needed to read entire file so I don't think if there is any possible gain here.

If your program just adds the line to existing file, an easy solution is just to DELETE (or make empty) file after import. AmIBroker keeps data in its database so, you don't need lines that were already imported. Simply deleting the file solves the problem as only new data will ever be imported that way.

1 Like

Hi Tomasz,

Excellent Stuff Man!! I deleted all file, ran my program for data and import took only 1 Second. Wow!!
Great!!

Well I really don't mind reading the entire file and check, if can I can get the plugin, My ASCII File format is - Date, Time, Open, High, Low, Close, Volume, Date format - DD/MM/YYYY and Time is HHMM. Want to use it on 1min huge data with to backseat. sorry, I don't understand C++, can work with python.

It just came to me that I could add simple

$SEEK offset

command to ASCII importer. And it would just do file seek to specified offset and start reading from given point. Negative offset will refer to end of the file and this way you would be able to read starting from ending "N" characters of the file. This, together with $SKIPLINES 1 (that will skip incomplete line that may occur due to seeking) will allow to read one or more full lines from the end.

I tried this but it takes 13 seconds, don't know whether I'm doing anything wrong, here is the code.

$FORMAT Date_DMY, Time, Open, High, Low, Close, Volume
$SEPARATOR ,
$CONT 1
$GROUP 255
$AUTOADD 1
$DEBUG 1
$NOQUOTES 1
$SEEKoffset -50
$SKIPLINES 1

There is no such command yet. I was just talking about possibility adding such option in the future.

Oops my mistake, I thought we have a command like that. :smiley:

Well, Tomasz, It would be of great help, with this I can continue with my existing database and need not delete anything. Further, this command may prove to be helpful to many other users also, who are importing their data from different formats, and having a piled-up data in a single file, Simply defining the no. of rows from the end, data import will become speedy like a rocket!!

Thanks a ton !!

Hi Tomasz,

Thanks for sharing this thread and guidance to use XMASTER, but I still have the problem that I can not see some symbols.

I have already tried to remove all folders, include again and retrieve but no success. Is there any way to clean or reset the symbols already connected so I can start from scratch and see if can see all symbols.

If you have any other tip or guidance that I can try please let me know. Thanks

Talk to your data vendor. MS plugin works fine for thousands of symbols using XMASTER file and that is proved by the fact that Premium Data were using MS format for years and there was never any problem with using MS plugin.