I would like to connect eSignal to Amibroker and I can't find clear information about limits of symbols I can follow/backfill (eSignal is not able to help - they do not see the difference 
).
I need to make exploration in Amibroker every 5 minutes and I need to explore around 8000 symbols. Of course, I have to force backfill to get data of all of the symbols - do I receive historical data of all of 8000 symbols or just of 500 (this is eSignal limit for real-time data, but maybe not for historical data backfill)? I don't need to display current quotes (real-time data), but I just need to have previously closed 5-min candlestick of every of 8000 symbols.
If it's impossible via eSignal, do you know any other data vendor that can do it?
Have you ever tested e.g. polygon.io ? Is it possible to connect it to Amibroker with data backfill?
First no "force backfill". Force backfill is manual operation and it downloads ENTIRE history and it is rather slow and too resource consuming to do for 8000 symbols every 5 minutes.
eSignal will BAN you if they detect you are doing such thing.
With NORMAL backfill (only say 5 bars that were missing) that occurs automatically it is theoretically possible but against eSignal subscription agreement. You are not supposed to explore 8000 symbols if your subscription covers just 500 symbols. eSignal has various packages including professional one that allows 2000 streaming symbols.
Thanks for the answer.
So what can I do to explore 8000 symbols? Is there any data vendor the works properly with Amibroker? Maybe Polygon, but can you give some advice where to start to connect it to Amibroker?
"Properly" ? Every vendor works properly. But 8000 symbols in streaming real time cost A LOT of money and A LOT of bandwidth (DSL is NOT enough). eSignal (or Interactive Data) offers enterprise-level solution for that many symbols but costs tons of money and as far as I know is not available to individuals. On the other hand you from what you wrote you don't need real time at all. You would need 5-minute snapshots. Something like that could be done with Tiingo, but I am not sure if they want their servers to be bombed with requests like this. Polygon is just a REST based web site that does not look to provide high-performance. And is not really supported by us.
Your user account isn't marked with License verified badge. Did you purchase AmiBroker license?
Only users with License verified badge are allowed to post on this forum.
So before posting, make sure you verify your license as explained here.
Yes, that's right - I need, as you wrote, 5-minute snapshots, because my procedure is pretty simple:
- exploration of (almost) all US stocks (8000 symbols) with some preliminary filtering (should be executed every 5-10 minutes),
- creation of WatchList that consists of stocks selected by exploration in the first step,
- another exploration (using other AFL formula) of WatchList (should be executed 1-2 minutes),
I was testing it on Polish GPW - it worked OK with Statica plug-in, but the number of stocks was smaller and sometimes there were problems with backfill (I know that Statica plug-in do not support "wait for RT backfill"). Now I want to run it on NYSE and Nasdaq and that is why I wonder if eSignal do the job for me.
@piotrd I don't use intraday data so can't help you with your question directly but may I suggest that you do some more research on the U.S. stock market before trading it. You want to load data on 8,000 stocks but there are really only approximately 3,600 publicly traded companies. Even the Wilshire 5000 (an Index that is intended to have 5,000 stocks and represent the entire market more or less) only can find 3,473 currently valid constituents.
I suspect a real-time data feed on 3500 stocks will still be very expensive.
A couple of decades ago there were more stocks (~7,600 at the peak) but that was long ago in the mid-1990's. Perhaps you intend to trade mutual funds, ETF's, preferred shares etc and that's why you are looking for 8,000 symbols. Good luck.
Well, it actually doesn't matter if it's 8000 or 3000. The problem is a huge number of different stocks. I filter them twice to reduce calculations of Amibroker. So, as I mentioned, my first step is general filtering that results in 300-400 tickers (out of 8000+; yes, with ETFs). Second step is signal-finder.
As Tomasz mentioned - "full" real-time data is not necessary because in this case I don't need every tick. I just need 5-min bars (but they have to be "fresh", it means not delayed), so the range of the data is very limited (around 40,000 records every 5 minutes).
I was also thinking of the first-step filtering in TradeIdeas and after getting the results exporting them to Amibroker. TradeIdeas can write the results to csv, but it has to be done manually. Then reading the file by Amibroker and creating the Watchlist that is built of TradeIdea results (i.e. tickers). For me the problem is manual creation of a file in TradeIdeas.
You could easily get your criteria down to 1-2k symbols just wth a liquidity filter.
Take the Russell 2000 and even its smallest caps are quite thin in volume compared to the top.
You're right, however, I also look for stocks that are usually not liquid and they become liquid just for a few hours, e.g. after some important news concerning such companies liquidity, volume and spread become "tradeable". That is why I need to scan (explore) many different symbols.
Canât think of any retail software thatâll handle 8000 symbols or any data vendor. Nanex possibly but youâd be writing your own plugin.
Any software Iâve seen that can get you 8000 symbol scan is usually very basic. For instance TC2000, scan criteria is very simple or many brokerage platforms offer basic scans. The way they operate is you set criteria( volume increase as an example) they compute on their end and send you the results. Bandwidth is way smaller this way as their not 8000 data points a second but merely sending a small scan result back.
Only thing I can think of is contacting QCollector and seeing how their software throttles Esignal so you donât get dinged. They offer a software that pulls from Esignal and IQFeed and puts it in .csv files. Not sure how it would work against Esignal with 8000 symbols every 5 minutes. Think IQFeed be more leniant but could be wrong.
Then you could import into Amibroker from .csv. Be a bit of a half ass way to do it. Other option is write a plugin thatâs throtteling requests and believe Esignal charges for their development kit and API to even develop such a tool. Theyâd rather you use their platform.
AmiBroker can handle more than 8000 symbols easily. Problem is not the software but data source.
I think I found some solution. I can make pretty strong filtering via TradeIdeas, maybe I can shrink the number of symbols to 100-300. Then I save the TradeIdeas' results to csv (probably there is some API, but it'll be easier to use some software that automatically click the mouse buttons (e.g. Mouse Macro Recorder or sth else)). Then I import the file to Amibroker and update the WatchList with symbols listed by TradeIdeas and make exploration.
I hope I'll be able to convert somehow csv file with many different columns that are exported by TradeIdeas and just read symbols and then update the WatchList...
Could you please give me some hints how to change the set of symbols in the WatchList by reading it from file (csv)?
With AFL I can read symbols in my WatchList, I can delete them and I can add new symbols as a result of my exploration. However is there any possibility to add new symbols on the basis of csv file?
In csv I've got a few columns and in one of them (first) there are symbols (stocks). Do I need to convert the column with tickers into one-line comma separated file only with tickers before importing to Amibroker (I can do it automatically in Excel, but maybe Excel is not needed here) ?
Havenât verified video but he has done some good videos Iâve watched before.
First link after googling âAmibroker import watchlistâ.
I saw it - it's fully manual and unfortunatelly doesn't solve my problem at all. I'm looking for automatic solution and launched from AFL.
My file csv looks like this:
Symbol,"Company Name","Price","Change from the Close","StockTwits Relative Activity","Cash","Position in Year Range"
As you can see, I've got many columns and I'm interested in the first column (Symbol) only.
I've used this to extract symbols only:
fh = fopen( "ti.csv", "r");
if( fh )
{
linecounter=0;
while( ! feof( fh ) )
{
line=fgets(fh);
if(linecounter>0)
{
printf(StrExtract(line,0));
}
else
{
}
linecounter++;
}
}
else
{
printf("ERROR: file can not be found (does not exist)");
}
fclose(fh);
It extracts symbols however it adds quotation marks "" (quotation marks are not in csv). So in csv there is Symbol (without "") and after reading it in Amibroker I receive "Symbol" (with ""). Is there any way to remove it? I was trying StrReplace but without success.
Now, I've got my list (displayed in interpretation window) and I would like to make of it a WatchList.
I use this to make Watchlist from filtering results (in some other case):
listnum = 10;
if ( Status( "stocknum" ) == 0 )
{
// retrieve watchlist members
oldlist = CategoryGetSymbols( categoryWatchlist, listnum );
// iterate through the list and remove tickers
for ( i = 0; ( sym = StrExtract( oldlist, i ) ) != ""; i++ )
{
CategoryRemoveSymbol( sym, categoryWatchlist, listnum );
}
}
// check how many times Filter variable was true in the tested range
// if non-zero value detected, add current symbol to a watchlist
if ( LastValue( Cum( Filter AND Status( "barinrange" ) ) ) )
CategoryAddSymbol( "", categoryWatchlist, listnum );
and I don't know how to change last if... to add symbols of my csv list. Any suggestions, please?
You could avoid writing any code at all and use the batch import function. You would create an import definition file, create the batch, configure the batch to import the list, and then schedule the batch to run automatically every morning.
https://www.amibroker.com/guide/d_ascii.html
https://www.amibroker.com/guide/h_batch.html
Another non-coding way would be to export the csv file directly to the watchlist folder in your AB database. AB should not be running when you export the file there. When you launch AB, the watchlist will magically appear.
But if you really want to use AFL to do it, you would have to open the csv, loop through all the tickers in the file, and add each one to the watchlist using CategoryAddSymbol. This is definitely the most complicated way to go.
I hope this helps!
I just thought of another non-coding way: setup the ascii import, and then add a custom menu item to run the import. It's manual but it's super easy to use as long as you remember to do it.
https://www.amibroker.com/guide/w_customize.html
See for example, you would change the Arguments to be your csv file and the Initial directory to contain your ascii format file.
@piotrd, when your strings contains a " (the double quotation mark i.e. the string delimiter in AmiBroker) you need to 'escape' it as in the following example:
str1 = "\"APPL\"";
str2 = StrReplace(str1, "\"", "");
Title = "Before: [" + str1 + "] - After: [" + str2 + "]";
Supported 'AFL escape sequences' are: \n, \r, \t, \" and \\ (the last frequently used in filenames).
See also this past thread, if you want to learn more about the use of double and single quotation marks in AmiBroker.
Thanks, I'll try with batch function, too.
I don't want it to be manual (that is why I'm looking for AFL solution), because my first problem was with huge number of symbols (8000+) analyzed every 5 minutes. I know that Amibroker can handle it, but data vendor - not really.
Now I decided to filter stocks in Trade Ideas first, then export (every 5 minutes!) the list of filtered stocks to csv (I tried it with Mouse Macro Recorder - it works OK, however it would be better to download it directly by Amibroker) and import the list into Amibroker and save it as Watchlist.
Below my solution with importing to Ami and saving as WatchList (maybe it would be useful for other traders). After it, I can run my main Exploration with strategy and with backfill option on limited number of stocks.
Thank you all for kind help.
listnumTI = 11; // we use watchlist 11 for storing results
// erase the watchlist when we process very first symbol
if ( Status( "stocknum" ) == 0 )
{
// retrieve watchlist members
oldlist = CategoryGetSymbols( categoryWatchlist, listnumTI );
// iterate through the list and remove tickers
for ( i = 0; ( sym = StrExtract( oldlist, i ) ) != ""; i++ )
{
CategoryRemoveSymbol( sym, categoryWatchlist, listnumTI );
}
}
fh = fopen( "D:\\file.csv", "r");
if( fh )
{
linecounter=0;
while( ! feof( fh ) )
{
line=fgets(fh);
if(linecounter>0)
{
importsymbol=StrExtract(line,0)+".US";
symbol=StrReplace(importsymbol,"\"","");
printf(symbol);
CategoryAddSymbol(symbol, categoryWatchlist, listnumTI );
}
else
{
}
linecounter++;
}
}
else
{
printf("ERROR: file can not be found (does not exist)");
}
fclose(fh);