Auto-Repeat Exploration Hangs/Slows Down (only when DB settings are incorrect)

[trongart]

I have a watchlist of 100 stocks on which I run auto-repeat analysis (exploration) every minute using eSignal. The process takes about 15 seconds.

However, after it runs many times, two problems occur which make auto-repeat/analysis very unreliable:

1. The process suddenly takes 10minutes or more to complete (if at all) unless I manually shutdown and reconnect eSignal. Then it it’s 15 seconds again.
2. The loading bar reaches its end in the analysis window, yet it stays there indefinitely and does not finish.

This is very frustrating. What can be done?

[Tomasz]

Do you have RT subscription for 100 symbols? What slows you down is probably that you might have exceeding subscription limit and constant subscribes/unsubscribes to RT stream get stuck at eSignal Data manager. If that is the case, you should up your subscription to the number of symbols you run scan for.

[trongart]

Is it possible to shutdown and reconnect from a datasource such as eSignal from AFL?

[trongart]

My subscription has 200 RT symbols which is the minimum with eSignal. Do you suggest I sign up for 500 which is the next possible level?

Also my "Number of Symbols Limit" is set to 30 in Amiborker's eSignal Configuration setting.

[Tomasz]

Also my "Number of Symbols Limit" is set to 30 in Amiborker's eSignal Configuration setting.

Oh ... so that is the reason... just increase it to subscription limit minus 1.

[trongart]

Will try! But how is this related? Should it not be safer to have a much smaller number like 30 to make sure subscription limit is never breached?

[trongart]

Really not sure how having a smaller limit affects this: https://www.amibroker.com/guide/h_esignal.html

[Milosz]

Expecting from AmiBroker to scan 100 stocks using realtime plugin and limiting its streaming-subscribed symbols to 30 is illogical, very illogical... It is like expecting from a car with a speed limiter set to 30 mph going 100 mph... You can try restarting the car, going backwards etc, but the simplest solution would be just getting rid of the speed limiter ...

No. Nothing is downloaded. The quotes of streaming symbols are already there in the plugin. Ready to use immediately without downloading anything. Real-time data sources like IQFeed or eSignal stream every tick of streaming-subscribed symbol and plugin holds that data for immediate access. This is nothing new - everything is described / documented in the ADK (plugin development kit) RT subscription limit is known. IB, eSignal and IQFeed have known limits and IQFeed even has methods to query f…

I recommend reading this thread:

Architecture for taking Amibroker /IB live to large universe of stocks on intraday data?

Hi, I was curious if anyone out there had good architectural solutions for trying to adapt Amibroker to actually automatically scan and trade all 6000 nyse/nasdaq listed stocks - and how to run multiple trading systems on that? It would seem like the actually opportunity to continually scan and place orders run into several different types of limits. I've got a few thoughts, but I was curious if the community (Bandy, Thomasz) had good solutions. In the past I've tried to port systems developed …

Besides if scanning only 100 stocks takes 15 seconds, I guess, that either your code is very ineffective or you don't use quick AFL in Automatic Analysis. It wouldn't be an issue if such scan was performed rarely, but if it is repeated every minute, it is a waste of computer's resources.

[trongart]

My understanding was that having a limt of 30 symbols meant that Amibroker would split a scan for say 90 symbols into 3 separate scans in order to avoid going above the subscription limit. And it would unsubscribe the previous 30.

Also I’m not sure why the RT plugin is used in the exploration analysis. What if I needed to scan 5000 symbols?

Sounds like I always have to stay within the max symbols subscription limit as they don’t get unsubscribed.

[Milosz]

Also I’m not sure why the RT plugin is used in the exploration analysis. What if I needed to scan 5000 symbols?

It's your choice if AmiBroker uses realtime plugin or local database. If you use local database, you can scan any number of stocks that you want without any issues. Also compare the time needed to complete the scan. If your code is effective, a Scan or an Exploration on local database, should be performed (a lot) faster.

[trongart]

But the local database would not have bar updates?

[Tomasz]

My understanding was that having a limt of 30 symbols meant that Amibroker would split a scan for say 90 symbols into 3 separate scans i

Your understanding is incorrect. As absolute necessity this "symbol limit" in Configure screen must be set to your subscription limit minus 1 AND you must NOT exceed number of symbols in single scan beyond that limit.

If you have set it to 30 symbols and scan 100 it would mean that there is constant subscription/unsubscription to RT data happening which is prohibited by eSignal. And eSignal data manager actively discourages such abuse by SLOWING DOWN responses to such requests.

The very same rule applies to all RT sources that have limits on number of symbols. Abusing data provider rules by frequently watching/unwatching would bring you trouble and in extreme cases may lead to account termination.