Architecture for taking Amibroker /IB live to large universe of stocks on intraday data?


#21

You really should re-read my posts and the ADK docs. The data are auto-appended. BY THE DATA PLUGIN. See also http://www.amibroker.com/guide/h_workspace.html

Data plugin has TOTAL control over data and it communicates with external data vendor APIs, subscribes to RT stream and builds (auto-appends) data to array that is passed AS POINTER in GetQuotesEx call. Everything (with examples) is in the ADK documentation.


#22

I originally was speaking of the already built in support for feeds like esignal, IQfeed, Dtn etc. From my reading above they do not auto append data files unless things like in quote window, currently charted scan or exploration etc. When you refer to data-plug in are you referring to a data plug-in we code ourselves and implemented via the SDK? If so then all quotes are stored, no matter if the symbol is in quote window, currently charted…etc.?


#23

Not everything that you read from somebody else than me is actually precise. Except me, everyone else is just a user. They may or might not be correct or may be partially correct, because no one except me actually coded the program. The problem is that people take my description from somewhere take 5 bits out of 10 that I wrote, fill the remaining with their “own interpretation” and then post somewhere. And information noise of half-truths is created this way.

What you “read from the above” is not (entirely) correct. In short with eSignal or IQFeed ALL symbols WITHIN subscription limit, will get updated and appended, if only you access them once. This is more or less the STANDARD way of working with ANY real-time data source as you need to SUBSCRIBE to updates. And symbol once subscribed remains subscribed and data are collected as long as you don’t EXCEED your subscription limit.

Things are A LOT more complicated that very simplified picture presented above. If I were to describe ALL inner workings of AMIBroker I would need to write a book that has 100000 pages. I don’t have time for that and I have no desire to do that because all the competition with everything handled on silver plate would steal ideas from us (they already do that).


#24

lol ok understood regarding debth of material and disclosing.

Just so I understand then, even if you ran an empty exploration Filter=True, which creates the subscription (an automatically subscribed watchlist might be nice to have, then we can explore logic to that list etc); Anyway, then all those symbols (post subscription) will be on autopilot, meaning initial backfill and real-time updates (10/sec each) appended price array and local storage? Then completely referenced thru ADK?


#25

Again, this is your interpretation, not facts. The reader’s interpretation is as always skewed a bit. That is similar to a kids game called “Chinese whispers” or “Chinese telephone”. The person on one end says something but on the receiving end we get something completely different. Yes an exploration would subscribe to RT updates, but for example you don’t need to even set Filter = True. It can be set to false. Also real-time updates are appended not 10/sec but immediately (if there is 5000 ticks per second then it gets updated 5000 ticks per second). 10/sec is NOT data update rate, but the maximum frequency of single CHART redraws when triggered by plugin. But if you use other refreshes (such as mousemove) it can be 1000 per second. As I wrote, there are GAZILLIONS of details. And RT data are kept in RAM, not in “local storage” (i.e disk).


#26

Tomasz, thanks to your replies, I know, that the part of my reply in which I wrote that after running a Scan/Exploration on some Issues, those issues are not fed in real-time after the Scan, is not true. Those symbols become subscribed and are updated - as long as they stay within RT subscription limit. Sorry for that. I don’t want to misinform anybody, so you can edit or delete the part of my reply containing wrong information. I hope, that some other things that I have mentioned are useful to some users…

… but let me briefly explain why I have written it. I’ve been scanning the market (using Statica plugin) for many years. I always start with a pre-scan to allow AmiBroker subscribing issues and filling data holes. Only when this phase is completed (it takes some time) I am ready to run live scans during the session. The session starts at 9 a.m. and the market closes at 5 p.m. What I have observed right from the beginning, is that as long as those scans/explorations are run, all symbols’ quotes are updated and when I save the database, the quotes are complete. But if my last scan is run for example at 4 p.m. (and I save database at 5 p.m. when the market closes) when I later review the charts/quotes after the session (offline) most symbols’ history ends at 4 p.m. (their quotes have not been updated after 4 p.m.) What might be the reason for that? Is it because:

  1. RT subscription limit is small and most issues are not within it

or

  1. The plugin downloads and collects all the necessary data, but because those symbols have not been requested by AmiBroker to the end of the session, they have not been saved in AmiBroker’s database.

I have asked “How can RT subscription limit be recognized if the data provider doesn’t specify it?” because I can’t find any information concerning Statica’s limit.

One more question. I have not used eSignal or IQFeed, so forgive me my ignorance, but what happens if a user who uses a plugin with RT subscription limit let’s say 1000 issues tries to scan 2000 symbols? He should be able to scan the first 1000 issues (because they are within the RT subscription limit) but when he exceeds this limit, (according to the FIFO rule) those 1000 issues should be gradually removed from the RT subscription list and replaced by the newest issues… So the first 1000 symbols should be replaced by the newest 1000 symbols. If that is the case, is the user able to get the latest quotes of all those 2000 issues when running one scan? Important remark: I am aware of the fact, that the latest quotes for those issues which are outside the RT subscription limit may not be available at once (but they should be available in subsequent scans even if the Wait for backfill option is not available). Of course I assume that the database is updated regularly (for example every day) and misses only the newest quotes. A quote from Tomasz’s reply from another thread (I have already provided the link to it):

Wait for backfill flag instructs Analysis window to wait until IQFeed sends all requested data. Without that Analysis would progress with your existing data (stored already in the database), without waiting for update. The update will eventually arrive later and will be included in subsequent scans.

Thank you


#27

Statica is NOT our plugin. It is 3rd party development that is done without our supervision. We are not responsible for 3rd party developments. Everything what I wrote applies only to official AmiBroker.com plugins that are developed by us.


#28

That’s not how it typically works. It’s a hard fixed limit. However, it’s theoretically possible that a plugin could unsubscribe – if the data feed supports that – to free up the slots for new tickers during a scan. But, that would be gaming the limits and its unlikely to be tolerated or officially supported.


#29

All our plugins indeed unsubscribe ‘oldest’ symbols once limit is hit, but it is not really reliable and as @TraderX noted it is not officially supported by data vendors. It is highly recommended to stay within limits.