How to force a backfill for a chart that is plotted using SetForeign()

[polomora]

Hello,

I have recently started using IQFeed as my RT datafeed provider, mainly to have a reliable source of TICK and ADVANCE-DECLINE data.
I use the SetForeign() to plot this data, I have this code is its own chart pane, as follows

 adIndex = SetForeign( "JINT.Z");  // IQFeed ticker for ADD
 adIndexColour = IIf( C >= Ref(C,-1), colorYellow, colorRed ); 
 Plot(C, "", adIndexColour, styleCandle);

In general, this works fine. However, it doesn't backfill, if I start Amibroker after the market open. To backfill, I have to enter the chart ticker in my main chart window, to force the backfill to be done. Is it because of the SetForeign() that this chart doesn't backfill?

[travick]

Backfill has been explained here
http://www.amibroker.com/kb/2014/09/23/how-to-backfill-all-symbols-in-rt-database/

As a general rule, all real-time data sources provide backfills on-demand, which means that backfill for each symbol has to be requested separately.

This is done when the symbol is primary either in chart or AA.

I don't know how much sense and feasible it is to request Backfill for a Foreign() symbol though.

[polomora]

Thanks @travick

I had seen that article. It explains the backfill in the context of displaying the chart of a given symbol, or accessing the symbol through the Analysis window. There is no dedicated AFL funciton to this, for example Backfill(symbol)

[snoopy.pa30]

@polomora Would running an Exploration on all the items you want backfilled work?

[polomora]

Here's a chart showing the problem. For the IQFeed symbol @NQH19, I want to display the NYSE-ADD and TICK, but they are not backfilled

[Tomasz]

Data are backfilled when they are needed (i.e. when given symbol is selected for chart).

Did you try to display those symbols SEPARATELY (select them as CURRENT SYMBOL)? And does IQFeed even offer backfill for those? The availability of backfill depends on DATA VENDOR, not application.

[polomora]

Hello Tomasz,

Yes, I can display these Tick and Advance-Decline symbols separately, when I enter their ticker symbols into the price chart. I assume that backfills supplied by the data vendor are working.

I have separate AFL coded into specific files to display the charts for Tick and Advance-Decline using SetForeign(). I display these studies as separate panes on the chart by using Charts -> Insert Linked. For these two chart panes the backfils are sometimes failing.

When I copy-paste the same SetForeign() AFL into my my price AFL file, then the backfills are working, when I select "Force Backfill".

As I already mentioned, it is inconsistent. Sometiems the backfill for these separately Tick and Advance-Decline chart panes work, sometimes not.

[Tomasz]

Backfill works the SAME regardless of how it is triggered. It is SAME code path and exactly the same code is executed no matter how it is triggered.
So there is NO inconsistency.

What may happen however is that if you issue multiple backfill requests they CANNOT be handled by IQFeed immediately. IQFeed would choke on too many backfills. Therefore, the requests that exceed IQFeed capacity are queued and done only AFTER previous backfills are completed. There is no way to 'force' anything. Physical world has limits. The internet connection has limits, IQFeed has limits. You just have to wait, not everything can be done "at once". That is the reason for existence of options like "wait for backfill" in the Analysis window, so you just wait for IQFeed to deliver data.

[rottor]

I was using Exploration to backfill data for many symbols

what I noticed is that if I wanted to add data at the beginning of the price array (to get longer history), Amibroker assumes nothing changed in last request and subsequent Explore didn't trigger any new request from my plugin

I guess it only checks the last timestamp

[polomora]

I was wrong in my previous message

When I copy-paste the same SetForeign() AFL into my my price AFL file, then the backfills are working, when I select "Force Backfill".
As I already mentioned, it is inconsistent. Sometiems the backfill for these separately Tick and Advance-Decline chart panes work, sometimes not.

The charts displayed using SetForeign() are NEVER updated using Backfill, even when I paste the SetForeign() AFL into the price chart AFL.
I don't understand what you mean by "There is no way to 'force' anything. Physical world has limits." The Backfill for charts displayed using SetForeign() never completes, unless I am doing something wrong.

What can I do to make this work? Or is this something that cannot be done?
In the attached chart I have displayed a basic price chart with a Tick chart superimposed.

[Tomasz]

They ARE updated.

Your picture just shows that THERE ARE data displayed till the very last bar.

[polomora]

The chart shows Price data overlaid with Tick data.
I stopped AB, and switched off my PC at approx 10:15 US market time. I restarted my PC and restarted AB at approx 11:10.
I backfilled the chart. The displayed Price data was successfully backfilled. But the displayed Tick data was not backfilled. There is a gap in the displayed Tick data between 10:15 and 11:10.

[polomora]

I've done some further tests. What I did was the following:

• start AB during market hours, and backfill the price chart pane. The Advance-Decline and Tick chart panes ARE updated
• put the PC into standby
• after one hour take the PC out of standby, and stop/restart AB
• backfill the price chart. The Advance-Decline and Tick charts are NOT updated
• stop AB, put the PC into standby
• the next morning, take the PC out of standby, and start AB
• backfill the price chart. The Advance-Decline and Tick charts are ARE updated

In summary, it seems that the backfills of charts plotted with SetForeign() are not updated during market hours. It's not a show-stopper, I've posted this update for furure reference

[Tomasz]

First: don't put PC on standby. This will disrupt data flow and create data holes.
Either shutdown it completely (or shutdown/close BOTH AmiBroker AND IQFeed CONNECT),
or leave it open.

During even lightest "S1" sleep state

Processor clock is off and bus clocks are stopped.
System Sleeping States - Windows drivers | Microsoft Docs

CPU clock off - means that software can not do ANYTHING. No activity is happening.

Putting computer on sleep creates a problem in first place.
You won't be needing ANY "manual" backfills if you did not place on sleep as there would be no hole. Also if you closed program completely it will automatically pickup from where it was last time once you start it again and backfill automatically.
If you do sleep, it is like the program got unconscious for a couple of hours not knowing what happened.

As for your latest (incorrect) assumption about charts they are they are updated always, but IQFeed servers during market hours actively do throttling and respond slower to backfill requests so during market hours it takes longer.

They are pretty silent about that but if you search the web deeply, you could find several report and mentions even if their own docs

From:

IQFeed_FAQs.pdf

Some find print here and there

During this market time, the download is limited to eight calendar days.

and some other limits if you are on IQFeed trial (it was the reason of several complaints by users who were not made aware of this practice)

Trial users are limited to four days of intraday history and 365 days of daily bars.

and

Trial users are limited to four days of tick history

Note that document is old (from 2019) and data vendors may implement changes to their feed on daily basis, so current situation may be different (even with more limits in place).

For example, in 2020 they added extra limit for ALL users, see:

Force backfill of IQFeed

I just received the notice from IQFeed that they are now implementing new historical rate limiting on their backfill servers. The limit will be upto 50 requests per second. AmIBroker is unaffected by this change because even without "wait for backfill" we don't do more than 4 simultaneous requests and WAIT for arrival of previous responses before requesting more (so not more than 4 simultaneous backfills are running at a time). Since ping time is typically well above > 20ms you technically not …

BTW: I will probably need to add detection of "sleep" state during active RT streaming and tell the users NOT to do so.

[polomora]

Hello Tomasz,

Many thanks for the detailed explantion. I won't put my PC into standby any more.

Today, I've been having stability problems with my internet connection. I reset both my modem and router. which took about 10 minutes. When everything was restored, I backfilled the chart panes. The price chart pane was successully backfilled, but the Advance-Decline and Tick chart panes were not backfilled.

[Milosz]

@polomora

Many thanks for the detailed explantion. I won't put my PC into standby any more.

See here:

Force backfill avoidance

@uvdsatish probably an alternative option is switching to local database before putting your PC to sleep and switching back to your plugin after resuming your work. At least I always save my database and switch it to local before putting my PC to sleep and I've never had any problems with automatic backfilling data when resuming my work. But I'm not using IQFEED - that's why I started my reply with "probably". [1] [1] BTW it is always better to run backtests, explorations etc on a local …

[polomora]

Many thanks for the tip about the local DB!