IQ Feed for real-time scan

Data Sources
#1

I had posted this question in yahoo forum, but did not get an answer. I am using IQ Feed as a data source for capturing real time data. I am doing scan periodically on watchlist of 30 symbols intra-day programatically throught scripts , exporting scan output to files, and other scripts read these files to send signals to other applications. ( This means I am not having any charts open on amibroker - just analysis window opens programatically and does the scan.) I am seeing delays in the signals i.e bars are created late by several minutes. It does not happen all the time, but it does happen occasionally. Any body had such issues?

Is there a pre-requisite (like symbols should be open in some charts etc) for real-time data to be stored in database before analyis window can work on it? ( I am using wait for backfill feature also in analysis window)

Thanks for your answer

Architecture for taking Amibroker /IB live to large universe of stocks on intraday data?
How to optimize backfill with IQFeed (reopened)
Local computer problem, was: Backtests are running extremely slow
#2

With “wait for backfill” turned on you should be fine with 30 symbols watch lists. The only advice is to run initial scan before session opening as backfills are faster outside market hours. Problems with IQFeed may only start if you exceed IQFeed subscription limit (more than 500 symbols, or whatever your subscription limit is).

#3

I have noticed what you advised. So i was indeed planning to run an initial scan programatically before market open. However I am not sure if that would help delays like 15 minutes-30 minutes. I have included more logging inside the AFLs now so that I can pin point exactly what is going on. The last time I noticed the issue was on 15/06 when one of the AFL working on XIV/VXX filter gave signals at 16:15.03 CET for a bar that closed 15:45. CET (chart interval was set to 15 minutes and scan done every 15 minutes with a delay of 3 seconds after system time). I have included the below into the AFLs so that I get a log. If you have suggestion to log something more, please advise. I will hopefully come back with more details when problem shows up again. On that day, logs were created, but I deleted them accidentally after analysing them. What I could infer is that GetRTData(“UpdateTime”) ) showed timestamps correctly - not sure if that means data for all symbols also arrived correctly - But timenum() was still showing values from yesterday’s market close time i.e. it looked like new bars were not created for 15:44:59 and for 15:59:59 until sometime after 16:00:03 CET. Signal got generated at 16:15.03 for the bar ending at 15:44:59.

fh = fopen( "C:\\daytradeagressive2.csv", "a"); 
if( fh ) 
{ 
       fputs( Name(), fh );
       fputs( " currentsystemtime ", fh );
       fputs( " lastvaluetimenum ", fh );
       fputs( " statuslastbarend      ", fh );
       fputs( " statuslastbarvalue      ", fh );
       fputs( " statusprevbarvalue      ", fh );
       fputs( " statuslastrtupdate      ", fh );
       fputs( "GetRTDataupdatetime      ", fh );        
       fputs( "GetRTDataupdatevalu \n", fh ); 
       fputs( Now(2), fh );
       fputs( "                ", fh );
       qs = StrFormat(" %.0f", 
                     LastValue(TimeNum()) ); 
       fputs( qs, fh );//timenum
       fputs( "            ", fh );
       
       qs = StrFormat(" %.0f", 
                 Status("lastbarend") ); 
       fputs( DateTimeToStr(Status("lastbarend")), fh );//statuslastbarend
       fputs( "     ", fh );
       
       qs = StrFormat(" %.4f", 
                     LastValue(C) ); 
       fputs( qs, fh );//timenum
       fputs( "            ", fh );       
       
       qs = StrFormat(" %.4f", 
                     LastValue(Ref(C,-1)) ); 
       fputs( qs, fh );//timenum
       fputs( "                ", fh ); 
       
              
       qs = StrFormat(" %.0f", 
                 Status("lastrtupdate") ); 
       fputs( DateTimeToStr(Status("lastrtupdate")), fh );//lastrtupdate
       fputs( "       ", fh );       
       
       
      qs = StrFormat(" %.0f", 
                     GetRTData("UpdateTime") ); 
      fputs( qs, fh ); //GetRTData
      fputs( "                  ", fh );       
       
       qs = StrFormat(" %.4f\n", 
                     GetRTData("Last") ); // RT data lastprice
      fputs( qs, fh ); //GetRTData
      
   fclose( fh ); 
}

You did not explicitly answer my question about pre-requisite for scans. But I assume amibroker dont have to do anything like opening charts ( although I do have all the symbols in realtimequote open ) etc before doing a scan, correct?

Moderator comment: when posting code use [code]....[/code] tags or use three backticks at the beginning of the code and three backticks at the end.

#4

There are no prerequisites as long as you use wait for backfill.
Wait for backfill flag instructs Analysis window to wait until IQFeed sends all requested data. Without that Analysis would progress with your existing data (stored already in the database), without waiting for update. The update will eventually arrive later and will be included in subsequent scans.
Once data are backfilled and streaming, there is no delay.

#5

I just saw the issue reappear today. SInce I have some logged information, I can clearly see that bar close was wrong in real time, which of course got corrected by the end of next interval. I have a guess regarding what could be the problem -

Does ‘wait for backfill’ option in analysis widnow wait for all symbols referenced in AFL? The reason why I am asking is I have few setforeign calls within the AFL. With wait for backbill option, is it guarenteed that all data pertaining to such symbols will be backfilled before AFL is executed?

#6

Since I did not get a reply - am posting this again…

Does ‘wait for backfill’ option in analysis widnow wait for all symbols referenced in AFL? The reason why I am asking is I have few setforeign calls within the AFL. With wait for backfill option, is it guaranteed that all symbols referred by setforeign will be backfilled before AFL is executed?

#7

You RECEIVED response and detailed advice in very first reply. Just follow the original advice.

Wait for backfill only waits for symbols in the Analysis list, i.e. symbols being CURRENTLY analysed, not the other ones.

It does not wait for FOREIGN calls. Foreign gives you EXISTING data at the moment of the call and does not wait for external source. The request for data is made and it will arrive after few seconds (depending on your data source) but execution will NOT wait for Foreign calls because non-responding data sources would then block the formula from completion at all.

The best way to make things current is to run PRESCAN once at the beginning of the day for all symbols you are interested in. Once you do that all that symbols are streaming and kept up-to-date so you don’t need to do wait for backfill anymore for given day. This was my advice from very first reply in this thread and I can just REITERATE this again.

#8

Thanks for the reply. I am doing the prescan few minutes before market open everyday. May be I will repharase my question because I guess I am expecting wait for backfill to do something which probably is not intended for.

The solution that I am looking for is for the below scenario - one of my systems is trading off 10 minute charts of ES and uses iqfeed as source. Amibroker together with iqfeed is running all the time on the machine. When the system scans for a signal at the end of 10 minute interval (this is triggered by system clock), how can i make sure the last 10 minute bar is complete? (i.e. how do I make sure iqfeed delivered all the relevant data to amibroker-including data for foreign symbols referenced in afl- for the last 10 minute bar? I had introduced a 5 seconds delay already hoping that that would solve the problem. Also I assumed wait for backfill will make sure last 10 minute bar is complete-may be wait for backfill is not relevant here?)

#9

Tomasz, I kindly ask you to give it another go at my problem. i dont think anyone else can help with this:) I will try to explain the issue once more - so you can forget whats written above.

Under what circumstances can a signal (buy/sell/short/cover) for the immediate previous bar appear/disappear a few seconds/minute(?) later on analysis window while using iqfeed as real time data provider?
I am using 5min as the base interval for database. and afl's are run through analysis window by a script that calls the saved .apx file at periodic intervals of 5 minutes ( starting 5-6 seconds later after system clock completes a 5 minute - I just gave it few seconds extra so that i can assume all the symbols referenced by setforeign calls have formed the last candle)

I can safely say i have ruled out any programming error for future references. I am using few setforeign calls in the afl.. insruments that are referenced by set foreign are /VXX/VIX.XO/VXN.XO./various sector indices in order to make a decision on trading ES. Can this setforeign calls have an influence?

some questions that have been in my mind are - does it have anything to do with when Amibroker decides a bar for a particular symbol can be closed? i.e does amibroker wait until a timestamp from next interval is received and then write to database? does analysis window work with data from memory or the database?

Thanks

#10

just to add how i see the problem - For example today i was watching my script run the scan on analysis window returning no buy/sell, but i just pressed scan again manually may be after 2 minutes to see there was a buy at the last completed interval.. of course i see it also from my logs later that such thing occurred as the script log all scan results when scan is done

#11

I'm not sure if I understood you well, but maybe the problem is, that you always check only the previous (not the last bar) when running your scans. If you run the scan just a few seconds after each 5 min bar is theoretically formed/closed, this bar might still be the last bar. It will become "previous" only if there is a new transaction (and a new bar). This might happen with any delay ( 1, 5, 30 seconds, 2 minutes or more).

You might consider modifying your formula to check (using Now() and Status("...") or DateTime() or TimeNum() or Minute() etc.) if the period is completed and if that is the case - but there is no new/next bar, treat this last bar analogically to previous in other cases).

Regards

#12

also, am doing pre-scan of all symbols in database before market start, all symbols are in real-time window - if that helps. i dont have many symbols - about 20, with just 10000 bars.

Milosz, thanks for the reply.
I probably read in a different thread what you were trying to achieve.. to take amibroker backtested code live without porting to another application - precisely what am trying to achieve too.

I am not worried about the ES contract itself since few seconds delay should make sure that immediate previous bar is complete. but not sure about those sectoral indices (i also use tick index JTNT.Z ) referred by set foreign calls within AFL. Also it makes me think now that iqfeed delays if any could be resolved if i find a solution to this.

just to make sure we are on same page - lets say system time is 15:00:06. this is when my script gets executed. am calling the bar that has timestamp 14:59:59 as 'immediate previous bar'. 'current bar' is the one that is being formed and will have a timestamp of 15:04:59 when complete. ( I am only interested in the signals on 'immediate previous bar'. - no delays, buy on close setting) I guess what you mean to say is that at 15:00:06 , 'current bar' might not exist for all symbols that are referenced by AFl if there was no data coming from iqfeed for the 'current bar'?

Do you think if I check say timenum() of 'curent bar' is in the future(15:04:09 in the example) for all symbols referenced in AFL, and if not somehow wait until that happens? ( am okay with waiting - but i want consistent signals that dont disappear/appear in future)

Thanks

#13

Exactly. At 15:00:06 some of your symbols' last bar time stamps might be 14:59:59 or 14:54:59 or 13:24:59 etc (if you have selected END Time of interval in Tools->Preferences->Intraday) http://www.amibroker.com/kb/2015/01/07/timestamps-explained/

Yes. You can simply run your Scan more often than every 5 minutes (i.e. every 1 minute or 30 seconds), but generate buy/sell signals only when those additional conditions are met.

You might find some additional info in these threads:

#14

thanks, yes i work with END time of interval.
I guess I will use something like Lastvalue(timenum()) is in the future as the condition to be met by all symbols referenced by afl through setforeign symbol.

Excited to check if that solves my problems!

#15

quick disappointment - i was logging value of last(timenum()) (amoung others) through the afl already using the code above from the thread,every time scan was activated. and from today's log i see that this value is already in the future for the symbols used in setforeign calls, i.e as expected. so back to square 1..any other hints?

#16

another possibility i can think of is corrections from data feed resulting in change of signals of 'immediate previous bar'? How often does correction come?

I see from my experience say 95% of the time, signals don't change.

#17

here is an example of the log from 26/01/2018 . I can try to explain what I find problematic for me

VXX currentsystemtime lastvaluetimenum statuslastbarend statuslastbarvalue statusprevbarvalue statuslastrtupdate GetRTDataupdatetime GetRTDataupdatevalu
21:50:11 155959 26/01/2018 15:59:59 27.5750 27.5750 26/01/2018 15:50:21 155020 2871.2500
VXX currentsystemtime lastvaluetimenum statuslastbarend statuslastbarvalue statusprevbarvalue statuslastrtupdate GetRTDataupdatetime GetRTDataupdatevalu
21:59:11 155959 26/01/2018 15:59:59 27.6400 27.5600 26/01/2018 15:59:21 155921 2871.7500

Please refer to the bold italic entries. Essentially its snapshot of 2 entires in log at 21:50:11 CET and 21:59:11 CET for symbol VXX. The bold entries are a result of logging Lastvalue(Ref(C,-1)). (please note setting are different in this example as time interval of scan is 10 minutes and not 5. The second entry is a special case to log market stat just before end of market hours, otherwise it will scan at close of every 10 minutes + 11 seconds in this case. You could just as well assume second log entry was created by manually pressing scan at 21:59:11. Also please dont be confused by the time difference as logging timestamp is system time and other times are from exchange times i.e timenum() etc). My expectation is that snapshot of Ref(C,-1) should be same at 21:50:11 and 21:59:11. ( The second column show lastvalue(timenum()) which correctly says 155959 (exchange time) for both entries) . so in spite of 'current bar' in place at the time of two snapshots, i am getting two different values for ref(C,-1). When I look at chart later, indeed the value shown as close is 27.56. So I am not sure why the value 27.575 shows up - which is not a closing value on any bars in the near vicinity looking at the 10 minute chart

Also this logging happens between setforeign() and restorerpicearrays() calls, if that helps to know

I hope this clarifies my problem..

#18

code used for logging is below

fh = fopen( "C:\\C1\\vxx.csv", "a"); 
if( fh ) 
{ 
       fputs( Name(), fh );
       fputs( " currentsystemtime ", fh );
       fputs( " lastvaluetimenum ", fh );
       fputs( " statuslastbarend      ", fh );
       fputs( " statuslastbarvalue      ", fh );
       fputs( " statusprevbarvalue      ", fh );
       fputs( " statuslastrtupdate      ", fh );
       fputs( "GetRTDataupdatetime      ", fh );        
       fputs( "GetRTDataupdatevalu \n", fh ); 
       fputs( Now(2), fh );
       fputs( "                ", fh );
       qs = StrFormat(" %.0f", 
                     LastValue(TimeNum()) ); 
       fputs( qs, fh );//timenum
       fputs( "            ", fh );
       
       qs = StrFormat(" %.0f", 
                 Status("lastbarend") ); 
       fputs( DateTimeToStr(Status("lastbarend")), fh );//statuslastbarend
       fputs( "     ", fh );
       
       qs = StrFormat(" %.4f", 
                     LastValue(C) ); 
       fputs( qs, fh );//timenum
       fputs( "            ", fh );       
       
       qs = StrFormat(" %.4f", 
                     LastValue(Ref(C,-1)) ); 
       fputs( qs, fh );//timenum
       fputs( "                ", fh ); 
       
              
       qs = StrFormat(" %.0f", 
                 Status("lastrtupdate") ); 
       fputs( DateTimeToStr(Status("lastrtupdate")), fh );//lastrtupdate
       fputs( "       ", fh );       
       
       
      qs = StrFormat(" %.0f", 
                     GetRTData("UpdateTime") ); 
      fputs( qs, fh ); //GetRTData
      fputs( "                  ", fh );       
       
       qs = StrFormat(" %.4f\n", 
                     GetRTData("Last") ); // lastprice
      fputs( qs, fh ); //GetRTData
      
   fclose( fh );
}
#19

Also it is interesting to see lastvalue(C) and lastvalue(Ref(C,-1)) are showing same value of 27.575 on the first log entry

i hope all these detailed information can help to pinpoint the root cause