Tick data download code for IQFeed

hi, I post some code for IQFeed users to be able to download tick data files (saved in files per day). The code uses a CSHARP plugin which is posted here:

https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/docs/USING-WITH-PYTHON.md

Then I communicate with this plugin in Python via Pythonnet. So you also have to do a
pip install pythonnet, (see also link above). I do not launch IQFeed from Python since I run it from Amibroker when IQFeed is already launched. So you need to have IQFeed running to do a download.

The CSHARP plugin you download from the link above and I put it in this directory:
C:\Program Files\AmiBroker\IQFeedCSharpApiClient

So this directory is used in the Python code. The data directory can be chosen from the AFL code.

You can download 180 days of tickdata. The progress can be followed in the log/Trace window.

I have 1 question. Maybe there are some people out there who know CSHARP. I should be able to turn off the !ENDMSG! that is written at the end of 1 days of tick data. In the source code (see github link) you can find that one can set endMsg False or True. I did not manage that yet. Because the !ENDMSG! is a pain. I get rid of it using Python (see my Python code) but that slows things down. When I just leave it in the tick data file and remove it for instance in Pandas that slows the processing down dramatically. So if someone can figure out how to set the endMsg to False from Python then that would be great.

AFL code:

_SECTION_BEGIN( "Download IQFeed Tickdata" );

pythonpath = "C:\\Users\\win 10\\AppData\\Local\\Programs\\Python\\Python38\\mypython\\iqfeed\\";
pythonfile = "iqfeedcsharp_download_AFL.py";
PyLoadFromFile( "ver1", pythonpath + pythonfile );

tempStorage = "C:\\Program Files\\AmiBroker\\"; // it initially puts the tick data here but moves this file to dataPath
dataPath = "D:\\AmibrokerData\\"; // the path where the tick data will be stored

startDate = ParamDate( "Start Date", "11/4/2022", 2 );
endDate = ParamDate( "End Date", "11/4/2022", 2 );
downloadtickdata = ParamTrigger( "Download Tick Data", "Press Here" );
removeEndMsg = 1; //if 1 then remove !ENDMSG! marker at end of tick data file

ndays = 1 + DateTimeDiff( endDate, startDate ) / inDaily;

// set your symbol list here
symbollist = "@ES#";
//symbollist = "@MES#,@MNQ#,QHG#,QMGC#,@ES#,@NQ#,QCL#,QGC#,QNG#,QPL#,QSI#,QSIL#,@YM#,@RTY#,QMCL#,XG#,QPA#,QMHG#";
//symbollist = "QMHG#,QMCL#";

if( downloadtickdata )
{
    Say( "download tick data" );

    for( s = 0; ( sym = StrExtract( symbollist, s ) ) != ""; s++ )
    {
        progressPerSymbol = 0;

        for( i = 0; i < ndays; i++ )
        {
            n1 = DateTimeAdd( startDate, i, inDaily );
            dow = DateTimeFormat( "%w", n1 ); // day of week
            dd = DateTimeFormat( "%d", n1 ); // day of month
            mm = DateTimeFormat( "%m", n1 ); // month
            yyyy = DateTimeFormat( "%Y", n1 ); // year

            fdate = yyyy + mm + dd;
            filename = sym + "_" + fdate + ".csv";

            // download tick data except for saturdays
            if( StrToNum( dow ) != 6 )
            {
                PyEvalFunction( "ver1", "getHistoryBetweenTwoDates",
                                sym,
                                StrToNum( yyyy ),
                                StrToNum( mm ),
                                StrToNum( dd ),
                                0, 0, 0,
                                StrToNum( yyyy ),
                                StrToNum( mm ),
                                StrToNum( dd ),
                                23, 59, 59,
                                filename,
                                dataPath,
                                tempStorage,
                                removeEndMsg );

                progressPerSymbol = Prec( ( i + 1 ) / ndays * 100, 2 );
                _TRACE( "downloading: " + filename + ",  progress: " + progressPerSymbol + " %" );
            }
        }
    }

    Say( "finished" );
}

SetChartOptions( 1, chartShowDates, chartGridMiddle, 0, 0, 0 );
Plot( C, "Data", ColorRGB( 3, 157, 252 ), styleDots, Null, Null, 0, 0, 1 );
_SECTION_END();

Python Code:

'''
filename: iqfeedcsharp_download_AFL.py

code uses CSHARP plugin. Read instructions here how to use it:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/docs/USING-WITH-PYTHON.md
'''

if '__' + __file__ + '_initialized' not in globals():
    globals()['__' + __file__ + '_initialized'] = True
    import sys
    import clr
    assembly_path = r'C:/Program Files/AmiBroker/IQFeedCSharpApiClient'
    sys.path.append(assembly_path)
    clr.AddReference("IQFeed.CSharpApiClient")
    clr.AddReference('System.Collections')
    from System import DateTime
    import time
    from IQFeed.CSharpApiClient.Lookup import LookupClientFactory
    import AmiPy
    import os
    import os.path
    import shutil
    import csv
    import datetime as dt
    from datetime import datetime

def getHistoryBetweenTwoDates(sym,
                            startYear,
                            startMonth,
                            startDay,
                            startHour,
                            startMinute,
                            startSecond,
                            endYear,
                            endMonth,
                            endDay,
                            endHour,
                            endMinute,
                            endSecond,
                            filename,
                            dataPath,
                            tempStorage,
                            removeEndMsg):

    sym = str(sym)
    startYear = int(startYear)
    startMonth = int(startMonth)
    startDay = int(startDay)
    startHour = int(startHour)
    startMinute = int(startMinute)
    startSecond = int(startSecond)
    endYear = int(endYear)
    endMonth = int(endMonth)
    endDay = int(endDay)
    endHour = int(endHour)
    endMinute = int(endMinute)
    endSecond = int(endSecond)
    filename = str(filename)
    dataPath = str(dataPath)
    tempStorage = str(tempStorage)
    removeEndMsg = int(removeEndMsg)

    # Create Lookup client
    lookupClient = LookupClientFactory.CreateNew()
    # Connect
    lookupClient.Connect()
    # Save ticks to disk
    d1 = DateTime(startYear,startMonth,startDay,startHour,startMinute,startSecond)
    d2 = DateTime(endYear,endMonth,endDay,endHour,endMinute,endSecond)

    try:
        ticksFilename = lookupClient.Historical.File.GetHistoryTickTimeframe(sym,d1,d2,None,None,None,1)
        # Move tmp filename
        dstTicksFilename = tempStorage + "ticks.csv"
        src = dstTicksFilename
        dest = dataPath + sym + '\\'
        if not os.path.exists(dest):
            os.makedirs(dest)
        dest = dest + filename
        os.replace(ticksFilename, dstTicksFilename)
        shutil.move(src,dest)

        # remove last line that contains !ENDMSG!
        # difficult way to do it since I have no control over file created with csharp
        if removeEndMsg == 1:
            with open(dest, 'r+') as fp:
                # read an store all lines into list
                lines = fp.readlines()
                # move file pointer to the beginning of a file
                fp.seek(0)
                # truncate the file
                fp.truncate()
                # start writing lines except the last line
                # lines[:-1] from line 0 to the second last line
                fp.writelines(lines[:-1])
    except:
        AmiPy.Print('No data found from: ' + str(d1) + ' to: ' + str(d2) + ' ....')

1 Like

removing the !ENDMSG! marker I now did a bit more precise actually testing for the string. See code below, you will see where to replace it in the Python file.

Still I am hoping for people who know CSharp and who can have a look in the source code. We should be able to turn this !ENDMSG! marker off using endMsg = false. Question is how ..

if you add this code you also need to import re at the top

.....
.....
from datetime import datetime
import re

        # remove last line that contains !ENDMSG!
        # difficult way to do it since I have no control over file created with csharp
        if removeEndMsg == 1:
            with open(dest, 'r+') as file:
                lines = file.readlines()
                last_line = lines[-1]
                if re.search("!ENDMSG!,", last_line):
                    # move file pointer to the beginning of a file
                    file.seek(0)
                    # truncate the file
                    file.truncate()
                    # start writing lines except the last line
                    # lines[:-1] from line 0 to the second last line
                    file.writelines(lines[:-1])

I'm not in touch with coding for a couple of decades but have been refreshing in the last few years.

IMO, reading a large file in memory is not a good approach. .readlines() has to put all file data in memory plus parse the file into a Python LIST which is more overhead.
After all the work, has to write everything back too!

There are two examples here, that open the file and work from the END and look for whatever character(s) is needed.
Reading is done in binary so encode your search character in b too. Its there in the link.

Then you just Truncate from that position.
It will give your code quite a boost i think.

with open(file,'r+b', buffering=0) as f:
    f.seek(0, os.SEEK_END)
    end = f.tell()
    while f.tell() > 0:
        f.seek(-1, os.SEEK_CUR)
        ## Go to end of file and read
        char = f.read(1)

        ## Do matching and searching here
        ## If condition found
            f.truncate()

        f.seek(-1, os.SEEK_CUR)

## rest of the cleanup

source and credits:

1 Like

thank you, i will have a look.

But there should also be a direct way to instruct the Csharp plugin I use. Maybe I will just turn back to plain only Python code and skip the plugin. I was using plain Python code earlier, to download the tick data code. An example how to use just Python code is for instance here:
http://forums.iqfeed.net/index.cfm?page=topic&topicID=5934

But I do not have an IQFeed developer account so then you can not access the manual and get all the message rules. These are all inside the Csharp plugin and the Csharp plugin also has methods to get streaming data. Basically the Csharp plugin gives you access to all data IQfeed has to offer.

Unfortunately the CSharp plugin programmer will not answer any mails.

1 Like

We have local Exchange Data vendors so i am not familiar with IQFeed but it was pretty obvious for code to slow down if you are loading entire data to list and writing back so i thought you could optimize there.

To understand the c# code, one requires knowledge of its workings too, not just the language. (As you said, manual access)

yes well once all data is downloaded I only have to make daily updates now so I only update a handful of symbols so it is not a big deal. Just was hoping someone could do that. Since it is not just C#. It is communicating with the C# plugin via Python (using pythonnet)

In fact probably we could do the same with the Amibroker IQFeed plugin that is probably written in C++. I think there is also a python bridge for that, we just do not have access to the source code.

Here you see the C# code where the ENDMSG is handled: https://github.com/mathpaquette/IQFeed.CSharpApiClient/search?q=endMsg

I will email mathpaquette or ask on the IQFeed forum

@empottasch, I briefly looked at the C# source code.

As far as I can see endMsg is simply a local variable, used inside functions, to signal when the stream is complete based on the detection of "!ENDMSG!" in combination with another terminator (these are defined as constants).
(Many of the entries you see when searching for it are from the "test" code, which won't be part of the compiled plugin).

Anyway, I don't see how you could set or change it from your code.

BTW, when encountered, the code that is in a loop that reads "messages" does a break, so it seems necessary and can't be removed from the source code.

I don't use IQfeed, so I can't test/debug the code and it's not clear to me when the "!ENDMSG!" is passed (included in the last 'message') to the client (or if sometimes it is eventually removed).

Perhaps you could ask the developer that "!ENDMSG!" is never actually included in the resulting stream (by removing it from the data stream, parsing the last message, when it is received).

In any case, take these comments with some skepticism, as I may have partially or incorrectly understood the source code.

hi beppe, thanks for your reply. Maybe it can not be set on and off. Before I was using code I extracted from here:
https://www.quantstart.com/articles/Downloading-Historical-Intraday-US-Equities-From-DTN-IQFeed-with-Python/

inside the Python code itself I could easily remove it when the buffer was finished using

    # Remove the end message string
    buffer = buffer[:-12]
    return buffer

see code in article. No big deal, maybe I turn back to the code in the article again.

@empottasch, FWIW, I think that the plugin author could do something similar to the code used in your last Python snippet example.

In particular, in the file: src/IQFeed.CSharpApiClient/Lookup/Common/LookupMessageFileHandler.cs (from line 64)

       binaryWriter.Write(args.Message, 0, args.Count);

       // check if the message end
       if (args.Message.EndsWith(args.Count, _endOfMsgBytes))
              res.TrySetResult(filename);

I imagine he could alter this code to check and remove the _endOfMsgBytes sequence before the write operation and set a flag to trigger when needed the following conditional.

The _endOfMsgBytes is defined in the same file as:

_endOfMsgBytes = Encoding.ASCII.GetBytes(IQFeedDefault.ProtocolEndOfMessageCharacters + 
                                 IQFeedDefault.ProtocolDelimiterCharacter + 
                                 IQFeedDefault.ProtocolTerminatingCharacters);

These are constants defined in src/IQFeed.CSharpApiClient/IQFeedDefault.cs:

        public const string ProtocolTerminatingCharacters = "\r\n";
        public const string ProtocolEndOfMessageCharacters = "!ENDMSG!";
        public const string ProtocolNoDataCharacters = "!NO_DATA!";
        public const string ProtocolSyntaxErrorCharacters = "!SYNTAX_ERROR!";
        
        public const char ProtocolLineFeedCharacter ='\n';
        public const char ProtocolDelimiterCharacter = ',';
3 Likes

thank you. Will have a look. Possibly I could also try that out myself and then do the compilation. I have sent him an email. If he responds I will suggest/request this.