hi, I post some code for IQFeed users to be able to download tick data files (saved in files per day). The code uses a CSHARP plugin which is posted here:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/docs/USING-WITH-PYTHON.md
Then I communicate with this plugin in Python via Pythonnet. So you also have to do a
pip install pythonnet, (see also link above). I do not launch IQFeed from Python since I run it from Amibroker when IQFeed is already launched. So you need to have IQFeed running to do a download.
The CSHARP plugin you download from the link above and I put it in this directory:
C:\Program Files\AmiBroker\IQFeedCSharpApiClient
So this directory is used in the Python code. The data directory can be chosen from the AFL code.
You can download 180 days of tickdata. The progress can be followed in the log/Trace window.
I have 1 question. Maybe there are some people out there who know CSHARP. I should be able to turn off the !ENDMSG! that is written at the end of 1 days of tick data. In the source code (see github link) you can find that one can set endMsg False or True. I did not manage that yet. Because the !ENDMSG! is a pain. I get rid of it using Python (see my Python code) but that slows things down. When I just leave it in the tick data file and remove it for instance in Pandas that slows the processing down dramatically. So if someone can figure out how to set the endMsg to False from Python then that would be great.
AFL code:
_SECTION_BEGIN( "Download IQFeed Tickdata" );
pythonpath = "C:\\Users\\win 10\\AppData\\Local\\Programs\\Python\\Python38\\mypython\\iqfeed\\";
pythonfile = "iqfeedcsharp_download_AFL.py";
PyLoadFromFile( "ver1", pythonpath + pythonfile );
tempStorage = "C:\\Program Files\\AmiBroker\\"; // it initially puts the tick data here but moves this file to dataPath
dataPath = "D:\\AmibrokerData\\"; // the path where the tick data will be stored
startDate = ParamDate( "Start Date", "11/4/2022", 2 );
endDate = ParamDate( "End Date", "11/4/2022", 2 );
downloadtickdata = ParamTrigger( "Download Tick Data", "Press Here" );
removeEndMsg = 1; //if 1 then remove !ENDMSG! marker at end of tick data file
ndays = 1 + DateTimeDiff( endDate, startDate ) / inDaily;
// set your symbol list here
symbollist = "@ES#";
//symbollist = "@MES#,@MNQ#,QHG#,QMGC#,@ES#,@NQ#,QCL#,QGC#,QNG#,QPL#,QSI#,QSIL#,@YM#,@RTY#,QMCL#,XG#,QPA#,QMHG#";
//symbollist = "QMHG#,QMCL#";
if( downloadtickdata )
{
Say( "download tick data" );
for( s = 0; ( sym = StrExtract( symbollist, s ) ) != ""; s++ )
{
progressPerSymbol = 0;
for( i = 0; i < ndays; i++ )
{
n1 = DateTimeAdd( startDate, i, inDaily );
dow = DateTimeFormat( "%w", n1 ); // day of week
dd = DateTimeFormat( "%d", n1 ); // day of month
mm = DateTimeFormat( "%m", n1 ); // month
yyyy = DateTimeFormat( "%Y", n1 ); // year
fdate = yyyy + mm + dd;
filename = sym + "_" + fdate + ".csv";
// download tick data except for saturdays
if( StrToNum( dow ) != 6 )
{
PyEvalFunction( "ver1", "getHistoryBetweenTwoDates",
sym,
StrToNum( yyyy ),
StrToNum( mm ),
StrToNum( dd ),
0, 0, 0,
StrToNum( yyyy ),
StrToNum( mm ),
StrToNum( dd ),
23, 59, 59,
filename,
dataPath,
tempStorage,
removeEndMsg );
progressPerSymbol = Prec( ( i + 1 ) / ndays * 100, 2 );
_TRACE( "downloading: " + filename + ", progress: " + progressPerSymbol + " %" );
}
}
}
Say( "finished" );
}
SetChartOptions( 1, chartShowDates, chartGridMiddle, 0, 0, 0 );
Plot( C, "Data", ColorRGB( 3, 157, 252 ), styleDots, Null, Null, 0, 0, 1 );
_SECTION_END();
Python Code:
'''
filename: iqfeedcsharp_download_AFL.py
code uses CSHARP plugin. Read instructions here how to use it:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/docs/USING-WITH-PYTHON.md
'''
if '__' + __file__ + '_initialized' not in globals():
globals()['__' + __file__ + '_initialized'] = True
import sys
import clr
assembly_path = r'C:/Program Files/AmiBroker/IQFeedCSharpApiClient'
sys.path.append(assembly_path)
clr.AddReference("IQFeed.CSharpApiClient")
clr.AddReference('System.Collections')
from System import DateTime
import time
from IQFeed.CSharpApiClient.Lookup import LookupClientFactory
import AmiPy
import os
import os.path
import shutil
import csv
import datetime as dt
from datetime import datetime
def getHistoryBetweenTwoDates(sym,
startYear,
startMonth,
startDay,
startHour,
startMinute,
startSecond,
endYear,
endMonth,
endDay,
endHour,
endMinute,
endSecond,
filename,
dataPath,
tempStorage,
removeEndMsg):
sym = str(sym)
startYear = int(startYear)
startMonth = int(startMonth)
startDay = int(startDay)
startHour = int(startHour)
startMinute = int(startMinute)
startSecond = int(startSecond)
endYear = int(endYear)
endMonth = int(endMonth)
endDay = int(endDay)
endHour = int(endHour)
endMinute = int(endMinute)
endSecond = int(endSecond)
filename = str(filename)
dataPath = str(dataPath)
tempStorage = str(tempStorage)
removeEndMsg = int(removeEndMsg)
# Create Lookup client
lookupClient = LookupClientFactory.CreateNew()
# Connect
lookupClient.Connect()
# Save ticks to disk
d1 = DateTime(startYear,startMonth,startDay,startHour,startMinute,startSecond)
d2 = DateTime(endYear,endMonth,endDay,endHour,endMinute,endSecond)
try:
ticksFilename = lookupClient.Historical.File.GetHistoryTickTimeframe(sym,d1,d2,None,None,None,1)
# Move tmp filename
dstTicksFilename = tempStorage + "ticks.csv"
src = dstTicksFilename
dest = dataPath + sym + '\\'
if not os.path.exists(dest):
os.makedirs(dest)
dest = dest + filename
os.replace(ticksFilename, dstTicksFilename)
shutil.move(src,dest)
# remove last line that contains !ENDMSG!
# difficult way to do it since I have no control over file created with csharp
if removeEndMsg == 1:
with open(dest, 'r+') as fp:
# read an store all lines into list
lines = fp.readlines()
# move file pointer to the beginning of a file
fp.seek(0)
# truncate the file
fp.truncate()
# start writing lines except the last line
# lines[:-1] from line 0 to the second last line
fp.writelines(lines[:-1])
except:
AmiPy.Print('No data found from: ' + str(d1) + ' to: ' + str(d2) + ' ....')