Python script: Download multiple historical IB TWS CME Futures contracts

Data Sources
1

I wrote this python script to download historical futures data from TWS since the built in IB data plugin does not seem to download all historical contracts.

Simply specify multiple contracts that you'd like to download, then it will output csv files for each contract in the folder you specify.

Code is based on: https://groups.io/g/insync/message/256

python3 script:

from ib_insync import *
import pandas as pd
import datetime

'''
things you will need: python3, TWS logged in and running.
pip packages: pip install wsgiref boto

About: This script downloads multiple historical futures contracts.

Usage: Fill out the information in the two variables below. You can get this information from TWS.
       TWS>Enter Symbol>Click Futures>Click "More & Multiple">Click "Show Historical Contracts"

Contracts structure: ['Symbol','TWS Contract Name','Contract Month','Last Trade Date','First Trade Date']
                     We are building an array of array's. So be sure to separate each contract within brackets with a comma, see example below.

'''

downloadFolder = 'C:\\Users\\YOURNAME\\Documents\\data\\' 
contracts = [['MES','MESH0','202006','2020-03-20','2019-12-14'],['MES','MESZ9','201912','2019-12-20','2019-09-14']]

##### Don't edit below unless you know what you're doing ;^)  ######

ib = IB()
ib.connect('127.0.0.1', 7496, clientId=1)

def getData(contracts):
    for contract in contracts:
        symbol = contract[0]
        fileName = contract[1]
        contractMonth = contract[2]
        end = datetime.datetime.strptime(contract[3], '%Y-%m-%d')
        start = datetime.datetime.strptime(contract[4], '%Y-%m-%d')

        print('Contract: '+ fileName + ' ' + contractMonth)
        print('Downloading...')

        barsList = []

        contract = Contract(secType='FUT',symbol=symbol, lastTradeDateOrContractMonth=contractMonth, exchange='GLOBEX', currency='USD', includeExpired=True)

        dt = end
        while dt > start:
            bars = ib.reqHistoricalData(contract, endDateTime=dt, durationStr='10 D', barSizeSetting='1 min', whatToShow='TRADES', useRTH=False) #,timeout=0
            barsList.append(bars)
            print(fileName + ' ' + dt.strftime('%m/%d/%Y') + ' Done.')
            dt = bars[0].date

        allBars = [b for bars in reversed(barsList) for b in bars]
        df = util.df(allBars)
        gfg_csv_data = df.to_csv(downloadFolder+fileName+'.csv', index = True) 

        print('Done.')
    print('All contracts downloaded :^)')

getData(contracts)
7 Likes
2

Thank you for this. I was trying to write something like this, while learning Python at the same time, which has been a challenge.

1 Like
3

Glad you found it useful @Ge55S!

There was an error in my example code above, but I can't seem to edit my post for some reason. I had the wrong contract month for MESH0. The contracts line should read:

contracts = [['MES','MESH0','202003','2020-03-20','2019-12-14'],['MES','MESZ9','201912','2019-12-20','2019-09-14']]
4

No problem. I had to change contract definition any way, as I do not need micros or even any other index futures. :smiley:

5

@Tomasz, is there any way I can update my first post? I've got more bug fixes that I'd like to provide. Maybe I should just put my code on github. :thinking:

6

You can edit post within 15 minutes. Later edits can't be done because this feature was abused in the past (people were totally changing the content of emails making all responses nonsensical).
You can post updates as new post in the same thread.

1 Like
7

First: Plugin downloads all historical contracts

Second: I would advise against using external scripts like this because importing to plugin driven database either:

  • does NOT work if data are controlled by the plugin (default case)
  • creates tickers that are excluded from RT update (if symbol wasn't present)

MUST READ:

1 Like
8

This script is designed for pulling historical contract data, not available through the standard IB plugin.

It does in fact create tickers that are excluded from RT because there is not real time update for historical contracts.

That being said, if you are afraid of it messing up your RT database, create an offline database for use in backtesting and import the symbols this script creates as you would normally through the wizard. (I have not had issues thus far.)

9

No, I am not afraid of corrupting database. I am afraid that people blindly copy-paste the code, run it and then complain that their symbols "don't update" after import. I have seen such "bug reports" thousands of times and I am tired with explaining for 20 years each and everyone things that are clearly stated in the manual:

http://www.amibroker.com/guide/h_rtsource.html

In section called:

Things you should NOT do, or you should do very carefully

but people NEVER read the manual.

Copying random codes from Internet (even if it is this forum) and running them without thinking / understanding what they are doing is recipe for trouble. Unfortunately it seems that 99% of people do just that.

5 Likes
10

Well that makes sense why you would discourage the use. Sounds like you should spin up some NLP in the support department to filter out the dummies :smile:

Regardless, this script will get you some data you couldn't get otherwise. Note to the copy paster's: don't.

11

Updated code, (now with disclaimer), @Tomasz feel free to update my first post with this post, since I can't do that.

DO NOT CONTACT TOMASZ IF YOU ARE HAVING ISSUES WITH THIS SCRIPT AND YOUR DATABASE. IMPORT RESULTING SYMBOLS TO A NEW OFFLINE (NON-REAL TIME) DATABASE ONLY.

Things I've changed: All you need to do is change the output folder, and enter the symbol you'd like to download.
The Python3 script will now download all of the available historical symbols (at the time of writing 2/1/2021).

from ib_insync import *
import pandas as pd
import datetime

'''
#################################################################################################
THIS WILL NOT WORK WITH REAL TIME TRADED SYMBOLS. IMPORT THESE SYMBOLS TO A NEW OFFLINE DATABASE.
IF YOU HAVE ISSUES, REPLY TO THIS FORUM THREAD AND I WILL HELP. DO NOT BUG TOMASZ.
#################################################################################################

things you will need: python3, TWS logged in and running.
pip packages you will need: ib_insync, pandas

About: This script downloads multiple historical futures contracts.

Usage: Fill out the information in the two variables below. You can get this information from TWS.
       TWS>Enter Symbol>Click Futures>Click "More & Multiple">Click "Show Historical Contracts"

Contracts structure: ['Symbol','TWS Contract Name','Contract Month','Last Trade Date','First Trade Date']
                     We are building an array of array's. So be sure to separate each contract within brackets with a comma, see example below.

'''

# CHANGE THESE VALUES FOR YOURSELF.
downloadFolder = 'C:\\Users\\USER-NAME\Documents\\data\\' 
contract = 'MNQ' # CME FUTURES ONLY

##### Don't edit below unless you know what you're doing ;^)  ######

contracts = [[contract,contract+'Z0','202012','2020-12-20','2020-09-14'],
             [contract,contract+'U0','202009','2020-09-20','2020-06-14'],
             [contract,contract+'M0','202006','2020-06-20','2020-03-14'],
             [contract,contract+'H0','202003','2020-03-20','2019-12-14'],
             [contract,contract+'Z9','201912','2019-12-20','2019-09-14'],
             [contract,contract+'U9','201909','2019-09-20','2019-06-14'],
             [contract,contract+'M9','201906','2019-06-20','2019-05-10']]

ib = IB()
ib.connect('127.0.0.1', 7496, clientId=1)

def getData(contracts):
    for contract in contracts:
        symbol = contract[0]
        fileName = contract[1]
        contractMonth = contract[2]
        end = datetime.datetime.strptime(contract[3], '%Y-%m-%d')
        start = datetime.datetime.strptime(contract[4], '%Y-%m-%d')

        print('Contract: '+ fileName + ' ' + contractMonth)
        print('Downloading...')

        barsList = []

        contract = Contract(secType='FUT',symbol=symbol, lastTradeDateOrContractMonth=contractMonth, exchange='GLOBEX', currency='USD', includeExpired=True)

        dt = end
        running = True
        while dt > start and running:
            bars = ib.reqHistoricalData(contract, endDateTime=dt, durationStr='10 D', barSizeSetting='1 min', whatToShow='TRADES', useRTH=False) #,timeout=0
            barsList.append(bars)
            try:
                dt = bars[0].date
                print(fileName + ' ' + dt.strftime('%m/%d/%Y') + ' Done.')
            except:
                print('No more data for '+fileName+'. Try adjusting First Trade Date.')
                running = False
        if running:
            allBars = [b for bars in reversed(barsList) for b in bars]
            df = util.df(allBars)
            gfg_csv_data = df.to_csv(downloadFolder+fileName+'.csv', index = True) 

        print('Done.')
    print('All contracts downloaded. :^)')

getData(contracts)
12

I already did. In the beginning of 2020.

3 Likes
13

Happy to hear some load will be taken off your back! :grin:

14

Here is a new script that downloads 1 Second data as far back as you need. I've set it up here to download 3 days worth of ES 1 second data. This will take a long time.

from ib_insync import *
import pandas as pd
import datetime, time

'''
#################################################################################################
THIS WILL NOT WORK WITH REAL TIME TRADED SYMBOLS. IMPORT THESE SYMBOLS TO A NEW OFFLINE DATABASE.
IF YOU HAVE ISSUES, REPLY TO THIS FORUM THREAD AND I WILL HELP. DO NOT BUG TOMASZ.
#################################################################################################

things you will need: python3, TWS logged in and running.
pip packages you will need: ib_insync, pandas

About: This script downloads 1 second data as far back as you need.

Usage: Fill out the information in the two variables below. You can get this information from TWS.
       TWS>Enter Symbol>Click Futures>Click "More & Multiple">Click "Show Historical Contracts"

Contracts structure: ['Symbol','TWS Contract Name','Contract Month','Last Trade Date','First Trade Date']
                     We are building an array of array's. So be sure to separate each contract within brackets with a comma, see example below.

'''

# CHANGE THESE VALUES FOR YOURSELF.
downloadFolder = 'C:\\Users\\USER NAME\\Documents\\data\\' 
contract = 'ES' # CME FUTURES ONLY
daysBack = 3 # Number of days to download. Sart day/ time is the moment you run this script.
contracts = [[contract,contract+'H1','202103',datetime.datetime.today(),(datetime.datetime.today() - datetime.timedelta(days=daysBack))]] # UPDATE THIS FOR DIFFERENT CONTRACTS

##### Don't edit below unless you know what you're doing ;^)  ######
ib = IB()
ib.connect('127.0.0.1', 7496, clientId=1)

def getData(contracts):
    for contract in contracts:
        symbol = contract[0]
        fileName = contract[1]
        contractMonth = contract[2]
        end = contract[3]
        start = contract[4]

        print('Contract: '+ fileName + ' ' + contractMonth)
        print('Downloading...')

        barsList = []

        contract = Contract(secType='FUT',symbol=symbol, lastTradeDateOrContractMonth=contractMonth, exchange='GLOBEX', currency='USD', includeExpired=True)

        dt = end
        running = True
        while dt > start and running:
            bars = ib.reqHistoricalData(contract, endDateTime=dt, durationStr='1800 S', barSizeSetting='1 secs', whatToShow='TRADES', useRTH=False) #,timeout=0
            barsList.append(bars)
            try:
                dt = bars[0].date
                print(fileName + ' ' + dt.strftime('%m/%d/%Y %H:%M:%S') + ' Done.')
                time.sleep(15)
            except:
                print('No more data for '+fileName+'. Try adjusting First Trade Date.')
                running = False
        if running:
            allBars = [b for bars in reversed(barsList) for b in bars]
            df = util.df(allBars)
            gfg_csv_data = df.to_csv(downloadFolder+fileName+'.csv', index = True) 
            

        print('Done.')
    print('All contracts downloaded. :^)')

getData(contracts)
3 Likes
15

Update to the above 1-sec data download python script.

Updates in this version:
This version of the script now has a dialog before you run it. There are two modes. One for download X days from now
The second mode downloads X days from specified date.

The script also now saves a CSV every 5 calls to the TWS API. This is basically a fail-safe. The last file will have the most data.
All other files can be deleted once the script is finished.

The Script:

from ib_insync import *
import pandas as pd
import datetime, time

'''
#################################################################################################
THIS WILL NOT WORK WITH REAL TIME TRADED SYMBOLS. IMPORT THESE SYMBOLS TO A NEW OFFLINE DATABASE.
IF YOU HAVE ISSUES, REPLY TO THIS FORUM THREAD AND I WILL HELP. DO NOT BUG TOMASZ.
#################################################################################################

things you will need: python3, TWS logged in and running.
pip packages you will need: ib_insync, pandas

About: This script downloads 1 second data as far back as you need.

Usage: Fill out the information in the two variables below. You can get this information from TWS.
       TWS>Enter Symbol>Click Futures>Click "More & Multiple">Click "Show Historical Contracts"

This version of the script now has a dialog before you run it. There are two modes. One for download X days from now
The other downloads X days from specified date. 

The script also now saves a CSV every 5 calls to the TWS API. This is basically a fail-safe. The last file will have the most data. 
All other files can be deleted once the script is finished.

'''

##########################################################
# CHANGE THESE VALUES FOR THE CONTRACT YOU WANT 
##########################################################

downloadFolder = 'C:\\data\\mes\\1second\\' 
contract = 'MES' # CME FUTURES ONLY
contractCode = 'M1'
contractExp = '202106'

##########################################################
## Don't edit below unless you know what you're doing.
##########################################################


mode = int(input("Mode: From now (1), From Date (2) -> "))
if mode == 1:
    daysBack = int(input("Number of Days back from now to download -> ")) # Number of days to download. Sart day/ time is the moment you run this script.
    contracts = [[contract,contract+contractCode,contractExp,datetime.datetime.today(),(datetime.datetime.today() - datetime.timedelta(days=daysBack))]]
else:
    dateAndTime = str(input("Date Time EX: 03/18/2021 22:48:28 -> "))
    daysBack = int(input("Number of days back from date to download -> "))
    contracts = [[contract,contract+contractCode,contractExp,datetime.datetime.strptime(dateAndTime, '%m/%d/%Y %H:%M:%S'),(datetime.datetime.strptime(dateAndTime, '%m/%d/%Y %H:%M:%S') - datetime.timedelta(days=daysBack))]]

##### Don't edit below unless you know what you're doing ;^)  ######
ib = IB()
ib.connect('127.0.0.1', 7496, clientId=1)

def csvOut(barsList, n, fileName):
    allBars = [b for bars in reversed(barsList) for b in bars]
    df = util.df(allBars)
    gfg_csv_data = df.to_csv(downloadFolder+str(n)+'-'+fileName+'.csv', index = True)
    #df.drop(df.index,inplace=True) # clear the dataframe
    print('Exported: '+str(n)+'-'+fileName+'.csv')

def getData(contracts):
    i = 0
    n = 0
    for contract in contracts:
        symbol = contract[0]
        fileName = contract[1]
        contractMonth = contract[2]
        end = contract[3]
        start = contract[4]

        print('Contract: '+ fileName + ' ' + contractMonth)
        print('Downloading...')

        barsList = []

        contract = Contract(secType='FUT',symbol=symbol, lastTradeDateOrContractMonth=contractMonth, exchange='GLOBEX', currency='USD', includeExpired=True)

        dt = end
        running = True
        while dt > start and running:
            if dt.weekday() == 5:
                # if it's saturday, go back to friday, eod.
                dt = dt - datetime.timedelta(days=1)
                dt = dt.replace(hour=13, minute=59, second=0)
            else:
                bars = ib.reqHistoricalData(contract, endDateTime=dt, durationStr='1800 S', barSizeSetting='1 secs', whatToShow='TRADES', useRTH=False) #,timeout=0
                barsList.append(bars)
                
                dt = bars[0].date
                print(fileName + ' ' + dt.strftime('%m/%d/%Y %H:%M:%S') + ' Done.')
                if(i == 4):
                    csvOut(barsList,n,fileName) #export dataframe
                   
                    i = 0
                    n+=1
                else:
                    i+=1
                time.sleep(10.2)
        if running:
             csvOut(barsList,n,fileName)
            

        print('Done.')
    print('All contracts downloaded. :^)')

getData(contracts)
2 Likes
16

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.