FYI, backfills are sometimes very slow or don't even work on weekends and in the pre-market hours. They generally work fine during market hours and after hours, except during the server resets that are around 5pm ET and then around midnight when you should expect to be disconnected briefly.
Well, yes. When the TWS went down at midnight as it does, the backfill ended as well. When I selected "Backfill All RT quite window symbols" this morning, the backfill started again from A (does not seem to be able to differentiate between the symbols that need backfilling and the symbols that do not). It obviously will not manage to backfill 500 symbols by midnight. Yes, I can set the TWS restart time at some later hour but I am not sure if it would help.
For me, it takes about 8 to 9 minutes to backfill 180 days of 1-minute data on a single symbol. So that's going to be over 60 hours at least on 500 symbols. Unfortunately, when TWS closes, the downloaded data may be lost unless you saved the DB before that happened. So consider writing a batch to automatically save your DB every few minutes when you are running a long backfill.
Also you'll need to break up your backfill list into pieces that can be backfilled before TWS closes.
You might want another data provider. IBKR just doesn't care about how slow the backfill is. I hear IQFeed is much faster for historical data.
Another idea is to use jscript to flip through a watchlist of all the symbols you want backfilled, waiting 9 minutes between each symbol change. Make sure it saves after each symbol change so data isn't lost.
Thanks! So my observation was not a user error at all.
Is it possible to auto-save backfills? Say, after backfilling each symbol...?
There is no way to programmatically discover if a symbol's backfill is finished.
You could use AB batch to auto-save every few minutes, regardless of backfill:
You could also write code to check the time stamp of the symbol's database file before backfilling and if it is different 8 minutes later, then new data was saved... but you still won't know if the backfill finished or not, just that new data was downloaded. You could do this in jscript, and you could even call the jscript from AB.
But that's a lot of complexity... you would be writing a bunch of code instead of trading.
Yes, way too complicated.
I just harvested 6 months of hourly bars in 3001 seconds from the IB with my own Python script. I will consider letting AB add hourly bars to this data.
1 day hourly bars in 1785 secs. Almost possible to run hourly taking into account that several symbols waited for the full 300 sec timeout for some reason.
I understand that AB has been designed with HFT in mind (for good reasons, I suppose) with little flexibility for backfilling longer bars. My experiments suggest that it would be a good idea to use alternative means to populate an hourly database.
Hi John, I am using Python 'ib-insync' library to get 2yrs of history on expired futures data (1min).
Here is a link to start with: https://github.com/erdewit/ib_insync/issues/43
You could run this on a daily basis and automate import in AB?
Thanks QD, I was not aware of that library.
I have run into the next issue.
I downloaded hourly data with a Python script and imported it to the AB database to be updated RT. But I have no idea how to turn off the automatic backfill. When I add symbols to the RT quote window, AB just starts to backfill them one by one, screwing up my hourly database.
Found it from the Database settings.
Once you get it all running, could you post your Python script ? The script earlier mentioned here is for downloading expired contracts, you are downloading currently traded stocks. Might be interesting for others to adapt this solution... I for example use 15 minute data, and your solution might be interesting to use.
@Henri , the code I am currently using is somewhat ugly. I started by copying an example from the website of Rob Carver ( This Blog is Systematic: Systematic trading ) and experimented with it quite a bit 2 years ago. RC's initial code is neat but my file contains lots of commented out lines. Currently, I do not have the time review it.
A random Python developer would have no trouble with solving the task by the help of the same website in 2-3 hours. I could post my code at some point in the future but anybody who needs a script to download historical prices from the IB would certainly get it faster by writing it himself using the examples provided by RC or some other website.
BTW, I doubt if the way RC wrote his queues and wrappers is the most optimal, but again, I cannot dive into this at the present.