Greetings All -
Running into an issue with Daily Bars and futures contracts. I use DTN IQFeed for Amibroker and collect 1 minute data. I also use data from CSI which I import into Amibroker and read in .csv format. DTN data is contract expiration specific - while my CSI data is back adjusted as per their format.
When I compare RECENT data (i.e. not subject to back adjustment) I can't seem to get the DTN Daily bars to replicate the CSI data. My DTN highs, lows, opens, and closes are "off". It's clearly a "session timing thing" - I just can't seem to figure it out.
I've played around with the data base settings - intraday settings - Filter weekend - Categories (Groups) - Organize Assignments (Groups) - etc. - but just don't seem to "get it right".
I've seen various articles within the Community Forum - but must have missed something - somewhere.
Can someone please point me to a specific article or guide that details how to properly establish a futures contracts database with Daily bars that reflect what the actual exchanges publish (the way CSI does)? I need a systematic "How-To" guide. If there isn't one and I figure this out - I'll write one up.
The end goal is have DTN data of Daily Bars replicate what CSI does - but with a "live" last bar that updates.
All the best - and thanks for your patience,
Posting a small update here....
When I run the IQFeed chart program - the daily bar data completely aligns with the data from CSI - which is great.
But the daily bars within Amibroker don't correspond with the DTN bars.
It's my (limited) understanding that DTN sends 2 types of data - intraday and end of day bars. And yes - I DO have the "Allow mixed EOD/Intraday data" box checked. I also have the Filter Weekends box checked to eliminate Sunday night bars....
Since Amibroker seems to create the Daily bars from the daily DTN data, I wonder what's creating a distortion in price bars with regards to OHLC.
Allowed mixed/intraday doesn’t work that way. It’s designed to request intraday then if data provider doesn’t have enough intraday data for your requested look back, it pulls EOD data. For something like futures. They provide a huge intraday look back.
The problem is with your time settings. Not sure how to do it since I’ve never worked with futures. But pretty sure a futures day is from 5 pm till 4pm the next day. Theirs an hour maintenance time. Opens Sunday at 5pm and closes 4pm Friday.
I’d start with looking at IQFeeds 1min data using those times. Then you can look into
And under intraday settings use show day/night perhaps and set the times. Then use for compression use night/day session.
Not sure if that will work going backwards.
Other simple option is run two instances of Amibroker, one for EOD and one for intraday. What I use to do for stocks in IQFeed since their intraday is not split adjusted while their EOD stock data is. Used to cross reference huge gaps.
Many things could be happening here and some things to look at. So let's see...
1.) CSI has Symbols which represent 24 hour, day session, electronic etc. Not to mention settlement which you can alter how it builds the daily bar I believe.
2.) Check any intra-day source you use for time stamp and time zone, especially when converting your chart to daily. Some use exchange based time zones, so CME is different from NYMEX etc.
3.) Intra-day data might only show last tick (tick when traded) at close and CSI might be set to use settlement. If the settlement is below the Low or above the High those will get changed by CSI, depending on your settings.
4.) In AB, there is a time stamp setting which defines the start and end times of the intervals, for example is the 3:59 bar the end or the start time of the bar, stuff like that.
5.) AB doesn't understand that some instrument's Monday bar can start on Sunday evening. Add that to time zones issues and it gets very involved based on date changes.
Bottom line is that you have to know where the CSI daily data is obtained from, how it is aggregated and what time zone. Then find out the same on the intra-day symbols. Look at the raw intra-day data before it gets into AB and compare the OHLC of the daily and you should be able to figure out what the mismatch is due to.
Hey Metamega -
You stated "Allowed mixed/intraday doesn’t work that way. It’s designed to request intraday then if data provider doesn’t have enough intraday data for your requested look back, it pulls EOD data. " If I'm reading that correctly, then it means that I get a prtial amount of intraday data - supplemented with historical daily data. Which in a perfect world won't happen without issues I suppose.
If that's how DTN works (or at least within Amibroker) - then I'll just stick to EOD data from CSI.
I think you're absolutely correct re: time settings. But when I created groups and assignments with session-specific inputs - nothing changed.
Thanks for your help.
Hi Sean -
Thanks for the info. I found it interesting that the DTN Daily bars were matching great with CSI's - but I can't seem to get AB to read just EOD data from DTN without creating a separate database.
If I have to switch between databases it's not a big deal I suppose. I'll have to wait until Sunday night to see if DTN's daily bars actually update live or not.
Thanks for the help!
If I get a chance I’ll see if I can’t try something with the intraday settings options. Think I have some intraday data laying around.
Now the mixed eod/intraday isn’t much of an issue with IQFeed. Their futures look back for 1minute is 10+ years. I could’ve swore I remember them saying ES was early 2000’s somewhere, so you’d have to request intraday data long enough to get past their intraday and then it will start pulling EOD data. Unless your backtesting, I can’t see any need to go that far back. It’s just a lot of unnecessary data to load up. IQFeed also is 180 days for tick data backfill.
Esignal for example I believe is 2 years intraday and decades I believe of EOD. When IQFeed plugin was wrote for Amibroker it probably seen more use with mixed eod/intraday but now the intraday database has gotten quite large. Especially with futures being pretty much 24/7. 10+ years is a lot of data points.
Thanks for your help and interest. Here's what I've discovered so far:
Mixing the 2 type of data - intraday and Daily causes two issues for me. A) The Daily data bars are just plain wrong with regards to OHLC. B) After a short period of time (3-5 days) some of my contracts become corrupted when displaying daily data. Deleting the symbol and re-entering it fixes it immediately. And again, I have the check mark where it should be to address the two types of data.
If I create a separate database and set it to only collect Daily bars, the bars plot 100% correctly. The last bar is also "live" so it is constantly updating during the session. So I'll have to run 2 databases - one set to 1 minute bars and one set to Daily bars - and switch back and forth when required.
DTN just told me that they send (in my case) two distinct types of data - 1 minute and Daily. What happens at MY end is how the program deals with constructing the charts - which is well above my pay grade. I've sent an email to Tomasz regarding this and will post a reply when received.
As far as daily data. One thing I use to notice with IQFeed and stocks, was that if I was watching a daily chart, last bar is updating, and I close Amibroker before closing of bar at exchange time, the next day that bar won’t update after new bar starts.
My routine was if I close Amibroker during exchange time’s, I’d open after market close and perform a simple scan of database to update bars. This solution wouldn’t work however with futures being pretty much 24/7.