TimeFrameExpand arrays from arbitrary data source / alignment

I have a ton of "auxiliary" data on the daily timeframe (i.e. shares outstanding, float, blah, blah). I understand there are many ways to store and retrieve such extra data from older posts here, e.g. OLE excel, fopen csv, rest api, odbc, foreign ticker, aux1/2 fields, etc. Let's rule out aux1/2 b/c (a) I have many more scalar values than this, and (b) my analyses are on the 1-min scale, so even if aux1/2/N were allowed, this would be prohibitive in terms of disk and memory space. We also rule out foreign ticker for similar reasons.

But, I think this post is less about the data source, and more about how to properly prepare that data for a TimeFrameExpand when also considering I may not have this auxiliary data for each date for which I have price data.

Hopefully that preface was useful :upside_down_face:.

My actual GOAL: What I'd like to be able to do is pull in this sparsely populated daily-period data (which can be stored efficiently in memory), pick whatever subset of field(s) I need for my analysis (e.g. current float > 10M, etc.), then TimeFrameExpand this to align with my 1-min data.

My current best understanding: I think I have a grasp on all of the steps to achieve this except for how to properly prepare/format the array of the daily data for TimeFrameExpand. My best understanding is that I need to prepare an array with same size as the OHLCV 1-min arrays, then put the daily values in some compressed portion of that array, maybe aligning dates using guidance from a TimeFrameCompressed date array. Or maybe I just directly try and construct the 1-min array. I very well am barking up the wrong tree, so any guidance on the best approach/design here would be appreciated.

The best reference I found discussing something similar was from @fxshrat in his solution in this post. It looks like the bit twiddling was done in a DLL there, but I could not find any additional details.

1 Like

Why did you rule out Foreign? It would do timestamp padding for you.

The other idea is to use ODBC plugin. It would also do the padding for you.

1 Like

@Tomasz Thanks. I will take another look at Foreign. It wasn't clear to me if I could store Daily period data in foreign and pull it into an intraday analysis, so I'll experiment a bit with it later and see what happens.

I thought about my proposed solution a bit, and was able to come up with something that works, i.e. timecompress the 1-min DateNum, call Python function that creates a new compressed array using the DateNum compressed array as guidance for where to put values, then expand again in AFL. Another way to do it would be create a BarIndex-sized Matrix and fill that out with my aux data (sounds more like what @fxshrat did). It works, but it definitely doesn't feel "right".

Okay :sweat_smile:

Foreign totally does what I needed. I can enter the data as EOD data and it auto-aligns and expands to the 1-min data. It may be limited to 8 columns, but I can always add more foreign symbols as needed. I like this solution because everything is self-contained in the database, no plugin(s), no python, no breaking.

Thank you @Tomasz. I still have a lot of work to do, but I'm sure now AmiBroker can almost completely replace my existing Python pipeline. I only wish I knew about it sooner. AmiBroker is an engineering marvel in so many ways.

6 Likes

Thank you for your kind words. Glad I could help :slight_smile:

2 Likes

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.