I have a custom database of securities, queryable over a web-based API. I can dump data into files if need be, do transforms, etc.
Ideally, I wish to integrate into AmiBroker such that it periodically (not realtime, EOD right now) fetches the latest symbol and prices from the database -- ideally as well -- it does this incrementally since the last refresh point for a symbol.
What's the best approach to do this -- sorta like the way AmiQuote works with automatic refresh/import.
I'm happy to write code, use SDKs, but don't want to duplicate functionality that may already exist.
But I'm not sure how it does incremental updates, e.g. refresh since last timestamp for a symbol. I had the crazy idea of adding a very basic ODBC wrapper to the custom datasource, just enough to support AmiBroker's queries. Probably overly complicated.
ASCII import is a simple mechanism, but I really am after some level of automation that's more 'tightly' integrated in.
Thanks very much for the product and this forum.
PS. I really have been searching for this prior to posting, apologies in advance if I could not structure my query well enough!
That batch action "Data import ASCII" is about import but not about creating data file. You said the data file(s) would be created by yourself. So the file to be imported has to exist.
If you re-import the same data file (not being updated) then nothing will change to data entries of AB database. You will have same data entries and same amount of entries there in DB.
It will not be created duplicate EOD time stamp with duplicate or different prices.
And if there is new (updated) price data for same existing EOD time stamp in your file then AB import will overwrite price data of exiting dates within DB (well, and if there is new EOD time stamp then new data entry will be created). So same here... AB will not create duplicate EOD timestamp (just having different price/vol data).
It is different if you use $TICKMODE 1 (concerning intraday (subsecond) data but not EOD). Then if you have duplicate timestamp(s) (same date and time) in your file no matter if having different or same price data then duplicate stamps will be created by AB too.
Tickmode is explained here https://www.amibroker.com/guide/d_ascii.html
But you should not bother about TickMode here because you just work with EOD data. It is just side note.
The date/time are combined into a single field representing in ISO date time with timezone data. I note that generally there's a separate field expected for date, time, and a timeshift -- for timezones.
Ah thank you very much, @fxshrat -- did not realise you could have multiple separators. This works as expected.
(Re: timeshift, yes, was actually referring to that command itself, ).
Okay, thus far I've added my format entry type to <AB_INSTALL>/Formats/import.types and a new format definition file to the same directory. This gets picked up fine, I see my new filetype in the dialog box and I can filter out dataset files appropriately.
However, the document on ASCII import also says:
This file, all other ".format" files and "import.types" file (described later) should be stored in \Formats subdirectory of AmiBroker's current working directory.
My AB_INSTALL above for example is: C:\Program Files\AmiBroker. Wondering if there's a way to store this in a separate folder, e.g. relative to where the datasets are or to where the database is.
For a manual ASCII import (point and click), I've tried it in both those locations, creating a Formats folder within them with the format definition file and import.types. Neither seems to get picked up, only the one in AB_INSTALL registers.
My concern is at AmiBroker upgrade time, will the files and folders in AB_INSTALL be overwritten, including custom formats.
Thanks for your help everyone, I have initial data imported, and have incremental pushes happening through OLE (more precisely a Python script that interacts with the API, then shells out to an OLE JS script passing on the dataset files just created).
I've pushed about 7 years worth of 1-minute interval OHLC data across 16 tickers: any suggestions, besides eyeballing the values (which I will be doing anyway at random points), on making sure I have loaded in everything correctly?
For example, via AFL I could potentially do a bar count and verify that with the no. of CSV records over the time period?
Wondering if people have approaches to doing sanity checks on imported data in AmiBroker.
I suppose an exhaustive check would be to export and compare to source datasets...