AmiQuote 4.00 released

A new public version of AmiQuote 4.00 is released now.

Direct links:
http://www.amibroker.com/bin/aq4000.exe (234KB) – 32 bit version
http://www.amibroker.com/bin/aq4000x64.exe (289KB) – 64 bit version

NOTE: 64-bit version of AmiQuote now requires Microsoft Visual C++ 2017 runtime. You may need to download the runtime separately and install it, if you are installing AmiQuote stand-alone (without AmiBroker 6.30 64-bit or higher already installed). If you already have AmiBroker 6.30 64-bit you don't need to do anything.

CHANGES FOR VERSION 4.00 (as compared to version 3.33)

New Generation of AmiQuote, now allows User-definable data sources for complete freedom of choice and quick adoption to data sources that appear on the web as well as existing sources changing their API:

  • UI: New top-level menu: Data sources: New/Edit/Delete/Import/Export
  • UI: New Data source creation/editing dialog allowing definition of user-defined sources
  • UI: Export and Import of Data source definitions allowing easy sharing of definitions between users
  • Removed defunct Google Finance built-in source
  • Removed BarChart.com built-in source (it can re-added as user-defined)
  • Added example importable data source definitions in "DataSources" folder (AlphaVantage, Tiingo Intraday IEX and more)

Full documentation on new features is included in the AmiQuote Read Me.

(This thread continues the discussion from AmiQuote 3.33 is released now)

9 Likes

QUICK START GUIDE TO USER-DEFINABLE DATA SOURCES (new feature in version 4.00)

Version 4.0 is a new generation of AmiQuote that brings user-definable data sources. The idea is to allow any web site that delivers quotes in "consumable" format (currently CSV) to serve as a data source without long development process, so adoption of any new sources and/or changes would be easier and much quicker.

The easiest way to get started with User-definable sources is to Import example data source files.

AmiQuote data sources are XML-based text files with .ADS extension (ADS stands for AmiQuote Data Source). Example files for AlphaVantage, Tiingo IEX intraday and more are included in the "DataSources" subfolder under AmiQuote folder.

How to import data source from file?

To import user-definable data source, select Data Source > Import menu. Then file dialog will appear, navigate to "DataSources" subfolder and look for exisitng files with .ads extension. Pick on and it will be imported. Newly imported data source will be automatically selected in "Source" combo box

image

How to use imported data source

You would use imported data source the very same way as you use built-in sources.

Select it from Source combo box, set From / To dates, select desired Interval , add symbols and press Start download.

Some data sources may require API key (kind of private code to authorize your access to given data source). If that is the case, AmiQuote will display the message like this:

image

If you press OK button, the data source Setting dialog will be displayed where you can enter API key that you can retrieve from given data source web page:

image

Each data source has separate setting for API key (authorization token), so if you use for example Tiingo.com server in more than one user-defined data source you would need to enter the API key in each data source setting separately.

How to add your own data source

To create a new data source from scratch, use Data source > New menu, then follow instructions below to edit settings

How to delete user data source

To delete user-definable data source , use Data source > Delete. Caution: this operation cannot be undone.

How to edit data source settings

Data sources can be edited by selecting Data source > Edit menu, or pressing Edit button next to Source selection combo box on the main screen.

Data source Settings dialog has currently to pages: General and Supported Intervals.

General page looks as follows:

image

The fields available are as follows:

  • Name - this provides short name of data source (will be used as name in "Source" combo) - required
  • Description - the extra description of what data given source provides (will be displayed next to name in "Source" combo) - optional
  • URL Template - the template which will be used to create actual URL to request data from - required

URL is the address of where data can be downloaded from. The template is a system that allows to replace certain parts, called tokens, enclosed in curly braces like this: {token_name} with certain values at runtime. For example {symbol} can be replaced with current symbol and {from_ymd} by from date in YYYY-MM-DD format. More on available tokens in later section

  • API key - defines {api_key} token value. May be required by some data sources
  • Download type - selects downloaded data type/format. Currently only CSV format is supported - required
  • Error message prefix - defines the text that will be searched for to detect any error messages generated by external web page - optional. If given text is found in the server response, the error message that follows this prefix will be displayed and this item download will be marked as failed
  • Remove double quotation marks (") - optional - marking this option will remove any double quotation marks from downloaded data
  • Convert ISO datetime into date and time columns - optional - marking this option will convert single CSV column ISO date/time stamps in the format of either: YYYY-MM-DDTHH:MM:SS where letter 'T' separates date and time parts, or YYYY-MM-DD HH:MM:SS where space is used between date and time parts, into two separate columns, one for date YYYY-MM-DD and one for time HH:MM:SS (with regular comma as a separator between columns)
  • Requests per minute - defines maximum number of requests per minute that given source allows - optional. Zero means no restriction.

Some data sources put strict limits on how many requests you can send per minute and if you exceed that the downloads will fail with error. For example AlphaVantage has a limit of 5 requests per minute. If the limit is hit, AmiQuote will pause download until minute passes by and then it will continue.
Such pause is announced to the user like this:

image

  • Daily format - describes how AmiBroker's ASCII importer should interpret column values for Daily (EOD) downloads. Required. This string is copied directly into $FORMAT command of the importer and stored in automatically generated import definition file for AmiBroker
  • Intraday format - describes how AmiBroker's ASCII importer should interpret column values for Intraday downloads. Required. This string is copied directly into $FORMAT command of the importer and stored in automatically generated import definition file for AmiBroker
  • File extension - defines the file extension (without dot) for downloaded files and name of automatically generated format definition file for AmiBroker ASCII importer. Required. If for example File extension is set to xyz then data files will be named like MSFT.xyz and automatically generated format definition file stored inside "Formats" subfolder will have the name of xyz.format
  • Separators - defines character(s) used as column separators for imported files. Required. This will be copied into $SEPARATOR command of the importer. By default comma is used, but it can be any other character. If multiple characters are typed here then each one acts as separator.

Supported intervals page looks as follows:

image

Here we have a list of intervals that are supported by AmiQuote: End-of-day , 1-minute , 5-minute , 10-minute , 15-minute , 30-minute and Hourly and two columns Max days and {inteval_extra} token value

Max days column defines how many days history is available from given data source at given interval. The value of 0 (zero) or "Not supported" means that given data source does not support given interval. If you define non-zero value for any interval it means that given interval is available and AmiQuote would allow to download upto as many days as specified. If you exceed that amount, AmiQuote will display the error message and will auto-adjust "From" date so the download range does not exceed specified number of days

{inteval_extra} token value - is optional field that defines the value of {interval_extra} token. This token can be used if given data source has different data end-points (URL parameters) for different intervals. For example Alpha Vantage uses URL parameter function=TIMES_SERIES_DAILY_ADJUSTED for Daily interval and function=TIMES_SERIES_INTRADAY for all Intraday intervals. By using the above settings you can use function={interval_extra} in the URL and {interval_extra} token will be replaced at runtime appropriately to selected interval.

Always use today as "To" date - optional - when turned on it forces "To" date to be always set to to Today. Some data sources only allow specifying "From" date and always deliver data until "now" and this option is useful in such situations.

Available URL template tokens

Here is the list of available tokens that will be replaced at runtime with respective values:

  • {api_key} - will be replaced with API key defined in General tab of the Settings dialog
  • {symbol} - will be replaced with current symbol
  • {interval_minutes} - will be replaced with numeric value of interval expressed in minutes, for example: 5 for 5-minute interval
  • {interval_seconds} - will be replaced with numeric value of interval expressed in seconds, for example: 300 for 5-minute interval
  • {interval_extra} - will be replaced with value defined in Supported intervals tab of the Settings dialog respective to selected interval
  • {from_ymd} - will be replaced with the "From" date in YYYY-MM-DD format
  • {from_day} - will be replaced with the "From" date in DD format (two digit day only)
  • {from_month} - will be replaced with the "From" date in MM format (two digit month only)
  • {from_year} - will be replaced with the "From" date in YYYY format (four digit year only)
  • {from_month_name} - will be replaced with short month name of "From" date (example: "Jan" for January)
  • {from_month_fullname} - will be replaced with full month name of "From" date (example: "January" for January)
  • {to_ymd} - will be replaced with the "To" date in YYYY-MM-DD format
  • {to_day} - will be replaced with the "To" date in DD format (two digit day only)
  • {to_month} - will be replaced with the "To" date in MM format (two digit month only)
  • {to_year} - will be replaced with the "To" date in YYYY format (four digit year only)
  • {to_month_name} - will be replaced with short month name of "To" date (example: "Jan" for January)
  • {to_month_fullname} - will be replaced with full month name of "To" date (example: "January" for January)
15 Likes

Now that is Superb Tom!! I'm really looking forward to playing with this.

Thank you.
:heart_eyes:

1 Like

Thanks for this revolutionary release. I've got my toy for the weekend.

2 Likes

@QuantTrader, @Sean - You're welcome ! I am glad that you like it :slight_smile:

There is a plan (for future versions) to add JSON support and even HTML table extraction and conversion to CSV for pages that don't offer computer friendly CSV format

7 Likes

WOW, that is fantastic

Tomasz,

Absolutely fantastic! In less than 10 minutes of tinkering, I have AmiQuote downloading and importing more than 20,000 symbols of US/TMX/other data from my account at EodHistoricalData.com.

If anyone uses them, I've included the body of my ADS file in the code block below. You can cut/paste it into Notepad and save it with an ADS extension (I called mine "EODHistoricalData - Stock Data API.ads"), then import into AmiQuote. Obviously, being so new this hasn't been tested exhaustively yet, but it seems to work for me. Make sure you enter your API key.

Looking forward to the JSON support as my data supplier makes loads of fundamental, classification and even stock options data available in JSON format. Right now I schedule parsing using Python scripts on an Airflow server, but clicking Import in AmiQuopte sounds much, much simpler. :smile:

Again Tomasz, really happy with this upgrade.

Thanks!

Daniel

<AmiQuote Version="4.00">
<DataSource>
<Name>EODHistoricalData - Stock Data API</Name>
<Description/>
<URLTemplate>https://eodhistoricaldata.com/api/eod/{symbol}?from={from_ymd}&amp;to={to_ymd}&amp;api_token={api_key}&amp;period=d</URLTemplate>
<ErrorMsgPrefix>EOD Historical Data - </ErrorMsgPrefix>
<Options>
<UseTodayAsEndDate>0</UseTodayAsEndDate>
</Options>
<Limits ReqPerMinute="350"/>
<CSVFormatOptions RemoveDoubleQuotes="0" ConvertISODateTime="0"/>
<ImporterOptions>
<FormatEOD>Date_YMD,Open,High,Low,Close,AdjClose,Volume</FormatEOD>
<FormatIntra>Ticker,Date_YMD,Time,Open,High,Low,Close,Volume</FormatIntra>
<FileExtension>eod</FileExtension>
<Separators>,</Separators>
</ImporterOptions>
<SupportedIntervals>
<Interval Period="1440" MaxDays="3650" ExtraToken=""/>
<Interval Period="1" MaxDays="0" ExtraToken=""/>
<Interval Period="5" MaxDays="0" ExtraToken=""/>
<Interval Period="10" MaxDays="0" ExtraToken=""/>
<Interval Period="15" MaxDays="0" ExtraToken=""/>
<Interval Period="30" MaxDays="0" ExtraToken=""/>
<Interval Period="60" MaxDays="0" ExtraToken=""/>
</SupportedIntervals>
</DataSource>
</AmiQuote>

P.S. I haven't posted since the Yahoo groups, so you made an old dog happy enough to come out of hiding... lol

11 Likes

I am happy to hear that after all those years I am still able to surprise users :slight_smile: Keep up the good vibe.

Daniel, thank you for your contribution! Here is your file in downloadable format, for even easier access:

EODHistoricalData.ads (1.2 KB) - data source definition file for AmiQuote 4.00 courtesy of @optim

2 Likes

Quick question if I may Tom. I wrote a OLE in C# to control and run many AQ Barchart runs. If and when I am able to re-create it in the new AQ version (as a new source instead) will I be able to execute those type sources as an AQ instance via OLE, as well as change settings and API keys in the registry?

-S

Sean, I am not sure what settings and API registry keys you mean, but you can run AmiQuote via OLE and select user-definable source by specifying its ordinal value in .Source property. For example if your user-definable source is at 8th position in the "Source" combo box then it has index 7 (counting starts from 0), so:

AQ = new ActiveXObject("AmiQuote.Document");
AQ.Open("C:\\mysymbols.tls");
AQ.Source = 7; // user-defined sources start at index 7
2 Likes

Cool, I just didn't know the User defined Sources would be accessable via OLE or not.

I edit certain AQ settings in the registry via c# code between launching instances that aren't available via OLE like Barchart API key etc. Of course my code always resets setting to the way it found them when I'm done.

You can define multiple sources using same web site but different key and just select between them.

1 Like

giphy

1 Like

Tom quick questions?

Getting the following (Tops)...
symbol,sector,securityType,bidPrice,bidSize,askPrice,askSize,lastUpdated,lastSalePrice,lastSaleSize,lastSaleTime,volume,marketPercent,seq
NLNK,pharmaceuticalsbiotechnology,commonstock,1.78,100,0,0,1558712045482,1.79,5,1558711461067,711,0.00373,
WVE,pharmaceuticalsbiotechnology,commonstock,25.34,100,0,0,1558711987854,25.48,100,1558711870587,4893,0.05515,
IPLDP,n/a,misc,0,0,0,0,1558704600001,0,0,0,0,0,
IGBH,n/a,etp,0,0,0,0,1558704600001,0,0,0,0,0,
HURC,capitalgoods,commonstock,0,0,0,0,1558711567231,37.13,300,1558710777989,606,0.07374,
ZIXI,softwareservices,commonstock,0,0,9.36,100,1558711995051,9.325,100,1558711821458,19403,0.04914,

-or for simplicity sake...-

symbol,price,size,time,seq
PALL,125.9,25,1558702830959,
UMH-D,24.2,100,1556113648051,
WNRL,23.825,100,1509393578822,
GHM,20.8,70,1558710281240,
ASLN,4,100,1555617504609,
BIOX,4.77,100,1558711132563,

I have a few question to make this a data source.

1.) The first issue is the DateTime fields are Unix Epoch, can you support this?

2.) The other is that AQ insists on the user to add a symbol to download, before downloading, even though this gets all tickers and takes no url ticker parameters etc. I was able to just add a ticker like "ALLTKR" to get past this.

3.) This can be run 1 time a second etc?

4.) For whatever reason there seems to be an extra null CSV field at the end?

5.) Can you add a feature/option to ignore fields of 0 or 0.0 etc?

1 Like
  1. Unix Epoch was used by Google, and I have that code is capable of handling this, but this functional is not yet exposed to user-definable sources.

  2. Yes it needs a ticker even if it is artificial one

  3. Only if you trigger via OLE, normally "Run every" feature has 1 minute minimum time

  4. Null in what sense? Empty string, Zero character or ?

  5. Importer automatically ignores entries with price == 0 (unless you set $ALLOWNEG 1)

2 Likes
  1. Null in what sense? Empty string, Zero character or ?
    line ends with a comma, then nothing after it, I showed it in my post.
  1. Importer automatically ignores entries with price == 0 (unless you set $ALLOWNEG 1)
    There are Zeros ",0," for price fields, when there is no trade yet during the day or nothing bid or nothing offered thru IEX. (See post).

Thanks Tom. Have a nice weekend.
-S

  1. Apparently it comes like this from data source, no?
  2. I understand, but ASCII importer would simply skip those (you can see errors generated in import.log for those lines, but other than logging an error, the import skips such line and continues without problem.
1 Like

Thank you for this great utility,
I am from india , I found timestamp in US format want to convert into IST , how to do ?
This is how downloaded timestamp
timestamp2

want this ....

timestamp3

little help regarding this will be very helpful.

Thanks Tomasz. I hope I can get this working with Tradingview. Tradingview is one of the most locked down sites I've ever come across, even for subscribers.