3rd party plugins control data, was: is that normal for

I recently tried AmiQuote (4.07x64, unregistered as I'm also using Norgate) with Tiingo to see if it would meet my needs for mutual fund data. I downloaded one symbol with "auto import" selected, and the chart appears in AB (Registered, beta 6.39.1x64) so all appeared to work fine. But when I change from Local DB to Norgate DB and then back to Local, the symbol still appears in the symbol list but it will not plot. If I redownload, 0 bars are downloaded but the chart plots again, until I cycle DBs as mentioned above.

I downgraded back to the official release AB 6.30.5/AQ 3.32, and get the same result.

Can anyone confirm this behavior is just a function of the Unregistered version of AQ, and will not happen with a registered version of AQ?

I just found this quote which seems to indicate it is..."It is advised to buy license so data are stored in database instead of re-importing everything all the time using unregistered version" but would like some confirmation before I spend the money to purchase AmiQuote.

TIA,
Jim

It is not about AmiQuote. AmiQuote is just downloader.

Only AmiBroker accesses its database.

In third party plugin-driven databases, the PLUGIN controls the data.
Read carefully how data plugins control data:
https://www.amibroker.com/guide/h_workspace.html

When database is driven by 3rd party plugin, the plugin CONTROLS THE DATA and you can't import anything onto plugin-controlled datatabase unless symbol is marked with "Use only local database for this symbol" and as such EXCLUDED from plugin control

Thank you, Tomasz. I had read and reread that document but I still didn't fully grasp it. (in the past, I've only used sources with plugins, namely FastTrack, Metastock and Norgate) I thought if you selected "Local Database," that all AQ data would be imported to that location. Once I created a separate Tiingo database and set that as active and downloaded into that, everything worked. Duh!

So, as I understand it, the "Local" database only stores drawings/notes/chart related stuff, whereas the separate databases store all the actual downloaded data files, so you need a separate user-created database for each datasource. Is that correct?

There are TWO settings, not one.
Per-database setting of data source that decides if data are controlled by the plugin, or NOT.

If data source is set to "(local database)" it means that there is NO plugin selected and database is writable and managed by AmiBroker.

image

However if you change it to "Norgate" the moment you change it, Norgate takes control and OVERWRITES anything you have in database with Norgate - content UNLESS
for particular symbol, you have "Use only local database for this symbol" turned ON in the INFORMATION window. This is per-symbol setting that allows certain symbols to be excluded from plugin control.
image

Thanks, Tomasz, sorry if I'm being thick here but I'm still confused. I checked the symbol setting in the Information window and it is the default "No."

So if I understand your last post correctly, I would NOT need a separate database for the Tiingo data, IF I have Data Source set to LOCAL and Local Data Storage enabled, is that correct? AmiBroker should then be handling everything, unless I change it on the per symbol level.

I'm certain that was my configuration at the time I posted the issue, and I understand that selecting the Norgate database overwrites all, and the Local symbol was not available. But when I go back in and change from Norgate to Local, shouldn't the Local database become active again and overwrite the Norgate, allowing the Tiingo (local) symbol to be plotted once again?

If "Data source" is set to "(local)" there is NO plugin involved and everything is saved in local database REGARDLESS of per-symbol setting.

The setting from second screenshot is RELEVANT ONLY IF PLUGIN is enabled
image

YOU SHOULD NOT CHANGE "Data source" in File->Database Settings EVER.
It must be set ONCE AT DATABASE CREATION TIME. It is not meant to be changed back and forth as you are doing.
It is not like that there are two separate databases. There is ONE database. If PLUGIN is activated then it OVERWRITES DATA that you have in the local database folder. So if you change back and forth, data from plugin would OVERWRITE what you have in your local database folder.
If plugin is selected LOCAL DATABASE FOLDER IS USED AS A FILE CACHE, so it gets overwritten by the data from plugin.

Go back to the Users' Guide and look at the picture how it is all organized

https://www.amibroker.com/guide/h_workspace.html

Data from plugin are WRITTEN BACK TO local database:
image

Tomasz,
Thankyou so much for your patience and detailed responses. That last one really cleared up a big misunderstanding on my part! Even with my previous FastTrack and Metastock databases, I had been doing things incorrectly for years.

I think I've got it now :smiling_face:

I thought I was good to go with the Data Source set to Local, but then realized I was not getting the Maintenance Update prompts from Norgate. I emailed their support and was told that the Data Source MUST be set to the Norgate Plugin.

As it would be very time-consuming to manually change the "Use Only Local Database" flag to Yes for hundreds of symbols, I tried the solution on the Norgate site (under AmiBroker Plugin) to import the Tiingo symbols and change the Database flag to Yes upon import. I tried it using the Wizard first, but the database flag wouldn't change. I then tried the Ascii Import feature (creating a new default.format file) but that seemed to ignore both the Watchlist and the Database flag. (I saw another user posted in 2019 that he couldn't get the importer to change the flag either)
$FORMAT FullName,Ticker,Skip,Skip,Skip,Skip,Skip
$SEPARATOR ,
$GROUP 255
$WATCHLIST 78
$AUTOADD 1
$NOQUOTES 1
$USEONLYLOCALDB 1

I also saw your post from 2006 about doing it programmatically but couldn't get that to work either.
image

So, unless I've done something above incorrectly (certainly possible), my only options are to just use a second database and switch between the two (which you recommended against) or else change hundreds of symbols manually.

Any other thoughts?
TIA

According to:
https://www.amibroker.com/guide/d_ascii.html

image

$USEONLYLOCALDB does not affect existing symbols.

So, you'd probably want to wipe out your existing symbols and reimport them with that flag (if you have the history).

Thanks, but I had been “permanently deleting” the test set of symbols before each import test, so that wasn’t it.

It works as described in the manual.

As @NorgateData says, if you want to use ASCII importer to import data into plugin-driven database, you have to:

  1. Make sure that the symbols you are trying to import DO NOT EXIST in plugin-driven data source.
  2. If they exist, you have to delete them (Symbol->Delete)
  3. Do the ASCII import with $AUTOADD 1 turned on to the database that uses "Data source" set to "Norgate" (or any other plugin).

When TWO CONDITIONS happen AT THE SAME TIME:

  1. Symbol is ADDED via ASCII import ($AUTOADD 1) (obviously to be added it MUST NOT exist before)
  2. Database is plugin driven (Data source set to plugin)

Then ASCII importer will ADD those new symbols with "Use only local database for this symbol" set to True.

You just do normal AmiQuote import with quotation data. You do NOT NEED to import symbols separately. Should NOT use $NOQUOTES flag.

As I wrote: YOU SHOULD NOT CHANGE DATA SOURCE EVER.

STEP1: Create database with NORGATE plugin if you want to use it and LEAVE IT.
Don't touch "data source" ever.

STEP 2: Then IMPORT DATA (for NEW SYMBOLS, NOT existing in Norgate).

That is all you have to do. Nothing more is needed.

Success!!! Thanks, Tomasz, this was most helpful, specifically the info on NOT doing separately, and just using the AQ import without the $NOQUOTES flag. I then need to add the Fullname in a second import but the database flag is getting set with the first import.

Still having problems with the $WATCHLIST. It works using the Import Wizard when already created, but not with the default Ascii Import process. Does the Watchlist already need to be created, or should the (default) Ascii Import create it? For example, if I use $WATCHLIST 85 in the default file, and that watchlist number does not yet exist, is it supposed to create it? It doesn't seem to be doing that, but at least the database flag is getting set properly. Thanks again!!!

No, it won't create watch list. The watch list needs to exist.
If you want to import watch list alone, there is separate functionality for importing symbols into watch list Working with watch lists

Thanks, Tomasz. The Ascii Importer is being difficult. I have the following group and watchlist named: (the watchlist is 85)
image

image

I'm using this default.format file:
$FORMAT FullName,Ticker,Skip,Skip,Skip,Skip,Skip
$SEPARATOR ,
$GROUP 201
$WATCHLIST 85
$AUTOADD 1
$USEONLYLOCALDB 1

I run the AQ download with AutoImport checked.

With these settings, the symbols get loaded into Group 254, and not into any Watchlist. However, if I import a CSV file (only full name & ticker, all else skipped, no quotes) using the Import Wizard, and manually select the group and watchlist from the dropdown, the symbols get added to the correct group and watchlist.

This would seem to indicate there's a problem with the default.format file, but I just don't see it. What am I doing wrong?

No it is not "difficult". It works perfect. Simply you don't understand it yet.
The wizard uses ASCII importer. The Wizard is just UI. All it does is to create import definition for ASCII importer.
Please note that default.format file is not used by "default". It is used only if it is specifically chosen via "Files of type" combo box. So most likely you are importing with OTHER importer definition than you think you are.
More on this:

The article explains how to create and use proper import definition. By all means you should not by typing manually things that can be created by the software automatically.

Haha. "Being difficult" was just a figure of speech, Tomasz. I wasn't trying to imply the software didn't work; I'm well aware the problem is between my ears. :wink:

You are correct. I assumed the default.format was being used. I'll review all the documentation on the ASCII importer again to try and get a better understanding. And thanks for the KB references.

Progress! I'm now able to import Tiingo mutual fund quotes, full names, and assign groups and watchlists as desired (along with local database flag).

What I've NOT been successful at is RE-importing an existing historical data file using these instructions:

Nothing gets imported and I don't get any errors. Does this process use the format file based on the data file extension? (.aqtc) BTW, it doesn't appear in the file filter.

Also, does the AQ "automatic-import" feature/checkbox use a format file? This process works great, but I have to keep re-downloading the files each time for testing unless I can figure out the manual import.

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.