I know a nearly identical question has been addressed and closed by Tomasz about 19 days ago. At the risk of being again the careless user who has "lost of the art of reading" man pages -- a justified label for me in at least one of my posts to this forum -- please bear with me for another minute or two.
The manual, cited in the topic above more than once, explains that the import file format is selected by file extension. However, this doesn't seem to provide for the case of multiple file format definitions with the same extension (e.g. *.csv), a feature that is available in "File/Import ASCII..." menu from application's GUI. ..
The Batch command "Data Import ASCII" has a similar appearance to the screenshot above, but the "File of type" doesn't allow for the selection of a specific ASCII format defined in "Formats" Folder, only the file extension types, which is consistent with the explanation in the much-cited man page.
Again, my apologies ahead of time to ask a probable "dumb user" question. If it's doable with a Batch script, please show me the relevant passage in the user doc or a screenshot. I've also looked into OLE Automation. There is one sentence in the "Import from ASCII File" manual that suggests it is doable via OLE Automation. However, I have yet found the relevant calls from the OLE Automation manual, which I have used successfully to automate exploration export from a Python script before the introduction of the Batch feature.
It does NOT matter what appears in the ASCII importer “Files of type” combo box. This is UI of ASCII importer not the batch processor. This combo box displays content of Formats\import.types file and is completely IRRELEVANT to batch.
What matters for BATCH is the extension of imported data file and actual name of format definition file, so to import yourfile.CSV data file, batch would use Formats\CSV.format file and ONLY that file. It does not matter what appears in the ASCII importer file picker UI.
If you have special format that you want to use you just need to have to RENAME YOUR DATA FILES so they use specific extension. For example you can use mydata.GRAYLEAF and then batch import will use Formats\GRAYLEAF.format file.
Again: if you use BATCH to import
yourdata.YOUREXTENSION
it will automatically use
Formats\YOUREXTENSION.format
definition file.
If you have more .CSV files each having different format you need to CHANGE the extension of your DATA FILE.
I really think it's quite annoying as opposed to elegant.
When I stumbled across the batch import feature, I had wished I could tell the batch analysis which custom format to use, just like in ASCII importer.
Here is Python script that will rename the extensions and leave all original files intact.
# -*- coding: utf-8 -*-
"""
Spyder Editor
Author: Matthias C. Kuschel
Date: 31 May 2018
version: 1
What it does:
1. Check if directory2 exists, if not, create it
2. If directory2 exists, delete all files ending with "new_extension"
3. Copy all files in directory1 ending with "old_extension" into directory2
4. Rename all file extension to "new_extension"
Used for Amibroker Batch Analysis
Creates custom extension so "Import ASCII quotes" can be automated
"""
import os
import shutil
new_extension=".mck"
old_extension=".CSV"
directory1="C:\\Python_SCRIPTS\\sampledata"
directory2="C:\\Python_SCRIPTS\\sampledata\\" + new_extension
if not os.path.exists(directory2):
os.mkdir(directory2)
todelete=os.listdir(directory2)
for file in todelete:
if file.endswith(new_extension):
os.remove(os.path.join(directory2,file))
for file in os.listdir(directory1):
filetocopy = os.path.join(directory1,file)
if file.endswith(old_extension):
shutil.copy(filetocopy,directory2)
for file in os.listdir(directory2):
filetorename = os.path.join(directory2, file)
filename, extension = os.path.splitext(filetorename)
os.rename(filetorename, filename + new_extension)
The script will change the data extension (say from MYDATA.csv to MYDATA.csv1) to a custom format (user-defined), then one can use the batch "import ASCII" for the specific, custom extension.
Lastly, if the new extension is not yet recognized by AB, in the folder "FORMATS", the
import.types file needs to be modified
Turns out, the above post has some errors, let me try to correct them.
How it (finally) works (after all, needed some hacking for my stuff to work):
Setup description: all the data to be imported is in one folder
In this case, all data ends with ".CSV"
Run python script to create a copy of the data, put it in another subfolder, re-label the ending to ".mck"
create a mck.format file in \Amibroker\Formats directory (containing your custom import requirements for OHLC, AUX1, etc)
add file extension and association to \Amibroker\Formats\import.types
Use batch process command "Data Import ASCII" PER SYMBOL to import the quotes. (so if you have a 1000 symbols, you need to hack in there 1000 symbols which is sort of troublesome, especially if you consider some symbols may change name or drop in/out. If you are used to importing ASCII via "select all" = "Ctrl+A" in ==YOURDATAFOLDER==, the batch import is not safe to run without extra measures)
Practically, the approach outlined has some issues, here are the workarounds, I really don't give a damn if that costs me a second or two in terms of execution time, my stuff is EOD.
My solution to programatically execute a PYTHON script is this: Exploration on 1 symbol with 1 recent bar(s)
The code below executes the .py file for changing the data file extensions. The ThreadSleep(100) is necessary because AB runs the exploration in a few ms, but the python script needs like 2 secs to execute. As there is no delay command for the batch to run or any other way to me currently known when the python script finishes that's a hack to solve it.
actually takes in for inputs, i.e. what other commands other than /dowload and /close are there.
The idea with .apx files is good as it really stores all the configurations, yet when combined with the batch run one really has to be careful because to successfully operate it one needs at least 3 files: .afl, .apx, .abb. Quite easy to make a mistake that remains unnoticed, especially when running several .apx's in one batch.
@Dionysos, just a short question: why do you use an exploration with a ShellExecute to run your Python script instead of an ExecuteAndWait command with something similar to:
Hi beppe!
I was not able to wrap my head around the ExecuteAndWait command in this case.
So what I tried was this: (for my case)
It says completed, yet nothing happened.
My command prompt flashes up quickly, but effectively the .py never ran. Would be beautiful to have that ExecuteAndWait working, especially the "Wait" part in this case.
I don't know what else to do.
Thanks.
@Dionysos in my example I placed the .py script name in " " quotes (no spaces allowed in unquoted parameters). The Python executable seems ok since you do not have any space in the full filepath.
I used your batch file (changing the directories), and here it works also using a folder where I have a specific version of Python (no space in the path).
One additional check for you: maybe in your system the executable is Python3.exe (verify in a DOS command prompt windows what is the correct version to run)
in command prompt, my python.exe is indeed called python.exe, not python3.exe or that alike.
Test 1:
Running python.exe directly in C:\ProgramData\Anaconda3>python.exe
No "" quotes for C:\Program Files\AmiBroker\Donchian System\RENAME_FILE_EXTENSIONS_v1.py
==> Error because there's a space. That's correct.
Test 2:
Running python.exe directly in C:\ProgramData\Anaconda3>python.exe
""Used for file path
Works!
Test 3:
Absolute path for both file locations and "" to be sure: works!
Now, I played with this for a couple of hours, I am still convinced that I got it right initially and it wouldn't launch my .py script. I am puzzled what went wrong, yet it works now. Says completed, and it is completed.
Thanks again for the pointers Beppe.
Have a nice day.
Here is another nifty python script:
If you want to import quotes via batch, you may not know what files are in the quote directory.
Especially if you want to add it to batch import, you need to type a lot of stuff.
This script reads all filenames in directory ending with ".mck1" and copies them in a file called "___OUT.TXT".
The "___OUT.TXT"-file is placed in the same directory where you launch the script from.
It adds the correct format for a .abb batch file, ready to be copy/pasted
remember to finalize the .abb file with </Amibroker-Batch>
Script:
# -*- coding: utf-8 -*-
"""
Created on Fri Jun 1 20:16:51 2018
@author: Matthias C. Kuschel - Private Hedge Fund
reads all files in "path" with specified ending
copies file names to a .txt-file, located in the same directory the .py script was launched
"""
import os
import csv
path="C:\\UA\Data\\All Futures\\.mck1"
allnames = os.listdir(path)
with open("___OUT.TXT", "w", newline="") as outfile:
writer = csv.writer(outfile, delimiter=",")
for file in allnames:
if file.endswith(".mck1"):
writer = csv.writer(outfile, delimiter=",")
row = [file]
writer.writerow(["<Step>"])
writer.writerow(["<Action>ImportASCII</Action>"])
writer.writerow(["<Param>C:\\UA\\Data\\All Futures\\.mck1\\" + row[0] +"</Param>" ])
writer.writerow(["</Step>"])
##writer.writerow("This is beautiful")
The command lines for AmiQuote are documented in the ReadMe2.html files that comes with the installation in the same folder of the application. You can read it also selecting the "Read Me" item from the "Help" menu item of AmiQuote.
In that document only 3 parameters are listed:
From version 1.61: /download - triggers automatic download on start /close - causes automatic close of AmiQuote after download From version 1.20: /autoupdate - this option forces AmiQuote to perform automatic update procedure without user intervention
(In a similar way, also the command line options for AmiBroker are documented in its ReadMe.html file. For more see this)
Personally, I prefer to use a .js script (switching databases, data sources, and date range) to control AmiQuote.
I launch it every day from the Windows Scheduler, but for sure it is possible to use a similar script also invoking it from and ExecuteAndWait AB's batch command.
As @beppe suggested, you should definitely use ExecuteAndWait because this function was especially designed to handle situations like yours.
Just for the reference (I don't recommend using ThreadSleep() this way) but in general, if you need to repeat some function 20 times, instead of this (example from your code above):
If you want to execute some part of the formula only once during Exploration, you don't necessarily need to run your Exploration on 1 Symbol and 1 recent bar(s). You can use: if (Status("StockNum") == 0) for that. For example:
Hi again,
one last thing I would like to see addressed:
for the ASCII import in batch analysis to work, one needs to select each data-file individually and hack it in the batch file.
If stocks keep changing names, or files are added/removed in the data directory, the ASCII import won't work reliably.
In short, I'm looking for a way (not that I need it now, but I can think other people do) to batch import ASCII "all files in directory "X:\PATHTOYOURDATA"
There's one solution I haven't tried, but it may work:
batch file1 that executes .py script1
script1 checks and reads all data files in "X:\PATHTOYOURDATA" ending with .example1 and writes for each .example1 file the correct ASCII import statement for AmiBroker in another batch file2
Execute batch file2 to import the data and do whatever else there needs to be done
Any thoughts on this? Is there a faster/better/more efficient way?
.
