What is the procedurally correct way to force data alignment to deal with data holes when simultaneously using pad and align, foreign and current that are all completely different tickers?
For instance, if I were to calculate the BPI (non-P&F) of the S&P 500 using ^GSPC as the pad and align symbol while using a loop to foreign call each of the S&P 500 watchlist components to check for the bullish or bearish condition -- and not even using any StaticVarAdd or Set -- the results will be completely nonsensible if either the current ticker the project happened to run upon (or the first ticker in a filtered watchlist) as stocknum==0 has less data than ^GSPC. Since the current ticker is completely unsettable when saving a project, it seems that it will be luck of the draw one way or the other (suppose you filter by the S&P 500 component watchlist instead of using current or another watchlist and then the S&P added a new ticker starting with A in the future unbenowst to you since the watchlist is automagically updated...)
WRONG (with ^GSPC pad and align ON, run as A current or in S&P 500 watchlist):
WRONG (with ^GSPC pad and align ON, run on AABA as current):
CORRECT (with ^GSPC pad and align history truncated or OFF, run on A current):
I'm looking for a robust solution here to eliminate the problem once and for all. So far I'm leaning towards just wiping the entire historical database each time before downloading historical data. Not just to be assured of date alignment for the vast majority of symbols, but also to deal with what happened during the one-off holiday last Wednesday (18/12/05) when bogus historical data for that day was being provided that wasn't subsequently overwritten by a later historical download.
I've read most all of the forum or KB articles about data holes (both now and over the years), but they really do not apply to alignment issues (which I assume data holes are a derivative of) and are just waxing theoretical. I'm looking for practical solutions other than wiping the database (even though alignment didn't really help with AABA as shown above).
Sorry, but lack of answer means that your posts are totally unclear.
Data alignment works perfectly for everyone
It is totally unclear why you think it is wrong for you. I think that you just don't understand how it works. Alignment is meant to bring ANY data in sync with reference symbol defined in Analysis Settings. And this is exactly what happens.
Foreign function synchronizes the data file
you are referencing with the currently selected symbol.
Synchronization makes sure that EACH bar of FOREIGN data
matches exactly with each bar of currently selected symbol.
So if DateNum() function returns 990503 for given bar
then Close array represents the CLOSE of currently selected symbol at May 3, 1999
and Foreign("SYMBOL", "C") represents close of foreign symbol at May 3, 1999 TOO. This is absolutely necessary because otherwise you won't be able
to do ANY meaningful operations involving both selected symbol and foreign symbol.
This also needed for the display so when you mark the quote with vertical
line it will always match the date displayed regardless if you use Foreign or not.
Please note that if you have data holes in currently selected symbol then in order to synchronize bars Foreign function will remove bars that exist in Foreign symbol but do not exist in currently selected symbol.
Pad and align essentially does the same thing with one exception that "reference" symbol is the one defined in the settings.
There are two kind of unalignment:
missing data bar (aka. "holes") - the bar exists in reference symbol but does NOT exist in selected symbol
extra data bar - the bar exists in selected symbol but does NOT exist in reference symbol
Missing data bars (present in reference symbol) are filled (padded) with previous known bar value. If there is no previous known bar, it gets filled with Null. Extra data (not present in reference symbol) are removed.
Use Tools->Data Purify to check your database for data errors.
Your formulas don't generate the results that you expect because a) you have something like 700 bars of holes in your data, that get filled with "last known" value and if you use StDev on CONSTANT you will get zero.
Thanks. Yes, I know it was difficult to explain because it seemed three layers deep. I was not aware that Foreign truncated to the currently selected symbol, assuming the "reference" settings symbol when turned on was always uber alles. Hmm!
Bottom line -- and correct me if I'm wrong -- I think you need to put in a feature to hard code the currently selected symbol when saving a project (or some other method) because any data holes/extra data in the currently selected symbol will cause a project to return nonsense results when calling a Foreign/SetForeign symbol and/or the "reference" settings symbol that also has data holes/extra data vs currently selected symbol, even if no processing is ever being done to the currently selected synbol (e.g. looping through a watchlist). It appears uncontrollable as to what the currently selected symbol will be when a project is run -- seems to be the last one that was being viewed when Amibroker was last closed. This is very frustrating as I cannot rely on automation working 100% not knowing all conditions stay exactly the same from launch to launch. I mean, its not really a realistic option not to use Fixup=True and pad and align. (If it matters, I'm using 6.20.1 and jscript).
@TraderX, as far as I understand you want to avoid the reset to the "last viewed" symbol when AmiBroker was last closed.
One way to ensure that AmiBroker will always start with the same symbol selected in the charts is to DISABLE the Auto-save layouts in the Preferences dialog:
Define your layout, load your "reference" symbol in the open charts, save the current layout manuallyas default and you are done.
Obviously, in this mode, when you make some changes to your layout, you should remember to save it again before exiting the application.
Please review the last section of the linked document to learn more about layouts that I consider a very useful feature of AmiBroker.
AFAICS, there is no reason to deactivate saving layouts.
Just activate symbol lock on one of your charts (e.g. last chart tab).
That setting will be saved to layout.
Then when restarting AmiBroker (since I presume you start it via automation, see your below quote) the next automated analysis will open with selected locked symbol of last chart because AB always starts with last chart tab selected if it (AB) gets (re-)started. And that (empty) last chart tab would have "^GSPC" being selected one and being locked (again, if symbol lock being activated). So automated analysis would start with "^GSPC" being selected symbol too.
There is an even better option:
Use Batch feature. It lets you set selected analysis symbol before each analysis run.
Just realized that fixing the current symbol run-to-run doesn't quite solve the other problem of where there is data holes/extra data in the symbol matching Status( "stocknum" ) == 0. So my solution is to add the fixed current symbol to the watchlist in position 0 and then exclude it when performing the calculations:
for( i = 0; ( symbol = StrExtract( List, i ) ) != ""; i++ )
{
SumCount += symbol !="^GSPC" AND InTrade(symbol);
}
Yes, here. Your code makes no sense.
And if you iterate through WL from selected chart symbol (e.g. "^GSPC") or applied "Current" symbol ^GSPC of analysis then that one has stocknum 0 already.
BTW your Intrade function is unknown. So who knows what it does.
No one knows your problem. Speak English with actual detailed example problem having occured but not Martian brainwaves language. No one "speaks" the latter one here.
Solution is batch as described above. You can add multiple tasks to single batch. First add your Scan/.../Backtest/... (of separate project) applied on current symbol ^GSPC (being set via Batch task, see picture above). In that 1st project (added to batch) you do your whatever AFL iteration (project applied to "Current"). Save that project run's AFL result to static var if you need result in other AFLs. Do add other tasks (such as backtest or whatever with different "Apply to" setting and that different task(s) calling the static var) to the same batch. Then run the batch (if needed to be automated).
Any proprietary code I'm using has nothing at all to do with these data alignment sandtraps. InTrade is simply a flip function.
This latest consideration is only an issue when specifically scanning over a watchlist filter vs Current, typically the same watchlist being looped through in the code. When using a watchlist, Stocknum = 0 can't be guaranteed to equal what is the current symbol set in JS, Batch or symbol lock, It must also be included in the watchlist and also in the first position. Capisce?
You don't seem to understand my previous post or have not read carefully.
Your upper loop code example is supposed to iterate from single symbol which is set by Batch. It does not iterate all the time on all symbols of watchlist. What sense does it make to iterate it on entire Watchlist n-times (n -> number of watchlist symbols)? You only want it to iterate when ^GSPC is set (n = 1).
So what's the problem? None.
I said to you that in your batch you would have to use (min.) two (separate) analysis projects but not just one. First one makes the iteration from current symbol (e.g. ^GSPC set by batch) and saves result to static var, for example. So in your case SumCount saved to static var. Next project does whatever other analysis but this time applied to your Filter -> Watchlist and calling that static var (SumCount) of loop iteration of project of before.
No stocknum == 0 required here.
First project guarantees that always ^GSPC is applied for your loop iteration. That first project has "Apply to" setting set to "Current". If there is "Current" set then you do not need stocknum == 0 (there is only single symbol set). The second project has "Apply to" -> "Filter" (n-symbols now).
I understand, but why would I want to run code twice for each project when it can be done all in one pass using Stocknum == 0 and exclusion? I run a lot of projects where doing so would be an added complication (and for that matter, the Batch interface is not practical enough for my needs). Is StaticVar guaranteed to be always be aligned to the current symbol of the first project (which would be ^GSPC) without exception when called elsewhere? Even when called from a current symbol or watchlist symbol with drastically shorter data than ^GSPC?
The iteration is not run twice but once.
The looping code is part of analysis 1 only.
The AFL is split up to two AFLs of two projects.
The first AFL (after the iteration part) does not have code that is required in second project's AFL only. Why would you add such code to first one?
The first one only goes till iteration and saves result. Then that's its last line.
You call result of first analysis in second one.
So why would you iterate in second one again? Makes no sense.
If you have settings or code that both projects need then create one single include file. Both use that same file with same code via #include line put at the top to both AFLs. Or even better and simpler... the code til the loop end is put to include file and only second projects AFL has #include line at the top. The include file is part of your first analysis project. In order to ensure that the looping code is only executed when ^GSPC is available you just add:
// code that both AFLs need above
// part of include file
if ( StrMatch(Name(), "^GSPC") ) {
// your loop for analysis1 project here
//At the end of the looping part
StaticVarSet( .... );
}
// end of include file
The whole thing is done this way only because you want to automate it. You want to ensure having ^GSPC available for your looping part or whatever. So if you want to automate and ensure ^GSPC as current symbol for certain code then this is the safest way to go, IMO.
I don't see much complication.
Analysis1.apx does job similar to an include file.
Do you consider Include file put to top complication too?
Read documentation of static variables.
There isn't a problem in your case.
^GSPC is your master symbol.
You use same Interval for other symbol and it is EOD (time based).
Yes, it is aligned if symbol has shorter BarCount.
So "safest way" is the operative phrase here. Thank you for your help.
The "complication" is that each project at present would need its own custom looping code to run separately as a project before the results can be called by main code (and within the same session) since any looping code would not be providing universal persistent vars to majority of the other projects. Since project files do not contain more than one AFL, it would require additional JScript code and up to twice as many project files to manage.
I didn't see it in the help file, but is it possible to load and then execute the Batch interface files via OLE/JScript?
Those looping codes (all dependent on ^GSPC?) could be put all to Analysis1 project's AFL by using #include command. And via switch each batch uses the appropriate include file (for your looping code to be executed).
So it means you would just need single Analysis1.apx for all your main projects.
So what I have in mind in my big brain would be something like this
Analysis1 project's AFL (let's call it Analysis1.afl) would look like this for example
// THIS code below would be part of Analysis1.afl of Analysis1.apx
// read here what it is about:
// @link https://forum.amibroker.com/t/triple-trouble-data-alignment-data-holes-in-analysis/10264/18
// Code for reading LAST line of batch.log file HERE
// (that line was written by batch command "Write text to log" before (see picture below))
// ....
run_what = /*your string read from batch.log file*/;
if ( StrFind( run_what, "Run Project1" ) ) {
// below AFL being part of Include folder
// code executes loop iteration (based on selected symbol ^GSPC)
#include <_GSPC_project1.afl>
else if ( StrFind( run_what, "Run Project2" ) ) {
// below AFL being part of Include folder
// another code executing different loop iteration (based on selected symbol ^GSPC)
#include <_GSPC_project2.afl>
}
So you always deal with just one single Analysis1.apx project file.
The batch's first task "Write text to log" decides which #include file of Analysis1.AFL will be executed. It would write to batch.log file's last line which #include command's AFL of Analysis1.apx is to be run during the batch process.
See picture below
OK?
To me it seems you just think too complicated.
You do not need OLE etc.
You just need to call via command line.
Broker.exe /runbatch "Path_to_Batch_file.abb"
So using a .bat file add line like this, for example
