How do I know this is the last symbol to be proccessed?

When we code for [Explore] or [Scan], could use

if( status("stocknum") == 0 )

to know "this is the first symbol to be proccessed".

Is it a way to know "this is the last symbol to be proccessed" ?

Thank you very much :smiley: !

Not in Multi-thread mode.

@alexlin this topic has been already discussed on the forum. See this thread:

1 Like

(sorry for the deleted post)

Actually there is another method. It consists in using a static variable to count how many jobs were run and subtract 1 at the end of your script. Then you only have to check this number...in a multi-thread safe way :wink: Of course you need to know how many symbols should be processed at first.

Here is a sketch:

// Start of script
if (Status("stocknum") == 0) { // other jobs will start only after this one ! cf doc
	watchlist = ""; // ADAPT ! eg GetCategorySymbols(categoryWatchList, 0) whatever
	total = StrCount(watchlist, ","); 
	StaticVarSet("countdown", total); // reset countdown
	// possibly other initializations 
	// ...
}

// Your script
// ...

// End of script, AFTER your job, check how many done
StaticVarAdd("countdown", -1);
// Post-processing part, will be run by ONLY 1 thread 
if (StaticVarCompareExchange("countdown", -1, 0) == 0) { 
// LAST 
	// Post processing here
}
// Note: don't do anything else outside of the previous "if ()" statement
// PS: be careful with name collision of 'countdown' var, use one per script

Note there is no guarantee that the post-processing is performed while processing the last symbol in the list, it's completely independent. But I believe it's what @alexlin was looking for. Of course this method looks a bit cumbersome and should be avoided every time you can use Batch jobs.

1 Like

Thank you all @Milosz, @alligator :smiley: ,
I don't know whether the running order
follows a ordinal sequence in MULTITHREADING execution.
= 1 operation * 1 symbol = 1 thread;

If every ticker takes a thread, then it is impossible to confirm
which ticker must be the very final one.

In above case, we can only follow the Tomasz, as @Milosz
replied above.

But if on ordinal one by one process,
we can use some code like @alligator replied,
I've re-arranged it as below:

if (Status("stocknum") == 0) { // for the first ticker only
	tickers = CategoryGetSymbols( categoryWatchlist, 1 );
	tickersCount = StrCount(tickers, ","); 
	StaticVarSet("final_stocknum", tickersCount - 1); 
	// you can do some thing for the first ticker only here …
}
// you can do some thing else here ...
final_stocknum = StaticVarGet("final_stocknum");
if (Status("stocknum") == final_stocknum) // for the final ticker only
{
	// you can do some thing for the final ticker only here …
}

Sorry but I think you don't understand. Let's try to figure it out what's the problem here. Basically you've two choices. Either you run your exploration in a single thread using the directive

#pragma maxthreads 1

and all symbols are processed one after the other (what you call ordinal sequence). Or you run it in a multi-threading context (default, recommended) and there's no order anymore. Actually in that case, you know when a thread is started (alphabetical order)
https://forum.amibroker.com/t/portfolio-backtesting-with-symbol-sequence/3852/3
but you can't know in which order they will finish. E.g the thread running the task for the second symbol can finish after the thread running the task for the third symbol.

Generally what one wants to achieve isn't something for the last symbol in the ticker list, but post-processing, that is doing something after all symbols have been processed. It's quite different. As there is no guarantee that the thread running the last symbol will finish AFTER all other threads EVEN IF all other threads were started before, you can't use Status("stocknum") to know when to perform this post-processing. It's plain wrong.

That's why I proposed the solution above, intended to be run in the multi-threading context not in a single-thread one (in that case it's trivial, see start of this post) as using a critical section wouldn't make sense. This kind of trick were useful when Batch processing wasn't an Amibroker feature. Now it should be avoided if you can. I mentioned it for completeness but maybe I shouldn't have.

If for some reason you need this kind of trick, please use as it is, order is important (what to do at first and last, when to decrease...) and don't discard the critical section, the StaticVarCompareExchange() test.

Batch is the way to go in most if not all cases imo. Pure and simple.

I hope it's a bit clearer now, multi-threading can be puzzling at start, it's the kind of topic requiring clear thinking. Good luck and keep up the good work.

PS: of course if you ever want/need to use my proposal, you should correct counting tickers in the previous code, adding 1.

tickersCount = 1+StrCount(tickers, ","); 
3 Likes

@alligator thanks for the reply and the code!

It's an interesting topic, because although I agree, that using Batch is the safest solution in this case, batch functionality has some pros and cons when comparing to a simple Exploration or Scan.

I have already tested a very similar solution to yours and I'm still not sure if such code is 100% reliable. Usually it works properly, but in some cases (high load and many threads working in parallel) it might pose some problems. As Tomasz said in other thread:

... in MULTITHREADING execution there is no such thing as “last” symbol as there can be N “last symbols” executing in parallel.

Maybe I simply don't understand how StaticVarCompareExchange work in details, but if I'm correct, you are not guarenteed to obtain a semaphore every time you check when working in a multithreaded environment. If that is true and your formula doesn't account for that (because there's no waiting and retrying included - as in the second example from: http://www.amibroker.com/guide/afl/staticvarcompareexchange.html) the outcome might not be always reliable. Please correct me if I'm wrong.

Best :slight_smile:

1 Like

Hello @Milosz,
you're welcome. I understand you use/need a similar solution and to be honest I didn't rewrite my old scripts so I'm in a similar situation but I've changed my workflow and for new tasks I prefer using batch processing.

About StaticVarCompareExchange: using a semaphore in that case is unnecessary. we only need to be sure that only one thread can run the post-processing, and as the code is the same for all threads, it means that only 1 thread should get a 0 when it reads countdown. It's exactly what is achieved there by StaticVarCompareExchange. The thread that reads a 0 replaces the value (atomically, together with test, that is it cannot be interrupted by other threads) by -1, so no other thread reading countdown after can run the post-processing. The main advantage of this solution is that it is quite simple and entirely wait-free.

It's unimportant which thread gets a null countdown, the only thing that matters is that countdown can only be equal to zero after all tasks were performed (because countdown is only decreased after each one).

This idea is very similar to some implementation of synchronization routines in concurrent programming. Maybe I miss a detail in how Amibroker performs exploration or in static variables implementations but it should be correct.

PS: more about this topic here Compare Exchange. Concurrent programming is a vast topic.

Regards

2 Likes

Thanks a lot for the detailed reply. I appreciate it!

Although @Tomasz is currently busy dealing with the recent Yahoo website changes and has already replied to similar topics (links in this thread), let's hope He finds a moment to comment on the correctness of the above solution.

Regards.

2 Likes