User error, was: Using mutex to protect AFL function calls

IBController
1

Hello,

I hope someone can help me.

I'm using a mutex to protect critical areas of my AFL code. This is to ensure that AFL calls that invoke IBC functions are protected when called from different threads, eg when displaying multiple symbol charts.
I've written the following code to reserve and free the mutex, based on code written by Tomasz.

function _GetIbcMutex( mutexName ) 
{ 
  _TRACE("_GetIbcMutex Entry " + mutexName);
  global _ibcMutex; 
  _ibcMutex = ""; 
  // Try obtaining mutex for ten second
  for( i = 0; i < 10000; i++ ) 
  {
    if( StaticVarCompareExchange( mutexName, 1, 0 ) == 0 ) 
    { 
      _ibcMutex = mutexName; 
      break; 
    } 
    else 
      ibc.Sleep( 10 );
  }
  if( _ibcMutex == "" )
    _TRACE("ERROR: Cannot obtain mutex " + mutexName);    
  _TRACE("_GetIbcMutex Exit " + mutexName);
  return _ibcMutex != "";
}

function _FreeIbcMutex() 
{     
  _TRACE("_FreeIbcMutex Entry " + _ibcMutex);
  global _ibcMutex; 
  if( _ibcMutex != "" ) 
  { 
    StaticVarSet( _ibcMutex, 0 ); 
    _ibcMutex = ""; 
  } 
  _TRACE("_FreeIbcMutex Exit ");
}

To test this, I have written the following sample code:

ibc = GetTradingInterface("IB");
for ( i = 0; i < 10 && ( ibc.IsConnected() != 2 AND ibc.IsConnected() != 3 ); i++ ) 
{ 
  ibc.Sleep( 200 );
}

SetChartOptions(0,chartShowArrows|chartShowDates);
_N(Title = StrFormat("{{NAME}} - {{INTERVAL}} {{DATE}} Open %g, Hi %g, Lo %g, Close %g (%.1f%%) {{VALUES}}", O, H, L, C, SelectedValue( ROC( C, 1 ) ) ));

Plot( C, "Close", ParamColor("Color", colorDefault ), styleNoTitle | ParamStyle("Style") | GetPriceStyle() ); 

_GetIbcMutex( "ibcMutex" );
_TRACE("Wait for 200ms");
ibc.Sleep( 200 );
_FreeIbcMutex();

I insert this code in a chart pane on Sheet 1.
On Sheet 2 I have another chart pane which displays volume.
In the test, I switch continuously from Sheet 1 to Sheet 2.
Eventually, a mutex deadlock ocurs, where the switch from Sheet 1 to Sheet 2 occurs while the mutex has been reserved and not yet released.
When I switch back to Sheet 1, the display is not updated, because the mutex is still reserved.

Here is the trace record for each refresh cycle of the AFL code

_GetIbcMutex Entry ibcMutex
_GetIbcMutex Exit ibcMutex
Wait for 200ms|F:\AmiBroker AFL\AFL\Basic Charts\Price.afl|55|8|20:21:25.64|
_FreeIbcMutex Entry ibcMutex
_FreeIbcMutex Exit

When the deadlock occurs, the Sheet switch occurred before _FreeIbcMutex is called.
This causes the next iteration of the code to block waiting for the mutex

_GetIbcMutex Entry ibcMutex
_GetIbcMutex Exit ibcMutex
Wait for 200ms
_GetIbcMutex Entry ibcMutex

Am I doing this correctly?
What can I do to solve it?

Polomora

2

@polomora are you talking about chart sheets or panes? If you really mean sheets, remember that:

Lower tabs (so called “chart sheets”) don’t represent windows. They represent “chart sets” that can be switched quickly but they are NOT physical windows. Instead physical windows are created/destroyed when you select/deselect given sheet (in lower tab). Deselected chart sheet (in lower tab) does NOT exist physically.

I quoted Tomasz from this thread:

Because deselected chart sheets do NOT exist physically, they don't receive any kind of refreshes and the code is not executed. Maybe (only maybe) it has something to do with your case ...

3

Thanks Milosz for the quick reply.

I mean chart sheets, selected using the tabs on the bottom.

The fact that deselected Sheets don't receive updates clarifies what I observed with the logging. No logging occurs for the chart when the Sheet is deselected.
However, am I correct that the problem with the mutex still exists? The mutex was still reserved from the previous time the Sheet was selected. When I returned back to that Sheet, the screen wasn't refreshed.

I've read the topic thread that you quoted in your reply. Does this mean that the Sheet is deselected, hte AFL script is also immediately destroyed? And the mutex in my example never has a chance to be freed?

Polomora

4

fxshrat sent me a message to change the code as shown below:

if( _GetIbcMutex( "ibcMutex" ) )
{
  _TRACE("Wait for 200ms");
  ibc.Sleep( 200 );
  _FreeIbcMutex();
}
else
  _TRACE("Unable to enter CS");

The result is the same. Logging:

_GetIbcMutex Entry ibcMutex
_GetIbcMutex Exit ibcMutex
Wait for 200ms F:\AmiBroker
_GetIbcMutex Entry ibcMutex
ERROR: Cannot obtain mutex ibcMutex
_GetIbcMutex Exit ibcMutex
Unable to enter CS

5

The moment that you deselect a chart sheet, you stop all activity there, so it might be possible that the semaphore could not be reset, but I wouldn't like to draw definite conclusions and leave that to Tomasz or other users more proficient than me.

You can also test it using chart panes instead of chart sheets...

6

First - don't do that. Do NOT use mutexes for IBController. It is not needed and it is bad coding. IBController calls (like any other OLE calls) are serialized (so if one call is pending the other one will wait till first completes) and you don't need mutexes. And you should be using orderIDs to differentiate orders placed with different codes.

Secondly - do NOT use ibc.Sleep. It is forbidden now. It blocks all other OLE calls. You should be using ThreadSleep() function instead. ThreadSleep() is the only allowed waiting function.

Thirdly - when you switch tabs the chart panes in "disappearing" tab are destroyed and new ones is created. Any code calling OLE or waiting for more than 1s will be forcefully terminated without waiting for completion and the rest of code is NOT executed. This is forced by AmIBroker to avoid deadlocking GUI by incorrect user code that implements any kind of busy waiting or OLE waiting. Waiting inside AFL is forbidden except using ThreadSleep().

2 Likes
7

Thanks Tomasz for the reply, especially for confirming OLE calls are serialised.

In my application, I have written AFL functions to prepare the environment for each IBC call, and to check for and handle error cases. These functions use mutexes, because I also log the activity inside the functions. Without the mutex, the logging from simultaneous calls to the same function from different threads is interleaved, which makes debugging difficult.

I use ibc.Sleep(), because the documentation for ThreadSleep() states

Works only from NON-UI threads. When called from UI thread the function does NOTHING and returns immediatelly

Is this still the case? I thought that a UI thread is one where the AFL code uses Plot() and other statements to display charts to the user. Or have I misunderstood?

Polomora

8

All AFL code seems to be executed in its own thread, which means its non-UI.
Apparently each Pane in a single chart sheet (lower tab) would probably have its own thread as well.

Plot(C, "C", colorDefault, styleBar );

ThreadSleep( 999 ); 
ThreadSleep( 999 ); 

PlotText( NumToStr( SelectedValue( BarIndex()), 1, 0), BarCount - 10, LastValue( H * 1.003), colorAqua);

The PlotText() will be very laggy, try using the arrows keys to move the quote marker quickly.

This means that ThreadSleep() does work.
Here, you get the illusion that PlotText() is lagging behind Plot() :slight_smile:

A UI thread could be the one that renders everything after all the formula computation. This would apply to Chart output as a whole and AA UI in terms of all the rows(& columns) with Summary etc.

TJ of course knows exactly.

9

@travick response is almost spot on, with the exception that rendering of plots is also done in separate non-UI thread.

So ThreadSleep() works pretty much always as as long as you don't turn OFF multithreading in the preferences, every AFL is run in separate thread, except one notable exception - custom backtest (2nd) phase.

As for logging, simply prepend ChartID to the log entry and it will be easy to identify which thread it comes from (or even SORT the output by chart ID).

10

Of course I have just scratched the surface but it is really beautiful to try and understand what is happening even though many users may overlook it.

When you purposely slow down an Exploration, you can see how the true UI thread is doing its job and the rows are slowly being added one after the other.
One can also see the Stop or pause execution along with Progress Bar and Thread information.
The whole AB appl doesn't even have to freeze.

image

Again, in normal time, the whole thing complete in 0.17 secs, a blink of an eye :smiley: super fast !!!

image

11

@travick & @Tomasz.

Many thanks for the clarifications and tips. I will try them out later.