Strange crashing

The program the causes the crash has only 1 fopen() and 1 fclose() statement both outside a loop… so it seems pretty straight forward … Also fopen() is wriiten as fh = fopen( Sourcename, “r”, True ); … but

A couple of questions:

  • Can I monitor within AFL the number of handles that are open - how?
    if not within AFL, Is there another way to monitor handles?

ara1,

Do you still get access violations if you use #pragma maxthreads 1?

I have a simple scan that causes access violations with more than one thread. Unfortunately I am using a third-party plugin whiich makes it harder to tell what is happening. My AFL code does not use fopen.

-Alan

Yes, for example “R” 3rd party plugin is not multi-threading safe. Setting maxthreads helps only if you have race condition in 3rd party plugin, but it won’t help if plugin does not work with non-UI threads.

Alan

Yes I still get a crash … but thanks for the suggestion

Ara

Tomasz

I did some more work ... here are the results

  1. I got processexplorer. As far as I can make out there is only 1 file handle open. Below are a couple of images I captures from a video

  2. I limited the ASCII file read loop to process only a fixed number of lines. Under 205 read operations it did not cause any problems. Sometimes it ran successfully up to 500 to 600 lines read without problems. My file has 1212 lines, so it always fails if I process entire file.

  3. The failure occurs when this statement is executed: VarSet_test1 = VarSet( Symbol, Value_Num );
    The test for successfull operation always shows success. I also cheched Value_Num for Null values. No problem.

  4. AFL code attached. Problem on line 155.


//File: Create Performance WatchList
//version 2.0
//June 28, 2017
//
//This program reads specified ASCII file with performance data from prior backtest, 
// - Prior backtest data created from "UTIL - Scan - Create Performance ASCII file"
// - ASCII filename is "TradableEquities - " + Date_string + "-Performance.txt";
//
//Performance evaluation:
// 1. Remove all issues with negative returns
// 2. Remove all issue with below average returns
//
//Program Flow
// - Read ASCII file
// - Calculate average P/L
// - Process data and create 2 watchlists:
//   - With profit greater than zero				// AK-HighPerf-zero
//   - With profit greater than average profit      // AK-HighPerf-Avg
// - Create statistics.
//
//Use Dummy watchlist as source in order to process data only once
//
//
//#pragma maxthreads 1 
//
//Specify Source File
Date_string 	= StrReplace( NumToStr( DateNum(), 1.0 ), ",", "" );
Filepath  		= "G:/DataFiles/Daily Backtest Summary/";
Filename  		= "TradableEquities - " + Date_string + "-Performance.txt";
//
SourceName	= Filepath + Filename;

//
if( Status( "action" ) == 3 AND Status( "stocknum" ) == 0 )
{
//TargetList	= "AK-Performing Issues";
//

//
    Target1Name = "AK-Perform-Zero";		// Issues with profit > 0
    Target2Name = "AK-Perform-Avg";		// Issues with profit > Average
    Target1Num  = CategoryFind( Target1Name, CategoryWatchlist );
    Target2Num  = CategoryFind( Target2Name, CategoryWatchlist );
//
    StaticVarSet( "Target1Num", Target1Num );
    StaticVarSet( "Target2Num", Target2Num );
//
    Target1List   = CategoryGetSymbols( CategoryWatchlist, Target1Num, 0 ) ; // Get ticker symbols in CSV format
    Target2List   = CategoryGetSymbols( CategoryWatchlist, Target2Num, 0 ) ; // Get ticker symbols in CSV format
    _TRACE( "WL_Utilities-1 - Target1Name " + Target1Name + "  Target1Num = " + Target1Num + "  List1 " + Target1List );
    _TRACE( "WL_Utilities-2 - Target2Name " + Target2Name + "  Target2Num = " + Target2Num + "  List2 " + Target2List );

//
//
//Remove Symbols from Target1 Watchlist
    for( i = 0; ( sym = StrExtract( Target1List, i ) ) != ""; i++ )
    {
        _TRACE( "WL_Utilities-3 remove symbol - i " + Writeval( i, 1.0 ) + "  sym = " + sym ); // + "  List " + Targetlist);
        CategoryRemoveSymbol( sym, categoryWatchlist, Target1Num );
    }

//
//Remove Symbols from Target2 Watchlist
    for( i = 0; ( sym = StrExtract( Target2List, i ) ) != ""; i++ )
    {
        _TRACE( "WL_Utilities-4 remove symbol - i " + Writeval( i, 1.0 ) + "  sym = " + sym ); // + "  List " + Targetlist);
        CategoryRemoveSymbol( sym, categoryWatchlist, Target2Num );
    }

//
} // end if Status

//
//  =============================================   LOOK  BELOW THIS LINE  =====================================
Num_symbols		= 0;
//
_TRACE( "WL_Utilities-x0 - SourceName " + SourceName );

//
if( status( "action" ) == actionScan AND Status( "stocknum" ) == 0 )
{
//
  //  files			= fdir( filepath + "*.*", 1 );			//  ZZZZZZ???
  //  _TRACE( "WL_Utilities-xdir - " + files );

    line			= "startline";
    len 			= 0;

//
    fh		= fopen( Sourcename, "r", True );				// Open file from disk
    _TRACE( "WL_Utilities-x1 - fhread " + fh );
//

    if( fh )
    {
        do
        {


            if( Num_symbols == 0 )											     // Forces to read first line of file
                line	= fgets( fh );											// Read first line from from disk file. Subsequent lines are read later in program

            num_symbols++;													// Count symbols read
            _TRACE( "WL_Utilities-7A - start of do loop - Num_sumbols " + WriteVal( num_symbols, 1.0 ) );

            len		= StrLen( line );											// one based. Counts new line character

            _TRACE( "WL_Utilities-7B - line " + line + "  len " + Writeval( len, 1.0 ));

            char			= "X";												// Dummy character filler
            count       	= 0;

            while( char != " " AND len > 0)										// Find space before value
            {
                char	= StrMid( line, len - 1, 1 );							// zero based - get last character
                len--;
                count++;
                _TRACE( "WL_Utilities-7C - len " + WriteVal( len, 1.0 ) + "  count " + WriteVal( count, 1.0 ) + "  ASCchar " + Asc( line, len ) + "  char " + char );
            } // end while char

            //_TRACE( "WL_Utilities-8A - count " + WriteVal(count,1.0));
            //_TRACE( "WL_Utilities-8B - len " + numtostr(len,1.0));
//


            Value		= StrMid( line, len - 0, count - 1 );					// Extract value from line
            Value_len   = StrLen( value ) - 1;
            charpos		= StrFind( line, " " );									// Find first blank after symbol - OK
            symbol		= StrMid( line, 0, charpos - 1 );						// get symbol                    - OK
            symlen		= StrLen( symbol );
            //
			_TRACE( "WL_Utilities-8A - Symbol " + Symbol + "  Val len " +  Value_len + "  Value " +  value );
//
            for( j = 0; j <= Value_len - 0; j++ )									// Verify ASCII codes for Value string
            {
                _TRACE( "WL_Utilities-8B - Symbol " + Symbol + "  Val len " + WriteVal( Value_len, 1.0 ) + " j " + WriteVal( j, 1.0 ) + " char " + StrMid( value, j, 1 ) + "  ASC " + Asc( StrMid( value, j, 1 ) ) );
            }

            //
            _TRACE( "WL_Utilities-9A - Vallen " + WriteVal( Value_len, 1.0 ) );
            _TRACE( "WL_Utilities-9B - symlen " + WriteVal( symlen, 1.0 ) );
            _TRACE( "WL_Utilities-9C - symnum " + WriteVal( num_symbols, 1.0 ) );
            _TRACE( "WL_Utilities-9D - symbol " + symbol );
            _TRACE( "WL_Utilities-9E - value  " + value );
            _TRACE( "WL_Utilities-9F - value  " + value  + "  symbol " + symbol + "  numsym " + WriteVal( num_symbols, 1.0 ) );

			if (Symbol == "")												// Test for empty Symbols - they cause problems with VarSet()
			_TRACE( "WL_Utilities-9FX - line  " + line  + "  symbol " + symbol + Asc(Symbol,1));
			//
//Assign values to dynamic variables
            Num_symbols_str = numtostr( Num_symbols, 1.0 );
            Value_Num		= StrToNum( Value );
         //   if (Symbol != "CCI")
         //   {
            VarSet_test1 = VarSet( Symbol, Value_Num );
            VarSet_test2 = VarSetText( Num_symbols_str, symbol );
            Value_Test	 = IsNull(Value_Num);
			_TRACE( "WL_Utilities-9G - VarSet test-1 " + WriteVal(VarSet_test1,1.0) + "   test-2 " + WriteVal(VarSet_test2,1.0) + "  Value Test " + WriteVal(Value_Test,1.0));
		//	}
//
            line	= fgets( fh );												// Read file from disk. First read is done at start of loop. This allows check for empty line
            _TRACE( "WL_Utilities-9H - after fgets() at end of code - len " + StrLen(line) +  "  "  + Line);

        }
        while( ! feof( fh ) and Num_symbols < 700); //AND line != "" );  // not oef

        //
        fclose( fh );

		_TRACE( "WL_Utilities-10A - file closed - out of loop " );
        //
    } // end if (fh)
    else
        _TRACE( "WL_Utilities-10B - fh " + WriteVal( fh, 1.0 ) + "  File " + SourceName + " Not found" );

//
}// end if status

//
//
//Calculate Average value of returns
Last_Value	= 0;
Avg_Value	= 0;

//
for( i = 1; i <= num_symbols; i++ )
{
    loop_num	= NumToStr( i, 1.0 );
    Symbol		= VarGet( loop_num );
    Value		= VarGet( Symbol );
    Cum_Value	= Last_Value + Value;
    Last_Value 	= Cum_Value;
    Avg_Value	= Cum_Value / i;					// Use as threshhold to eliminate poor ferforming issues.
//
    _TRACE( "WL_Utilities-11 - Numsymbol " + WriteVal( i, 1.0 ) + "  symbol " + symbol + "  Value " + WriteVal( Value, 1.2 ) + "  Cum " + WriteVal( Cum_Value, 1.2 ) + "  Avg " + WriteVal( Avg_Value, 1.2 ) );
//
}

//
Target1Num		= StaticVarGet( "Target1Num" );
Target2Num		= StaticVarGet( "Target2Num" );

//Create new Watchlists with low performing issues removed.
for( i = 1; i <= num_symbols; i++ )
{

    Symbol		= VarGet( NumToStr( i, 1.0 ) );
    Value		= LastValue( VarGet( Symbol ) );

//
    if( Value > 0 )														// Select issues with positive returns
    {
        CategoryAddSymbol( symbol, CategoryWatchlist, Target1Num ) ;
        _TRACE( "WL_Utilities-12 - symbol " + symbol + "  Value " + WriteVal( Value, 1.2 ) + "  Avg " + WriteVal( Avg_Value, 1.2 ) );
    }

//
    if( Value > LastValue( Avg_Value ) )								// Select issues with above average returns
    {
        CategoryAddSymbol( symbol, CategoryWatchlist, Target2Num ) ;
        _TRACE( "WL_Utilities-13 - symbol " + symbol + "  Value " + WriteVal( Value, 1.2 ) + "  Avg " + WriteVal( Avg_Value, 1.2 ) );
    }

//
} // end for loop



You may have symbol that contains invalid characters or spaces or whatever. Valid variable name
MUST begin with letter (a-z only) and subsequent characters must be letters (a-z only, no other chars, digits (0…9) and underscore _). Attempt to create variable (via VarSet) with ANY other character than mentioned above leads to errors.

If that does not help you may run this in Analysis->Old Automatic Analysis and see if that helps. Or you can try 32-bit version.

Tomasz, I tried all of your suggestions. All symbols started with an alpha character. Some symbols contained a period (such as BF.B) … Got rid of all of those … no difference.

Also Old Analysis and 32 bit system did not resolve the problem.

I finally verified the VarSet() crashing problem …

Code was VarSet(symbol, value);

Some symbols contain a “.” character that was causing the problem. such as VarSet(AB.X, value)

Why customers have this peculiar talent of shooting themselves in the foot?

What told you to use “.” in the variable names???

It is written quite clearly in the manual, right in the very beginning:
http://www.amibroker.com/guide/a_language.html

Identifiers are arbitrary names of any length given to functions and variables.

Identifiers can contain the letters (a-z, A-Z), the underscore character ("_"), and the digits (0-9). The first character must be a letter.

AFL identifiers are NOT case sensitive.

Tomasz,
The names of variables with “.” are not my choice. These are stock symbols that contain "."
This forces me to pre-screen stock lists before I do any processing.

Ara

You don’t need to prescan. You just need to replace or remove forbidden characters.

str = StrReplace( str, ".", "_" );

Tomasz,

The issue of VarSet( “X”, value) crashing was discussed previously … essentially that the first argument has to start with an Alpha character and contain alpha, digits and “_” only.

That was part of the problem.

I just found that VarSet( Symbol, Value) crashes when symbol is ONE character.
I created an ASCII input file with a few one character symbols and it crashed on first symbol.
I tried VarSet( “X” + Symbol, Value) and it worked OK.

Ara

If you do things like that VarSet(“C”, 0) then you are asking for trouble. VarSet is too powerful to be used without thinking. This is high-voltage function.

1 Like

Tomasz, sorry this response is not helpful to me.

What do you suggest we do with symbols that have only 1 character.

Ara

Use Prefix so it does NOT collide with built-in variables or other variables! It is obvious. Never ever use uncurated names as variable names. You can’t expect program to do all thinking for you. It is explained in the manual what variable identifier can be
http://www.amibroker.com/guide/a_language.html (scroll to “Identifiers”).

You either have POWERFUL program for THINKING people or TOY that protects you from everything and does not allow to do anything.

If you take a hammer you can use it to “nail the problem” or hurt yourself depending whenever you think what you are doing or not.

If you expect me to wrap all hammers with foam it is not going to happen because it will block ability to use the hammer at all

2 Likes