Question on error using AmiPy

hi, I have been implementing a bunch of example Neural Network codes from this site relataly. I hope @Kuba or @Tomasz can have a look at a small problem I have explained below.

It works but when I initially start up Amibroker and then press the "Start Forecast Calculation" button (see param window). Then Amibroker freezes up giving me the following output window:

But then I go out of Amibroker (I have to kill it to get out) and start Amibroker back up and then it works. Below you see that when I press the button "Start Forecast Calculation" it will show the "in-training predictions" (orange) and the forecast (violet). So it works just fine, only it crashes Amibroker just after I start up Amibroker fresh in the morning and press "Start Forecast Calculation" button.

Thank you


the AFL code:

SetBarsRequired( sbrAll, sbrAll );
_N( pythonpath = "C:\\Users\\win 10\\AppData\\Local\\Programs\\Python\\Python38\\mypython\\neural\\relataly\\" );
_N( pythonfile = "" );
PyLoadFromFile( "Excersize2", pythonpath + pythonfile );

calculateforecast = ParamTrigger( "Calculate Forecast", "Start Forecast Calculation" );
trainfactor = Param( "Train Factor", 0.8, 0.1, 1, 0.01 );
nforecast = Param( "Number Forecast Points", 10, 1, 50, 1 );
sequence_length = Param( "Sequence Length", 50, 10, 250, 1 );
_N( startDate = ParamDate( "Start Date", "1-1-2021", 1 ) );
_N( endDate = ParamDate( "End Date", "7-28-2022", 1 ) );
datatype = ParamToggle( "Data Type", "Real|Synthetic", 1 );
resetstaticvariables = ParamTrigger( "Reset Static Variables", "Reset" );

if( resetstaticvariables )
    StaticVarRemove( "prediction" );
    StaticVarRemove( "forecast" );

if( datatype )
    pi = 3.14159265359;
    sinusFunctionArray = 0.02 * Cum( 1 ) + sin( 0.125 * pi * Cum( 1 ) );
    sinusFunctionArray = C;

yearArray = Year();
monthArray = Month();
dayArray = Day();

if( calculateforecast )
    Say( "calculate forecast" );

    PyEvalFunction( "Excersize2", "getDataFromAmibroker", yearArray, monthArray, dayArray, startDate, endDate,
                    sinusFunctionArray, nforecast, sequence_length, trainfactor );
    PyEvalFunction( "Excersize2", "calculateForecastUsingPython" );
    forecastFromPython = PyEvalFunction( "Excersize2", "getForecastFromPython" );
    StaticVarSet( "forecast", forecastFromPython );
    predictionFromPython = PyEvalFunction( "Excersize2", "getPredictionFromPython" );
    StaticVarSet( "prediction", predictionFromPython );

    Say( "finished" );

SetChartOptions( 1, chartShowDates, chartGridMiddle, 0, 0, 0 );
Plot( sinusFunctionArray, "Data", ColorRGB( 3, 157, 252 ), styleDots, Null, Null, 0, 0, 1 );
Plot( IIf( Nz( StaticVarGet( "prediction" ) ) == 0, Null, Nz( StaticVarGet( "prediction" ) ) ), "Prediction", ColorRGB( 249, 160, 72 ), styleDots | styleNoRescale, Null, Null, 0, -1, 3 );
Plot( IIf( Nz( StaticVarGet( "forecast" ) ) == 0, Null, Nz( StaticVarGet( "forecast" ) ) ), "Forecast", ColorRGB( 243, 50, 230 ), styleDots | styleNoRescale, Null, Null, nforecast, -1, 3 );

mask = DateTime() >= StrToDateTime( startDate ) AND DateTime() <= StrToDateTime( endDate );
startmask = mask AND !Ref( mask, -1 );
endmask = !mask AND Ref( mask, -1 );
Plot( mask, "", ColorRGB( 0, 0, 40 ), styleArea | styleOwnScale | styleNoLabel, 0, 1, 0, -3, 1 );
Plot( startMask, "", ColorGold, styleHistogram | styleOwnScale | styleNoLabel, 0, 1, 0, -2, 1 );
Plot( endMask, "", ColorGold, styleHistogram | styleOwnScale | styleNoLabel, 0, 1, 0, -2, 1 );

the PYTHON code:


import sys
sys.path.append('C:/Users/win 10/AppData/Roaming/Python/Python38/site-packages')

import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from keras.models import Sequential
from sklearn.preprocessing import MinMaxScaler
from keras.layers import LSTM, Dense, TimeDistributed, Dropout, Activation
from sklearn.preprocessing import RobustScaler
from sklearn.metrics import mean_absolute_error, mean_squared_error
import AmiPy

def getDataFromAmibroker(year, month, day, startdate, enddate, data_array, nforecast, sl, tf):

    global df
    global df1
    global rolling_forecast_range
    global sequence_length
    global trainfactor
    global mask

    start_date = startdate
    end_date = enddate

    df = pd.DataFrame()

    df['Year'] = year.astype(int)
    df['Month'] = month.astype(int)
    df['Day'] = day.astype(int)
    df['Date'] = pd.to_datetime(df[['Month', 'Day', 'Year']], format='%m%d%Y')

    df['Data'] = data_array
    df['Predict'] = data_array # fill with data array to overwrite later
    df['Forecast'] = data_array # fill with data array to overwrite later
    rolling_forecast_range = int(nforecast)
    sequence_length = int(sl)
    trainfactor = tf

    mask = (df['Date'] >= start_date) & (df['Date'] <= end_date)
    df1 = df.loc[mask]

    df['Predict'] = 0.0;
    df1['Predict'] = 0.0;
    df['Forecast'] = 0.0;
    df1['Forecast'] = 0.0;

def calculateForecastUsingPython():

    # Get the number of rows in the data
    nrows = df1['Data'].shape[0]

    # Convert the data to numpy values
    np_data_unscaled = np.array(df1['Data'])
    np_data_unscaled = np.reshape(np_data_unscaled, (nrows, -1))

    # Transform the data by scaling each feature to a range between 0 and 1
    scaler = RobustScaler()
    np_data_scaled = scaler.fit_transform(np_data_unscaled)

    # Prediction Index
    index_Close = 0

    # Split the training data into train and train data sets
    # As a first step, we get the number of rows to train the model on 80% of the data
    train_data_len = math.ceil(np_data_scaled.shape[0] * trainfactor)

    # Create the training and test data
    train_data = np_data_scaled[0:train_data_len, :]
    test_data = np_data_scaled[train_data_len - sequence_length:, :]

    # The RNN needs data with the format of [samples, time steps, features]
    # Here, we create N samples, sequence_length time steps per sample, and 6 features
    def partition_dataset(sequence_length, data):
        x, y = [], []
        data_len = data.shape[0]
        for i in range(sequence_length, data_len):
            x.append(data[i-sequence_length:i,:]) #contains sequence_length values 0-sequence_length * columsn
            y.append(data[i, index_Close]) #contains the prediction values for validation (3rd column = Close),  for single-step prediction
        # Convert the x and y to numpy arrays
        x = np.array(x)
        y = np.array(y)
        return x, y

    # Generate training data and test data
    x_train, y_train = partition_dataset(sequence_length, train_data)
    x_test, y_test = partition_dataset(sequence_length, test_data)

    # Configure the neural network model
    epochs = 25; # An epoch is an iteration over the entire x_train and y_train data provided.
    batch_size = 1; # Number of samples per gradient update. If unspecified, batch_size is 32.

    # Model with n_neurons = inputshape Timestamps, each with x_train.shape[2] variables
    n_neurons = x_train.shape[1] * x_train.shape[2]
    model = Sequential()
    model.add(LSTM(n_neurons, return_sequences=True, input_shape=(x_train.shape[1], 1)))
    model.add(LSTM(n_neurons, return_sequences=False))
    # model.add(Dense(25, activation='exponential'))
    # model.add(Dense(1))
    model.compile(optimizer="adam", loss="mean_squared_error")

    # Train the model
    history =, y_train, batch_size, epochs,verbose=0)

    # Reshape the data, so that we get an array with multiple test datasets
    x_test_np = np.array(x_test)
    x_test_reshape = np.reshape(x_test_np, (x_test_np.shape[0], x_test_np.shape[1], 1))

    # Get the predicted values
    y_pred = model.predict(x_test_reshape)
    y_pred_unscaled = scaler.inverse_transform(y_pred)
    y_test_unscaled = scaler.inverse_transform(y_test.reshape(-1, 1))

    offset1 = df1.index[0][train_data_len + offset1:df1['Predict'].size + offset1,'Predict'] = y_pred_unscaled
    df['Predict'].loc[mask] = df1['Predict']

    # Making a Multi-Step Prediction
    new_df = df1.filter(['Data'])
    for i in range(0, rolling_forecast_range):
        last_values = new_df[-n_neurons:].values
        last_values_scaled = scaler.transform(last_values)
        X_input = []
        X_input = np.array(X_input)
        X_test = np.reshape(X_input, (X_input.shape[0], X_input.shape[1], 1))
        pred_value = model.predict(X_input)
        pred_value_unscaled = scaler.inverse_transform(pred_value)
        new_df = new_df.append(pd.DataFrame({'Data': pred_value_unscaled[0, 0]}, index=new_df.iloc[[-1]].index.values + 1))
        new_df_length = new_df.size

    new_df_shift = new_df.shift(-rolling_forecast_range)
    st1 = df1.index[df1['Forecast'].size-1] + 1
    df1.loc[st1-rolling_forecast_range:,'Forecast'] = new_df_shift.loc[st1-rolling_forecast_range:,'Data']
    df['Forecast'].loc[mask] = df1['Forecast']

def getPredictionFromPython():
    return df['Predict'].to_numpy()

def getForecastFromPython():
    return df['Forecast'].to_numpy()

1 Like

Call stack is just garbage. It looks like code inside plugin has gone to some uncharted territory or run out of stack? Guessing only.

ok thanks for your reply. The call stack was the only thing I got. I will check again in the AmiPy.log file but there was no error in there, just information about loading files and starting functions as far as I can recall. But will check later.

i tried again this morning after a fresh startup of the computer and it did the same thing. I do not get an error message in the AmiPy.log, it just looks normal. When I then kill Amibroker with the Windows Task Manager and start it back up, then it works.

I however made 6 similar Exercises which I put below different Tabs (see jpg file). They basically load the same Python stuff but I all give them unique names (Excersize1, 2, 3 etc ) when using PyLoadFromFile( "name", "filename ). Not sure if this could have anything to do with it.


I wrote you - the problem happens OUTSIDE of AmiBroker, inside 3rd party code.

You are getting report from AmiBroker because it has global unhandled exception handler that catches ALL exceptions that occur in program and in all DLLs loaded into program. But exception happens INSIDE PYTHON part and I can't do anything about it. The Python codes that you run create many threads on their own and apparently this creates issues with embedded Python (1. Embedding Python in Another Application — Python 3.9.6 documentation) probably due to Python GIL (global interpreter lock) quirks.

ok, thanks for your reply

@empottasch thank you for your examples Spline & Pnf, just so good.

I'm coding eemd with VS code or Spyder .
Unsuccessful attempts to remote or attaching debuggers to the python fscript while running code from AmiBroker. Haven't tried adding dedug code in the python script.

VS code says attached but only shows call stack showing dummy threads, no variables displayed.
Nothing helpful I could get.

Reason for the blurb is a call if anyone has succeeded debugging python script while running app from Amibroker. I've a hunch that embedded & isolated python is diffiicult to look at.

hi @unkufunku thanks for your reply. I have not tried Spyder yet. I am fairly new with Python. I have been coding on and off only for about 6 months or since the plugin has been there. Myself I use the Atom editor together with a program called Kite (both the free version).

I thought all my versions had this problem but I also have a "Multivariate" neural net version and that does not give the error. So maybe indeed there is something in my Python code that it doesn't like. But this "Univariate" code is just for practice. The goal is a working "Multivariate" version. I have that working but now I want to be able to do the calculations in 2 steps:

  1. the training of the model and then save the model to disk.
  2. then load the model back up and do the forecast.

This way when you have a good model you will not have to train it for every forecast. Also you can use it on another computer or inside a smaller database.

I succeeded with this for the Univariate version but not yet for the Multivariate version. But this is such a large topic that I wonder if I will ever master it :smiley: So maybe at some point I will post that Multivariate code as well and maybe there are some people out there with some higher level knowledge of time series forecasting using neural nets and/or Python and we can make a joint effort :smiley:

Is python really "embedded" in AB?

To my mind, embedded means a single process space. The PyEvalFunction context argument makes me imagine the dll forks a python instance for each context.

Is this how things work? Any documentation?

Have u ever got this code to work consistently using the same data?

GIL issue doesn't seem likely. Isn't there some python option to disable parallel processing and check?

tomasz statement it might b stack seems worth looking into. Is stack size static? Which eval call is causing hang? If this is an issue seems likely process is hanging on GetData call. Does reducing data passed help?

No, Python is NOT embedded in AmiBroker application
AmiBroker by itself DOES NOT use Python, does NOT need Python and does NOT know anything about Python and does NOT care about Python at all.

Python is only involved IF AND ONLY IF you install AmiPy plugin.

AmiPy PLUGIN embeds Python (using API that is documented in link I provided earlier).

IF you installed AmiPy plugin THEN it is LOADED into AmiBroker process space and executed. Then Python code that you call using PyEvalFunction works within AmiBroker process space and can crash AmIBroker if it happens to generate unhandled exception. Once control is handed to the DLL, the DLL is free to do anything, including crashing application that loaded it. And no AmiPy does not "fork" anything.

I've been struggling to learn these new approaches and ecosystems.
You're very much leading in this respect, though direction is similar.
I'll post once current project when it's working sufficiently.
Haven't decided/don't know whether to go timeseries analysis or ML NN.
No problem cooperating but realistically not coming to the table with anything much.

This debug info I'm using

plus stackoverflow

I note @Tomasz's comments which clarifies that attaching debugger to python script is doable for the experienced and knowledgeable. I'm don't really understand what he's mentioned, particularly the implications from the link provided. I just don't know what I don't know.
Need to build on the knowledge.

thanks for your reply I am also just a beginner with this. I am currently just implementing lots of examples. There are so many examples out there it will keep me busy for a long time and in the mean time you slowly learn. I found this 1 yesterday which also looks interesting and I am also going to try to implement:

But you gave me an idea to look into. Since I also have Visual Studio Community loaded I will look into it if I can use that. Not sure if I have to install an extension or that it is standalone application but I will figure it out.

I have checked the python code you've provided in my development environment. Unfortunately the code crashes after going through 'external code' that is not debuggable from where it returns with invalid python thread state. I would need to further investigate this case.

thanks for your help. Don't put too much time into it. My multivariate code is working fine. I will post that code tomorrow under a new topic name

i know a senior guy who does ml development as his day job. He told me that python packages are significantly python version dependent. He said he runs several python versions because of this. Next time i talk to him i'll ask for more information RE version dependencies.

My understanding is that ml learners (trainers/fitters) do not always follow the same logic so debugging intermittent issues may be difficult,

My understanding is that learners may not converge. But the ml guy said not to worry about this.

If u don't mind my asking... is the python code in this thread presented as working code somewhere else?

Sorry @Kuba... i forgot to thank u 4 supplying the exception... ::

If u don't mind my asking... is the python code in this thread presented as working code somewhere else? 

yes, I posted the links to the code I used at the top of my Python code. Mostly I used these 2 links:

So the code has been tested outside of Amibroker ran from the Python console. But the code works. It is just an initial problem when I run it after starting the computer. I do not have this with the multivariate version so I will ignore this problem for now.

Multivariate problem is described here. Will post that version later.

i think my crash problem was caused because I also had another instance of Amibroker running. If that was the case I needed to get out of Amibroker and back in again and then it works. But if I only start 1 instance of Amibroker I do not have this problem. So everything is still working great!

that does not explain why kuba was able to recreate the crash

at them moment I do not seem the have any problems (except of course for a fake scamdemic :rage:). Did you try to run the code? I also finished a multi features version that saves the model to disk and can be loaded back up again. But I am still in the process of coding all the examples I can find (in the hope 1 day I will understand what I am doing). And I figured out that not all the people who post code know what they are doing. But I think the guy from the site knows what he is doing. I found another example yesterday of somebody who seems to know what he/she is doing. But his/her approach is different. Earlier examples I seen split the data in a training part and a test part. This one splits the data in 3 parts" training, validation and test. So I have this code running as well and have to dig into this since I do not yet understand why they use a different approach.

example that splits data in training, validation and test part

Anyone here who has a lot of experience with LSTM modelling?