Strange problem when trying to load keras

[empottasch]

hi,

I have a strange problem.

I am playing around with some neural network examples, in particular this 1

univariate NN test

Python is running and working fine but for some reason it has problems with tensorflow when I run it from Amibroker. If I run the code from the Python console everything works fine. And when I run it from within Amibroker on a second computer I also have no problems. It is just on my test computer where it can not find keras.

Only thing that is strange is that when I load keras and tensorflow on the console it is loaded from 2 separate directories ("Roaming" and "Local") on the problem computer and on the computer that works it load both from the "Roaming" directory. So this is the only difference.

>>> ks
<module 'keras' from 'C:\\Users\\win 10\\AppData\\Roaming\\Python\\Python38\\site-packages\\keras\\__init__.py'>
>>> tf
<module 'tensorflow' from 'C:\\Users\\win 10\\AppData\\Local\\Programs\\Python\\Python38\\lib\\site-packages\\tensorflow\\__init__.py'>
>>>

I un-installed Python, then removed all the remaining directories after I removed Python. Then re-installed Python and installed, numpy, pandas, scipy, sklearn, and tensorflow. All have the same versions on both computers. I added all the necessary paths to the environment variables. And other AFL files in which I call Python code works. Just not the tensorflow stuff.

When i type in from the Python console:

from keras.layers import LSTM, Dense

there is no problem. But when I try to load it using Amibroker it says: ModuleNoFoundError No module named keras

But this is only on 1 computer. On the other computer I do not have this problem.

So the strange thing is that it works from the Python console but not when I call it from within Amibroker. Like I said the path information I put inside the "environment variables". All the package files are the same on both computers (and also the same versions). When I type in CMD console of windows

C:\Users\win 10>pip list

it shows the same packages on both computers and same versions. Only thing that is different is the location where everything is installed. I did not do this on purpose, it did this automatically. But since I added these directories to the Path and also because it runs without a problem from the console it should work.

Anyone has an idea?

I know it is probably an impossible question but who knows maybe someone knows

[empottasch]

on the computer where it worked I had http://ev1.pl/amipy/AmiPy_0_4_1.zip

installed. After installing http://ev1.pl/amipy/AmiPy_0_4_2.zip

I had the same problem. So going back to version 0_4_1 resolved the problem.

So not sure what is going on but it seems tensorflow / keras will not load or can not be found when plugin version 0_4_2 is run but it can when 0_4_1 is run.

I will do some more testing if tensorflow / keras is reall loading in 0_4_1 but there is definitely a difference since with 0_4_2 it crashes and not with 0_4_1

I will test if I can run tensorflwo/keras from Amibroker using 0_4_1

[bysoaa]

You could check if it is somehow related to change introduced in 0.4.2:

https://ev1.pl/amipy/ChangeLog.html

It is stated that:

2. Python is now in isolated mode (see Isolated Configuration)

[empottasch]

thanks for the information. Not sure what to make of it. For now I am working with 0_4_1 trying to get some output from tensorflow. In any case with 0_4_1 tensorflow/keras seems to load. It crashes when I am fitting the model even though I put it on verbose. So I am trying step by step to run the code shown in my first post (see link in that post). But I definitely can not use version 0_4_2 at the moment since it will not load tensorflow/keras (unless of course there is some trick I am not aware of yet).

[empottasch]

when I run this code in the Python console it works:

NN code

however calling it from Amibroker it get stuck here:

Step #5 Train the Model
In [12]:
# Training the model
model.fit(x_train, y_train, batch_size=16, epochs=25)

I adjusted that code so that it does not output data using:

    # Training the model
    tst = model.fit(x_train, y_train, batch_size=16, epochs=25,verbose=0)

still there is a problem, not sure what it just crashes. So I will wait a bit for a response from Kuba or Tomasz

[yellowslug]

the ModuleNoFoundError is probably because keras is in one of your personal site-package folders. i ran into similar errors trying AmiPy. i fixed this by making all ~/program files/python38 folders writeable then doing a pip --force-reinstall on all my personal packages. This may be a kludge if there is a proper way please advise.

The post RE isolated configuration supports this... the python man page says this option leaves some site-package folders out of paths. i do not believe amipy uses ur config when it launches itself.

Do u have only one python version installed on all machines?

[empottasch]

hi, yes I only have 1 Python version installed.

So I think my installation is correct since with AmiPy version 0_4_1 I can load tensorflow and keras but not with 0_4_2. It will not load. Other stuff loads with 0_4_2, like numpy and pandas. I was using 0_4_2 successfully but only yesterday started playing with tensorflow and keras.

Are you also working with tensorflow and it works for you?

With respect to installation problems I also encoutered that but you can install from the Administration CMD window and also add --user. So for instance

C:\WINDOWS\system32>pip install --user tensorflow

so if anyone is using tensorflow / keras with 0_4_2 without any problems please let me know. Else I will try support later.

The code I have shown with the link works in the Python console. You can just copy and pate it into the console or put it in a python file and run it from the Windows CMD window. You of course first have to install all the packages like tensorflow, yfinance, numpy etc. all the packages it imports you need to have installed.

So I hope someone is running is without any problems from 0_4_2 and can tell me what the trick is or else I hope @Kuba or @Tomasz can maybe try to run the code (posted in the link) using AmiPy 0_4_2. In 0_4_1 it manages to import tensorflow but I couldn't run the code successfully even in verbose mode.

[yellowslug]

i have numpy, pandas, scikit installed not tensorflow. They work so far.

i still believe the issue is having packages installed under your user(s) appdata. Please check your appdata/.../site-packages folders and verify there is nothing in them.

[empottasch]

yes numpy, pandas and scikit also work for me in 0_4_2

no it is all installed correctly. If it would not be installed correctly I also could not run the code from the Python console window. And I also can run it from the Windows CMD window in the form of a Python file.

You can also try it if you are running 0_4_2

Just install tensorflow using:

C:\WINDOWS\system32>pip install --user tensorflow

and then inside 1 of your Python codes put in the line

import tensorflow as tf

you will see that does not work in 0_4_2. It will not find the module tensorflow. While when using 0_4_1 it will find it. But again: I also run the code from the Python console and also run it from the CMD console as a python file. Both work. So everything is loaded correctly. It just does not work running it from Amibroker.

[empottasch]

i prepared a simple example. I hope @Kuba or @Tomasz can have a look.

Amibroker code:

PyLoadFromFile( "tft", "C:\\Users\\win 10\\AppData\\Local\\Programs\\Python\\Python38\\mypython\\fitting\\tensorflowtest.py" );

SetChartOptions( 0, chartShowArrows | chartShowDates );
Plot( C, "C", colorWhite, styleCandle, Null, Null, 0, 0, 0 );

ynew = PyEvalFunction( "tft", "tftest" );

Python code (file tensorflowtest.py):

import tensorflow as tf
import AmiPy

def tftest( ):
    ver = tf.__version__
    AmiPy.Print(ver + "\n")

what it does it prints the tensorflow version number in the Interpretation window of Amibroker. In my case the version it prints is 2.5.0.

This works with AmiPy version 0_4_1 but not with AmiPy version 0_4_2

So version 0_4_1 shows my tensorflow is installed correctly. When running 0_4_2 it shows error messages like:

AmiPy 0.4.2 log:
	AB: 6.39
	PY: 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)]
Logging started 2021-07-14 10:10:18
       0.00 ms (TID: 00001734) : Python started
       0.01 ms (TID: 00001734) : Start PyLoadFromFile( "tft", "C:\Users\win 10\AppData\Local\Programs\Python\Python38\mypython\fitting\tensorflowtest.py" )
       0.10 ms (TID: 00001734) : Initializing dict map
       0.76 ms (TID: 00001734) : ERROR: Error occured during Python execution:
* PyErr type: ModuleNotFoundError
* PyErr value:  No module named 'tensorflow'
* PyErr callstack:
* * File: 'C:\Users\win 10\AppData\Local\Programs\Python\Python38\mypython\fitting\tensorflowtest.py', line 1   (in <module>)

But also AmiPy version 0_4_1 can not be used when fitting a model, then it also crashes. So my overal request would be to have a look at the code I have show in the link in my first post (see: univariate NN test code). I can not run this code from within Amibroker (using AmiPy 0_4_1) while it runs without any problem from the Python console and the windows CMD console (from a py file). Using AmiPy 0_4_2 it will not even load and gives a similar error message as the error message shown above. So I hope this code can be used to possibly improve the plugin or direct me how to use it correctly

thank you

[Kuba]

I have tested it on my computer but for me it works just fine, but I have an idea why it is not working.

Python searches for modules in only a few directories. These paths are stored in sys.path.
The only difference in the set of lookup paths (at least for me) is that there is no 'C:\\Users\\<Username>\\AppData\\Roaming\\Python\\Python38\\site-packages' in 0.4.2 in comparation to 0.4.1, but I don't have any modules installed there.

It looks like Your modules are there what is probably because that's where you've installed Python.

[empottasch]

thanks,

I have it working now. I had all the paths added to the environment variables of Windows 10 but that didn't seem to work. So now I added code below to the Python test code (Amibroker code unchanged) and it works. Will report back later if I get the entire test code running

import sys
sys.path.append('C:/Users/win 10/AppData/Roaming/Python/Python38/site-packages')

import tensorflow as tf
import AmiPy

def tftest( ):
    ver = tf.__version__
    AmiPy.Print(ver + "\n")

[empottasch]

the Neural Net testcode is working when called from Amibroker. So everything seems to be working fine!