Update pred deps and instructions

[piiq]

Thanks to some findings by @deeleeramone and @martinb-bb, the deps blocker was resolved.

The app launches and doesn’t crash, but it still produces warnings that will disapper as soon as someone closes this issue unit8co/darts#927

As for the instructions and the docs, I've pointed out that we recommend using the rosetta version of conda on macs. The native m1 build is too fragile and was removed from the docs until it will work out of the box.

I've tested that the environment works on an m1 mac with a rosetta conda. I would appreciate a test of the prediction menu on windows and linux.

Test scenario for Windows and Linux:

1. Create and activate a fresh conda env conda env create -n gst -f build/conda/conda-3-8-env-full.yaml && conda activate gst
2. poetry install
3. poetry install -E prediction
4. python terminal.py stocks/load f/pred/expo

[martinb-ai]

Currently testing on linux.

Two things I noticed with poetry install:

1. Wrong version of Darts

[2022-05-30 08:09:06,074] WARNING | darts.models | Support for Torch based models not available. To enable them, install "darts", "u8darts[torch]" or "u8darts[all]" (with pip); or "u8darts-torch" or "u8darts-all" (with conda).

This is not the right subversion of darts. It must include pytorch.

2. Pip installer is not handling library versions correctly.

 numeric_forecast = predicted_values.quantile_df()["AdjClose_0.5"].tail(n_predict)

Which causes this error:

Pip must be changing a specific library version that breaks the format.
Even if I then just uninstall pip uninstall u8darts and reinstall pip install "u8darts[torch]" - still breaks.

---

Only when I remove darts and reinstall via conda conda install -c conda-forge -c pytorch u8darts-torch does it function properly.

I shall investigate why.

[piiq]

when i add u8darts-torch to the conda env file it doesn't want to pull pytorch from the pytorch channel that gives an error when you launch the terminal.
i've tried explicitly install pytorch::pytorch before but it didn't help.

i don't yet understand how to install correct u8darts-torch from a .yml file

The settings that we have in pyproject.toml at the moment should execute identically to pip install u8darts[torch], so if it doesn't work with poetry I would not expect it to work if you install it with pip

re: errors, I see similar things, but in my case the chart pops up

[martinb-ai]

re: errors, I see similar things, but in my case the chart pops up

Chart pops up for me too. Just printing the table on the console is a problem due to the array format.

[piiq]

the problem arises here

[martinb-ai]

Yup, its because python numeric_forecast = predicted_values.quantile_df()["AdjClose_0.5"].tail(n_predict) provides a different output array type when using Pip vs. Conda...

I will speak with the Devs.

[deeleeramone]

If I uninstall u8darts[torch] and then reinstall using through the conda command above, I get an error saying:

(🦋) /stocks/ $ pred
One of the optional packages seems to be missing:  No module named 'darts' 

[martinb-ai]

If I uninstall u8darts[torch] and then reinstall using through the conda command above, I get an error saying:

(🦋) /stocks/ $ pred
One of the optional packages seems to be missing:  No module named 'darts' 

If you exit out of terminal and just open python and try to import darts, does it work?

[deeleeramone]

Yes, in a Jupyter notebook, import darts works.

[deeleeramone]

Got it now with no errors on this command, but RNN is now making a different error than before:

(🦋) /stocks/pred/ $ rnn
Training on 641 sequences of length 40.  Using 72 sequences  of length 40 for validation. Model will run 1 loops
Metal device set to: Apple M1

systemMemory: 8.00 GB
maxCacheSize: 2.67 GB

Epoch 1/50
2022-05-30 08:23:07.606039: F tensorflow/core/framework/tensor.cc:721] Check failed: IsAligned() ptr = 0x2ad5699f0
Abort trap: 6
(tester) Danglewoods-MacBook-Pro:openbbterminal danglewood$ /Users/danglewood/GitHub/GST1/miniforge3/envs/tester/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[martinb-ai]

2022-05-30 08:23:07.606039: F tensorflow/core/framework/tensor.cc:721] Check failed: IsAligned() ptr = 0x2ad5699f0
Abort trap: 6
(tester) Danglewoods-MacBook-Pro:openbbterminal danglewood$ /Users/danglewood/GitHub/GST1/miniforge3/envs/tester/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

This is the same issues I was having with Tensorflow on M1. Seems the whole library dependancy chain is spaghetti. That is why I needed to write out full commands (in specific order) for installing on M1. (as we did back a week ago)

conda env create -n tester --file build/conda/conda-3-8-env.yaml
conda activate tester 
conda install -c apple tensorflow-deps
conda install -c apple tensorflow-deps==2.8.0
python -m pip install tensorflow-metal 
conda update -c defaults numpy  
pip3 install -r requirements.txt 
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 
pip3 install -r requirements.txt
conda install -c conda-forge tensorflow==2.7.1
conda install -c conda-forge -c pytorch u8darts-torch
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

Worst case scenario - we delay the addition of the new forecasting Dart models for now until we can package it up properly for the installer and have this resolved offline.

[piiq]

@deeleeramone let's come back to testing the m1 native build in a few months? it's very raw

[piiq]

@Chavithra @andrewkenreich anyone of you can test if the environment works on windows?

[martinb-ai]

Update!
I have found the culprit to the problem with Error: unsupported format string passed to DataArray.__format__

It has to do with how pandas manipulates<class 'pandas.core.series.Series'>. This has been updated in later versions of pandas.

Are we able to update pandas in poetry from 1.3.5 --> 1.4.2 (aka. latest) ?

[Chavithra]

@Chavithra @andrewkenreich anyone of you can test if the environment works on windows?

Seems to work on windows.

Update! I have found the culprit to the problem with Error: unsupported format string passed to DataArray.__format__

It has to do with how pandas manipulates<class 'pandas.core.series.Series'>. This has been updated in later versions of pandas.

Are we able to update pandas in poetry from 1.3.5 --> 1.4.2 (aka. latest) ?

I don't see reason against.

We will need to check for side effects though :

• there might be libraries that we use which are not yet compatible with 1.4
• it might break some existing code due to Pandas breaking changes (already append in the past)