Feat/deployable backtesting

[cardosofede]

Added new functionalities:

• Install Hummingbot library
• Adapt backtesting engine to use hummingbot
• Save configuration files for the trials selected
• Deploy hummingbot instances with multiple controller configs in the same bot

[rapcmia]

Test commit 92bf3b46246d222d4e58504312683d19eb092e2e

• Run tests on local mbp (m1)
• Successfully install dashboard env using make env_create on both osx64 and osx_arm64 platforms
• Ran the initial tests below:

Start master_bot-conf

• Getting this error response if there are no existing hummingbot-master_bot_conf container
  
  • While this happened, dashboard is able to create hummingbot-master_bot_conf container successfully ✅
• What if we stop hummingbot-master_bot_conf?
  • Run start again from bot orchestration → credentials ✅
  • Stop the hummingbot-master_bot_conf then start again ✅
  • There is a available stop button now after running start
  • What happens on the data inside the hummingbot-master_bot_conf? Stays the same ✅

---

Launch bot

• Getting the error below even when hummingbot-master_bot-conf active or not

  TypeError: can only join an iterable
  Traceback:
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 537, in _run_script
      self._session_state.on_script_will_rerun(rerun_data.widget_states)
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/state/safe_session_state.py", line 68, in on_script_will_rerun
      self._state.on_script_will_rerun(latest_widget_states)
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/state/session_state.py", line 500, in on_script_will_rerun
      self._call_callbacks()
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/state/session_state.py", line 513, in _call_callbacks
      self._new_widget_state.call_callback(wid)
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/state/session_state.py", line 260, in call_callback
      callback(*args, **kwargs)
  File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit_elements/core/callback.py", line 91, in dispatch
      callback(**frontend_params)
  File "/Users/rapcomia/github/dashboard/69/ui_components/launch_strategy_v2.py", line 54, in launch_new_bot
      "-e", f"controller_configs={','.join(self._controller_config_selected)}"]
  

---

Instances

• When creating instances from the parent master_bot_conf:
  • Getting warning related to st.experimental_rerun

    323959240c6ad6992908bf635f684fd08a464eaafbf7b7cf1180cd33036825de
    2023-09-22 13:34:02.910 Please replace 'st.experimental_rerun' with 'st.rerun'.
    
    `st.experimental_rerun` will be removed after 2024-04-01.
    2023-09-22 13:34:08.231 Please replace 'st.experimental_rerun' with 'st.rerun'..
    
    `st.experimental_rerun` will be removed after 2024-04-01.
    

    • While this happens, the container is running ok and able to docker attach ✅
• When stopping the instance:

  docker: Error response from daemon: Conflict. The container name "/hummingbot-test01" is already in use by container "323959240c6ad6992908bf635f684fd08a464eaafbf7b7cf1180cd33036825de". You have to remove (or rename) that container to be able to reuse that name.
  See 'docker run --help'.
  hummingbot-test01
  

  • While this happens, the container successfully stopped ✅
  • Tried to start then stop again and it seems to work as expected

Note: The errors reported above only occurs when it was initially created and stopped. When a container is started 2nd time, the warning does not occur

---

Build docker images from this PR

74.19   Getting requirements to build wheel: started
74.19   Getting requirements to build wheel: finished with status 'done'
74.19   Preparing metadata (pyproject.toml): started
74.19   Preparing metadata (pyproject.toml): finished with status 'done'
74.19 Collecting hummingbot (from -r /tmp/condaenv.fpvo1ve6.requirements.txt (line 1))
74.19   Downloading hummingbot-20230930.tar.gz (8.2 MB)
74.19      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.2/8.2 MB 28.2 MB/s eta 0:00:00
74.19   Installing build dependencies: started
74.19   Installing build dependencies: finished with status 'done'
74.19   Getting requirements to build wheel: started
74.19   Getting requirements to build wheel: finished with status 'error'
74.19
74.19 failed
74.19
74.19 CondaEnvException: Pip failed
74.19
------
Dockerfile:13
--------------------
  12 |     COPY environment_conda.yml /tmp/environment_conda.yml
  13 | >>> RUN conda env create -f /tmp/environment_conda.yml && \
  14 | >>>     conda clean -afy && \
  15 | >>>     rm /tmp/environment_conda.yml
  16 |
--------------------
ERROR: failed to solve: process "/bin/sh -c conda env create -f /tmp/environment_conda.yml &&     conda clean -afy &&     rm /tmp/environment_conda.yml" did not complete successfully: exit code: 1

• This error occurred either the following docker build command:
  • docker build -t hummingbot/dashboard:test01 .
  • docker build -t hummingbot/dashboard:test01 -f Dockerfile .

Pending:
Not tested on cloud server running Ubuntu and related AMD platforms

[cardosofede]

@rapcmia regarding the master bot conf testing, don't worry since this PR is more for the second part.
I will record a video showing how this should be used so you can test that.
Regarding to the docker build, seems like the error happen when trying to install hummingbot. @david-hummingbot when do you uploaded the wheels? because docker uses linux so probably if you tried to do it before David finished the upload it's expected

[cardosofede]

@rapcmia I added error handling if there are no controller configs selected. I'm also uploading a video showing how to use it

[rapcmia]

PR update:

• Run tests on mac(m1)
• Clone this PR on osx64 and osx_arm64 platforms for conda and successfully pip install hummingbot ✅

In progress:
Testing in progress on backtests base on fede's demo

[fengtality]

@cardosofede I think we have to create the hummingbot_files/controller_configs directory. Otherwise, it throws an error since it can't save the controller config file.

[cardosofede]

Yes I forgot to add it to git because I already have the folder

…

On Tue, 26 Sep 2023 at 17:12 Michael Feng ***@***.***> wrote: @cardosofede <https://github.com/cardosofede> I think we have to create the hummingbot_files/controller_configs directory. Otherwise, it throws an error since it can't save the controller config file. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIZJOSHTKL2KXKWLALAFUILX4MZKPANCNFSM6AAAAAA45AAAMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rapcmia]

Test commit 7cd1a6417d8f3f2f05a968aee85bd405258b4dd6

• On Get data, downloaded 100, 300 then 1000 of historical data from intervals 1m,3m and 1hr for BTC-USDT,DOGE-USDT,XRP-USDT and DOGE-USDT ✅
• On this test, we use default bollinger template

---

On Optimize, Test using default bollinger template for BTC-USDT

• Change trials from 10, 30 and 50 and run, all ok ✅
  
• However we also observe these instances on the terminal when running trials:

  472, 'bb_std': 3.88, 'bb_long_threshold': -0.31999999999999984, 'bb_short_threshold': 2.12, 'std_span': 181}. Best is trial 840 with value: 0.02676680891678896.
  Traceback (most recent call last):
    File "/Users/rapcomia/github/dashboard/69/quants_lab/optimizations/bollinger_v_26-09-2023a.py", line 64, in objective
      backtesting_results = engine.run_backtesting(initial_portfolio_usd=initial_portfolio_usd, trade_cost=trade_cost,
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/backtesting_engine_base.py", line 104, in run_backtesting
      executors_df = self.simulate_execution(processed_data, initial_portfolio_usd=initial_portfolio_usd, trade_cost=trade_cost)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/directional_trading/directional_trading_backtesting_engine.py", line 11, in simulate_execution
      df = self.apply_triple_barrier_method(df,
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/backtesting_engine_base.py", line 45, in apply_triple_barrier_method
      df = self.apply_tp_sl_on_tl(df, tp=tp, sl=sl)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/backtesting_engine_base.py", line 86, in apply_tp_sl_on_tl
      df["close_time"] = df[["tl", "take_profit_time", "stop_loss_time"]].dropna(how="all").min(axis=1)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/frame.py", line 3902, in __getitem__
      indexer = self.columns._get_indexer_strict(key, "columns")[1]
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6114, in _get_indexer_strict
      self._raise_if_missing(keyarr, indexer, axis_name)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6178, in _raise_if_missing
      raise KeyError(f"{not_found} not in index")
  KeyError: "['take_profit_time', 'stop_loss_time'] not in index"
  

On Analyze v2

• Select a Trial ID base on win signals > 10
• Set max records to default 1000
• Run backtesting but getting an KeyError below:
  
  • Tried to change the max_records from 1000 to 100 or 300
  • Change intervals from 1h, 1m and 3m base on the 1000 data that was initially downloaded
  • This time getting the error below:

    2023-09-27 12:12:04.476 Uncaught app exception
    Traceback (most recent call last):
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3790, in get_loc
        return self._engine.get_loc(casted_key)
      File "index.pyx", line 152, in pandas._libs.index.IndexEngine.get_loc
      File "index.pyx", line 181, in pandas._libs.index.IndexEngine.get_loc
      File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
      File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
    KeyError: 'BBP_517_3.98'
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 541, in _run_script
        exec(code, module.__dict__)
      File "/Users/rapcomia/github/dashboard/69/pages/backtest_manager/analyze_v2.py", line 214, in <module>
        backtesting_results = engine.run_backtesting(initial_portfolio_usd=initial_portfolio_usd,
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/backtesting_engine_base.py", line 101, in run_backtesting
        processed_data = self.get_data(start=start, end=end)
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/hummingbot/smart_components/strategy_frameworks/backtesting_engine_base.py", line 95, in get_data
        df = self.controller.get_processed_data()
      File "/Users/rapcomia/github/dashboard/69/quants_lab/controllers/bollinger.py", line 50, in get_processed_data
        long_condition = df[f"BBP_{self.config.bb_length}_{self.config.bb_std}"] < self.config.bb_long_threshold
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/frame.py", line 3896, in __getitem__
        indexer = self.columns.get_loc(key)
      File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3797, in get_loc
        raise KeyError(key) from err
    KeyError: 'BBP_517_3.98'
    

Steps to reproduce;

1. Get data of 100 or 1000 for BTC-USDT,ETH-USDT
2. Optimize and run trial for either 30 and 50
3. Analyze (v2) data by selecting trial id and run backtesting using default
4. Observe returning errors

---

On Optimize, Test using default bollinger template for XRP-USDT and ETH-USDT

• Run trials for 10, 30 and 50 but getting this ValueError: CategoricalDistribution does not support dynamic value space.

  [I 2023-09-27 11:51:32,276] Trial 898 pruned.
  Traceback (most recent call last):
    File "/Users/rapcomia/github/dashboard/69/quants_lab/optimizations/bollinger_v_26-09-2023.py", line 18, in objective
      trading_pair = trial.suggest_categorical('trading_pair', ['XRP-USDT', 'DOGE-USDT', ])
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/trial/_trial.py", line 405, in suggest_categorical
      return self._suggest(name, CategoricalDistribution(choices=choices))
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/trial/_trial.py", line 636, in _suggest
      storage.set_trial_param(trial_id, name, param_value_in_internal_repr, distribution)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/storages/_cached_storage.py", line 171, in set_trial_param
      self._backend.set_trial_param(trial_id, param_name, param_value_internal, distribution)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/storages/_rdb/storage.py", line 568, in set_trial_param
      self._set_trial_param_without_commit(
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/storages/_rdb/storage.py", line 603, in _set_trial_param_without_commit
      trial_param.check_and_add(session)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/storages/_rdb/models.py", line 351, in check_and_add
      self._check_compatibility_with_previous_trial_param_distributions(session)
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/storages/_rdb/models.py", line 367, in _check_compatibility_with_previous_trial_param_distributions
      distributions.check_distribution_compatibility(
    File "/Users/rapcomia/anaconda3/envs/dashboard/lib/python3.10/site-packages/optuna/distributions.py", line 670, in check_distribution_compatibility
      raise ValueError(
  ValueError: CategoricalDistribution does not support dynamic value space.
  

Pending tests:

• Save controller config and launch bot

[rapcmia]

Run tests on save controller config

• Successfully run backtesting for specific trial ID
• Added ../hummingbot_files/controller_configs and click save controller config as per previous comments
• Successfully created a strategy file and confirmed saved on ../hummingbot_files/controller_configs ✅

Note: current commit does not support hummingbot_files/controller_configs

---

Run tests on launch bots

• Observed that UI now shows a strategy file named after the directional + trial id
• If no name is define for the container, it displays You need to define the bot name and select the controllers config that you want to deploy
• Setup test01 then create using hummingbot latest branch
• Created container successfully
  • Data folders like conf and log files are within the container folder
  • Tested by creating two separate bots with same strategy file create from backtests ✅
  • However getting Unhandled error in background task: No module named 'hummingbot.smart_components.strategy_frameworks' which seems to be expected since the changes needed are now available on development branch ✅ (not adding the traceback, else we can add upon request)
  • Recreate container this time using development branch ✅
    

[david-hummingbot]

Build multiarch Docker image based on feat/deployable_backtesting branch = OK
Fresh Source install - OK
Observe if any errors installing wheel - OK
Bot Orchestration, create master_bot_conf = OK
Attach and add Binance credentials = OK
Instances, check if Binance credentials were added to new instance = OK
Get Data, download Binance Perpetual, ETH-USDT, 15m, 1H, 360 = OK
Create = OK
Optimize, modify bollinger strategy and run trial (300 runs) = OK
Analyze v2, run backtesting and save controller config = OK
Launch Bot, select at least 2 controllers and start instance = OK