Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature request] progressively save backtest results #50

Open
marcfon opened this issue Dec 24, 2021 · 2 comments
Open

[feature request] progressively save backtest results #50

marcfon opened this issue Dec 24, 2021 · 2 comments
Assignees
@timelyart
Labels
enhancement New feature or request

Comments

@marcfon
Copy link

marcfon commented Dec 24, 2021

I was running a large backtest and after 13 hours it exited with a timeout (see below) but unfortunately all the results up to that point weren't saved.

Would be nice if Kairos progressively saves the backtest results. For instance after each completed strategy.

2021-12-24 05:20:47.136 INFO tv - get_strategy_statistic: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:21:47.166 INFO tv - get_strategy_statistic: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:22:47.179 ERROR tv - open_chart: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 1403, in open_chart
    test_data = back_test(browser, strategy, symbols, atomic_inputs, atomic_properties)
  File "/kairos274/tv/tv.py", line 3815, in back_test
    strategy_summary['summary']['total'], strategy_summary['summary']['interval'], strategy_summary['summary']['symbol'], strategy_summary['raw'] = back_test_strategy(browser, inputs, properties, symbols, strategy_config, number_of_charts, strategy_number, number_of_strategies)
  File "/kairos274/tv/tv.py", line 3907, in back_test_strategy
    first_symbol = refresh_session(browser)
  File "/kairos274/tv/tv.py", line 4223, in refresh_session
    refresh(browser)
  File "/kairos274/tv/tv.py", line 380, in refresh
    browser.refresh()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 831, in refresh
    self.execute(Command.REFRESH)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:23:17.221 ERROR tv - snapshot: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 1403, in open_chart
    test_data = back_test(browser, strategy, symbols, atomic_inputs, atomic_properties)
  File "/kairos274/tv/tv.py", line 3815, in back_test
    strategy_summary['summary']['total'], strategy_summary['summary']['interval'], strategy_summary['summary']['symbol'], strategy_summary['raw'] = back_test_strategy(browser, inputs, properties, symbols, strategy_config, number_of_charts, strategy_number, number_of_strategies)
  File "/kairos274/tv/tv.py", line 3907, in back_test_strategy
    first_symbol = refresh_session(browser)
  File "/kairos274/tv/tv.py", line 4223, in refresh_session
    refresh(browser)
  File "/kairos274/tv/tv.py", line 380, in refresh
    browser.refresh()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 831, in refresh
    self.execute(Command.REFRESH)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2041, in snapshot
    element = find_element(browser, 'html')
  File "/kairos274/tv/tv.py", line 451, in find_element
    element = WebDriverWait(browser, delay).until(
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/wait.py", line 71, in until
    value = method(self._driver)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 64, in __call__
    return _find_element(driver, self.locator)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element
    raise e
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element
    return driver.find_element(*by)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:23:17.495 INFO tv - run: No alerts or signals set

2021-12-24 05:23:17.496 INFO tv - run: No signals triggered. Nothing to send
2021-12-24 05:23:47.516 ERROR tv - close_alerts: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 359, in close_alerts
    alert = browser.switch_to.alert
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/switch_to.py", line 55, in alert
    alert.text
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/common/alert.py", line 67, in text
    return self.driver.execute(Command.W3C_GET_ALERT_TEXT)["value"]
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:24:07.531 ERROR tv - destroy_browser: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2970, in destroy_browser
    close_all_popups(browser)
  File "/kairos274/tv/tv.py", line 343, in close_all_popups
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:24:37.572 ERROR tv - snapshot: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2970, in destroy_browser
    close_all_popups(browser)
  File "/kairos274/tv/tv.py", line 343, in close_all_popups
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2041, in snapshot
    element = find_element(browser, 'html')
  File "/kairos274/tv/tv.py", line 451, in find_element
    element = WebDriverWait(browser, delay).until(
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/wait.py", line 71, in until
    value = method(self._driver)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 64, in __call__
    return _find_element(driver, self.locator)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element
    raise e
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element
    return driver.find_element(*by)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:24:57.587 ERROR tv - run: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 3128, in run
    destroy_browser(browser)
  File "/kairos274/tv/tv.py", line 2981, in destroy_browser
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:25:27.621 ERROR tv - close_alerts: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 3128, in run
    destroy_browser(browser)
  File "/kairos274/tv/tv.py", line 2981, in destroy_browser
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 359, in close_alerts
    alert = browser.switch_to.alert
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/switch_to.py", line 55, in alert
    alert.text
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/common/alert.py", line 67, in text
    return self.driver.execute(Command.W3C_GET_ALERT_TEXT)["value"]
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:25:47.663 ERROR tv - destroy_browser: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 3128, in run
    destroy_browser(browser)
  File "/kairos274/tv/tv.py", line 2981, in destroy_browser
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2970, in destroy_browser
    close_all_popups(browser)
  File "/kairos274/tv/tv.py", line 343, in close_all_popups
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)

2021-12-24 05:26:17.671 ERROR tv - snapshot: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)
Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 3128, in run
    destroy_browser(browser)
  File "/kairos274/tv/tv.py", line 2981, in destroy_browser
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2970, in destroy_browser
    close_all_popups(browser)
  File "/kairos274/tv/tv.py", line 343, in close_all_popups
    browser.close()
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 688, in close
    self.execute(Command.CLOSE)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kairos274/tv/tv.py", line 2041, in snapshot
    element = find_element(browser, 'html')
  File "/kairos274/tv/tv.py", line 451, in find_element
    element = WebDriverWait(browser, delay).until(
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/wait.py", line 71, in until
    value = method(self._driver)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 64, in __call__
    return _find_element(driver, self.locator)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 415, in _find_element
    raise e
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/support/expected_conditions.py", line 411, in _find_element
    return driver.find_element(*by)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/site-packages/selenium-3.141.0-py3.8.egg/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 30.000
  (Session info: headless chrome=96.0.4664.110)

Message: unknown error: failed to close window in 20 seconds
  (Session info: headless chrome=96.0.4664.110)


========================================
2021-12-24 05:26:37 - End Program
Elapsed time: 13 hours, 7 minutes
@marcfon marcfon changed the title [feature request] save intermediate backtest results [feature request] progressively save backtest results Dec 24, 2021
@timelyart
Copy link
Owner

This is one of the biggest problems with backtesting. TradingView simply ends the session after a (long) while. I agree that some form of saving the state in between should be added.

Unfortunately, it is not that easy to do since Kairos aggregates data first in memory before presenting it. Since Kairos doesn't use a database - other than flat files - it is not straightforward.

@timelyart timelyart self-assigned this Jan 4, 2022
@timelyart timelyart added the enhancement New feature or request label Jan 4, 2022
@marcfon
Copy link
Author

marcfon commented Jan 7, 2022

I can see why this wouldn't be straightforward using just the filesystem. Might be a benefit of having the option to push the backtest results progressively to a google sheet. But of course it would be better if there was a solution that didn't depend on that.

By the way usually the connect is stable for hours. Just on rare occasions it stops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timelyart timelyart

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@marcfon @timelyart