Skip to content

casperm/ctrader-backtest

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
tdbc34
tdbc34
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
setup.py
setup.py
 
 

Repository files navigation

Read tdbc34 backtest files with Windows WSL

This Python library deserialize tdbc34 cache files from your cTrader Backtesting Cache directory.

Installation

pip install git+https://github.com/casperm/ctrader-backtest.git

Import library

from tdbc34 import CacheSerializer as cache

List all available cache files.

Run DescribeAvailableCacheData function use glob regex command to match the cache location. Parameter: env Options: wsl (default) - Windows Subsystem for Linux

cache.DescribeAvailableCacheData()[-4::]
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Ticks - has zero files
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Ticks - has zero files


[{'account': 'demo_456789',
  'instrument': 'USDCAD',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCAD/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDCAD',
  'type': 'Ticks',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCAD/Ticks',
  'from': datetime.datetime(2013, 7, 22, 0, 0),
  'to': datetime.datetime(2019, 9, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDCHF',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)},
 {'account': 'demo_456789',
  'instrument': 'USDJPY',
  'type': 'Hour',
  'path': '/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Hour',
  'from': datetime.datetime(2013, 7, 1, 0, 0),
  'to': datetime.datetime(2013, 7, 1, 0, 0)}]

Run DescribeAvailableCacheDataInDataFrame function to list cache location in Pandas DataFrame

cache.DescribeAvailableCacheDataInDataFrame()
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDCHF/Ticks - has zero files
/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/USDJPY/Ticks - has zero files
<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; } 
.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}
</style>
account instrument type path from to
0 demo_12345678 EURUSD Minute /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2019-12-28 2019-12-31
1 demo_12345678 EURUSD Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2019-12-31 2020-01-13
2 demo_456789 AUDJPY Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2011-01-01 2013-07-01
3 demo_456789 AUDJPY Minute /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2011-02-04 2020-08-01
4 demo_456789 AUDJPY Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-22 2020-08-01
... ... ... ... ... ... ...
61 demo_456789 US500 Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2015-01-23 2022-10-04
62 demo_456789 USDCAD Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01
63 demo_456789 USDCAD Ticks /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-22 2018-10-23
64 demo_456789 USDCHF Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01
65 demo_456789 USDJPY Hour /mnt/z/Users/Home/AppData/Roaming/XXXXXXXXX... 2013-07-01 2013-07-01

66 rows × 6 columns

Load Cached files

The GetBackTestData takes three parameters (cacheFiles, start_date, end_date). And returns PyArrow datatable.

%%time
_cacheFiles = "/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/US500/Ticks"
US500Ticks = cache.GetBackTestData(_cacheFiles,"2021.01.01","2021.03.01")
CPU times: user 2.47 s, sys: 45.4 ms, total: 2.51 s
Wall time: 3.3 s

Query PyArrow datatable with SQL (DuckDB OLAP Engine)

import duckdb
import pandas as pd
import sqlalchemy
# No need to import duckdb_engine
#  SQLAlchemy will auto-detect the driver needed based on your connection string!

# Import ipython-sql Jupyter extension to create SQL cells
%load_ext sql
%sql duckdb:///:memory:
%%sql
SELECT
    utctime,
    ask, bid,
    (ask + bid)/2.0 as mid,
    utctime + INTERVAL 91 second as WindowEnd,
    last(bid) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowbid,
    last(ask) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowask
FROM US500Ticks
ORDER BY utctime 
LIMIT 5
 * duckdb:///:memory:
Done.
utctime ask bid mid WindowEnd Windowbid Windowask
2021-01-03 23:00:10.072000 3758.100000 3757.500000 3757.8 2021-01-03 23:01:41.072000 3758.300000 3758.900000
2021-01-03 23:00:10.486000 3758.500000 3757.900000 3758.2 2021-01-03 23:01:41.486000 3757.900000 3758.500000
2021-01-03 23:00:10.885000 3758.600000 3758.000000 3758.3 2021-01-03 23:01:41.885000 3757.900000 3758.500000
2021-01-03 23:00:11.091000 3758.900000 3758.300000 3758.6 2021-01-03 23:01:42.091000 3757.900000 3758.500000
2021-01-03 23:00:11.686000 3759.000000 3758.400000 3758.7 2021-01-03 23:01:42.686000 3757.600000 3758.200000
2021-01-03 23:00:12.113000 3759.100000 3758.500000 3758.8 2021-01-03 23:01:43.113000 3757.400000 3758.000000
2021-01-03 23:00:12.487000 3759.400000 3758.800000 3759.1 2021-01-03 23:01:43.487000 3757.400000 3758.000000
2021-01-03 23:00:13.685000 3759.200000 3758.600000 3758.9 2021-01-03 23:01:44.685000 3757.400000 3758.000000
2021-01-03 23:00:13.887000 3760.100000 3759.500000 3759.8 2021-01-03 23:01:44.887000 3757.400000 3758.000000
2021-01-03 23:00:14.884000 3759.600000 3759.000000 3759.3 2021-01-03 23:01:45.884000 3757.000000 3757.600000 
%%time
_cacheFiles = "/mnt/z/Users/Home/AppData/Roaming/XXXXXXXXXXX/BacktestingCache/XXXXXXXXX/NZDUSD/Ticks"
NZDUSDTicks = cache.GetBackTestData(_cacheFiles,"2021.01.01","2021.03.01")
CPU times: user 5.85 s, sys: 177 ms, total: 6.03 s
Wall time: 7.21 s

%%sql
SELECT
    utctime,
    ask, bid,
    (ask + bid)/2.0 as mid,
    utctime + INTERVAL 91 second as WindowEnd,
    last(bid) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowbid,
    last(ask) OVER(ORDER BY utctime range BETWEEN INTERVAL 0 second preceding AND INTERVAL 91 second following) as Windowask
FROM NZDUSDTicks
ORDER BY utctime 
LIMIT 10
 * duckdb:///:memory:
Done.
utctime ask bid mid WindowEnd Windowbid Windowask
2021-01-03 22:01:00.004000 0.719410 0.719080 0.719245 2021-01-03 22:02:31.004000 0.719010 None
2021-01-03 22:01:01.105000 None 0.719100 None 2021-01-03 22:02:32.105000 0.719010 None
2021-01-03 22:01:01.909000 None 0.719080 None 2021-01-03 22:02:32.909000 0.719010 None
2021-01-03 22:01:02.429000 None 0.719070 None 2021-01-03 22:02:33.429000 0.719010 None
2021-01-03 22:01:02.954000 None 0.719050 None 2021-01-03 22:02:33.954000 0.719010 None
2021-01-03 22:01:03.307000 None 0.719030 None 2021-01-03 22:02:34.307000 0.719010 None
2021-01-03 22:01:03.548000 None 0.719010 None 2021-01-03 22:02:34.548000 0.719010 None
2021-01-03 22:03:02.290000 0.719350 0.719060 0.719205 2021-01-03 22:04:33.290000 0.718970 None
2021-01-03 22:03:18.453000 0.719410 0.719010 0.71921 2021-01-03 22:04:49.453000 0.718990 None
2021-01-03 22:03:31.321000 0.719240 None None 2021-01-03 22:05:02.321000 0.718970 None

About

Python library to load tdbc34 files for backtesting

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages