How to set up the data for production system from scratch?

[anna-gie]

Recently, I discovered pysystemtrade and, after reading through documentation, I'm really enthusiastic about its functionality.

However, I’m having trouble understanding the data workflow, particularly with update_sampled_contracts.py.

I pulled the develop branch and the latest changes, as I read on some posts, that it includes built-in parquet functionality.
The repo includes CSV files with multiple and adjusted price data up 28.03.24.

1. To convert the CSV data to Parquet files, I used the following scripts:

• repocsv_adjusted_prices.py
• repocsv_mulitple_prices.py
• repocsv_spotfx_prices.py
  I also used this script to write spread costs into MongoDB:
• repocsv_spread_costs.py

2. Afterwards I get the actual prices from IB:

• seed_price_data_from_IB.py
  I tested it on the instrument 'AEX' and the contracts are written as Parquet files up to the future.

3. Since the roll calendars were outdated by several years, I ran:

• rollcalendars_from_arcticprices_to_csv.py
  The last entries of the rollcalendar 'AEX' are:

2024-03-14 16:00:01,20240300,20240400,20240400
2024-03-28 23:00:00,20240400,20240500,20240500
2024-04-15 23:00:00,20240400,20240500,20240500
2024-05-15 23:00:00,20240500,20240600,20240600
2024-06-14 23:00:00,20240600,20240700,20240700
2024-07-15 23:00:00,20240700,20240800,20240800
2024-08-15 23:00:00,20240800,20240900,20240900
2024-09-16 23:00:00,20240900,20241000,20241000
2024-10-15 23:00:00,20241000,20241100,20241100
2024-11-01 23:00:00,20241100,20241200,20241200

Now the actual price data is stored in Parquet files and the rollcalendar as csv.

Could someone confirm if these steps are correct?

To keep the multiple and adjusted price data up to date from Parquet contract prices, is it necessary to runupdate_sampled_contracts.py and update_historical_prices.py?

update_sample_contracts.py script returns following error:

File "/home/home/pysystemtrade/sysproduction/data/contracts.py", line 198, in get_contract_from_db_given_code_and_id contract_object = self.db_contract_data.get_contract_object( File "/home/home/pysystemtrade/sysdata/futures/contracts.py", line 40, in get_contract_object raise ContractNotFound( syscore.exceptions.ContractNotFound: Contract AEX/20240500 not found

It seems the script is looking for contract data in MongoDB, but there is none.
The contract data is stored in the Parquet fileAEX#20240500.parquet.

Did I miss a step here? Does MongoDB need to be populated with contract data beforehand?

I’d greatly appreciate any guidance on how to set up the data correctly for the production system, as some parts of the documentation seem outdated.

[tgibson11]

Yes, you missed a couple steps. Your steps 1 & 2 were good.

Your step 3 was unnecessary. The roll calendars are a means to an end - the end of getting up-to-date multiple & adjusted prices. Once you have those, the roll calendars are unnecessary. They are not used in production (or backtesting for that matter).

The steps you seem to have missed were:

• Extend the provided multiple prices up to the current date (using the recent contract prices you downloaded from IB in step 2).
• Generate adjusted prices from the up-to-date multiple prices.
• Now you can run update_sampled_contracts

For details, see this comment, and the entire thread would probably be worth reading to see some examples of issues people have encountered and how to address them.

To answer/clarify a couple other things:

To keep the multiple and adjusted price data up to date from Parquet contract prices, is it necessary to runupdate_sampled_contracts.py and update_historical_prices.py?

Yes. Those are meant to be run daily (more or less).
Edit: but you also need to run update_multiple_adjusted_prices

It seems the script is looking for contract data in MongoDB, but there is none. The contract data is stored in the Parquet fileAEX#20240500.parquet.

The script is looking for data in MongoDB, and you're correct that there is none. However, the contract price data in Parquet is not the data it's looking for. It's looking for the properties of the contract itself, which will come from IB if they're not already in the database. The real reason for that error is that the data for that contract is no longer available from IB. Getting your multiple prices up-to-date will fix this.

Does MongoDB need to be populated with contract data beforehand?

update_sampled_contracts will do this. But because your multiple price data is out of date, it's currently trying to do it for contracts that are no longer available from IB.

[anna-gie]

Thank you Todd, your answers and your script update_provided_multiple_prices.py from this thread are very helpful.
I will give it a try to update all the instruments

[bug-or-feature]

Hi @anna-gie there is a push on at the moment to improve the documentation - especially for new users. The docs are outdated. If you could make a note of any specific areas where the docs could be improved, it would be very useful. See ticket here

[anna-gie]

Yes, this is a great idea!
As I'm starting from scratch, I'm going through the entire documentation and will make note of any outdated sections I encounter. I will add these to the ticket you mentioned for reference.