Now possible to spread price sampling out throughout the day

[robcarver17]

Last piece of work from #913

This code allows you to sample Asian markets in the morning, Europe late afternoon, US in the evening. When you're sampling a large number of instruments, this makes life better than downloading a single large blob in one go.

To achieve this:

1. first add code like the following to your private_control_config.yaml file:

arguments:
  run_daily_prices_updates:
    update_historical_prices: # everything in this block is passed as **kwargs to this method
      download_by_zone:
        ASIA: '07:00'
        EMEA: '18:00'
        US: '20:00'

This will download Asian regional instruments at 7am, local machine time; Europe Middle East Africa at 6pm, and US at 8pm. These are local machine times not UTC. Feel free to change them.

2. We need to define the regions for each instrument. Regions are now setup in the instrument .csv configuration; use interactive_controls, option 5, option 52 to apply these changes to your production database.

If you want to use different regions, you will need to change these.

3. You should also ensure that run_daily_price_updates has a start time set in private_control_config.yaml earlier than 7am, and is started by the crontab (or other scheduler) before 7am.

Nerdy details: The run_daily_prices process will need to run once, continously throughout the day, starting before the first sample time (although if it doesn't, it will start sampling as soon as it can). It won't obey a STOP or PAUSE process control command, since we're running a single method that does it's own scheduling. If you need to kill it, you will have to use command line kill. If you restart it, it will lose all memory of what it has downloaded before, and redownload any regions whose start time has passed.

[meldinman]

I've been using the new and improved price update methods (its definitely faster) but I'm not sure if I've implemented things correctly or if there is an issue (probably the former).
I'm running run_daily_price_updates in the morning, it runs the first batch of downloads and then closes.

2023-02-03 07:33:17 {'type': 'update_historical_prices', 'instrument_code': 'YENEUR', 'contract_date': '20230900', 'broker': 'IB', 'clientid': 55, 'component': 'ibFuturesContractPriceData'}  Pausing 0.438012 seconds to avoid pacing violation
2023-02-03 07:33:18 {'type': 'update_historical_prices', 'component': 'arcticFuturesContractPriceData', 'instrument_code': 'YENEUR', 'contract_date': '20230900'}  Wrote 76 lines of prices for YENEUR/20230900 at Frequency.Day to Arctic connection: host 127.0.0.1, db production, collection futures_contract_prices
2023-02-03 07:33:18 {'type': 'update_historical_prices', 'component': 'arcticFuturesContractPriceData', 'instrument_code': 'YENEUR', 'contract_date': '20230900'}  Added 1 additional rows of data
2023-02-03 07:33:18 {'type': 'update_historical_prices', 'instrument_code': 'YENEUR', 'contract_date': '20230900', 'broker': 'IB', 'clientid': 55}  Added 1 rows at frequency Frequency.Day for YENEUR/20230900
2023-02-03 07:33:18 {'type': 'update_historical_prices', 'component': 'arcticFuturesContractPriceData', 'instrument_code': 'YENEUR', 'contract_date': '20230900'}  Wrote 76 lines of prices for YENEUR/20230900 at Frequency.Mixed to Arctic connection: host 127.0.0.1, db production, collection futures_contract_prices
2023-02-03 07:33:18 {'type': 'run_daily_prices_updates', 'instrument_code': 'AEX'}  update_historical_prices executed 1 times so done
2023-02-03 07:33:18 {'type': 'run_daily_prices_updates'}  Finished doing all executions of provided methods
2023-02-03 07:33:18 {'type': 'run_daily_prices_updates', 'instrument_code': 'AEX'}  update_historical_prices still alive, done 1 of 1 executions every 60 minutes
2023-02-03 07:33:18 {'type': 'run_daily_prices_updates'}  Process control run_daily_prices_updates marked close

It doesn't restart for the second or third downloads in the afternoon
My control config looks like this.

  run_daily_prices_updates:
    update_historical_prices: # everything in this block is passed as **kwargs to this method
      download_by_zone:
        ASIA: '07:00'
        EMEA: '12:00'
        US: '17:30'
      #max_executions: 1
  run_daily_fx_and_contract_updates:
    update_fx_prices:
      max_executions: 1
    update_sampled_contracts:
      max_executions: 1
  run_daily_update_multiple_adjusted_prices:
    update_multiple_adjusted_prices:
      max_executions: 1

Is there anything else that I am missing to ensure the process doesn't close?
Note I am on EST time so I changed the times to reflect. I also made sure to update my DB with the region times...

[robcarver17]

Weird it doesn't look like this log event was triggered:

        data.log.msg("All instruments downloaded today, finished")