You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
currently, we have the num_partitions parameter for heavy requests that allows users to partition their data into chunks of time series. this process, however, is being done sequentially, which can ve slow in some cases. a good option might be to parallelize these multiple requests. to make sequential partitioned requests we used the following decorator:
instead of calling the methods using a for loop, this process can be optimized to call them in parallel using concurrent.future.ThreadPoolExecutor (since the process is I/O and CPU bound) or multiprocess.Pool (the election of the method should be based on empirical performance).
the election of the number of parallel processes should be elected using min(num_partitions, os.cpu_count()); this behavior must be documented in the corresponding docstrings. ideally, the pr should include before and after processing time.
The text was updated successfully, but these errors were encountered:
currently, we have the
num_partitions
parameter for heavy requests that allows users to partition their data into chunks of time series. this process, however, is being done sequentially, which can ve slow in some cases. a good option might be to parallelize these multiple requests. to make sequential partitioned requests we used the following decorator:nixtla/nixtlats/nixtla_client.py
Lines 748 to 773 in 7a3e52a
instead of calling the methods using a for loop, this process can be optimized to call them in parallel using
concurrent.future.ThreadPoolExecutor
(since the process is I/O and CPU bound) ormultiprocess.Pool
(the election of the method should be based on empirical performance).the election of the number of parallel processes should be elected using
min(num_partitions, os.cpu_count())
; this behavior must be documented in the corresponding docstrings. ideally, the pr should include before and after processing time.The text was updated successfully, but these errors were encountered: