Scalability to run large number of time-series without crashing

[ekurniawan-ispt]

Description

I get frustrated that it crashes or runs for days when running greater number time series such as 2500 time series.
I am referring to from_pandas_dynamic function (which is the Dynamic NOTEARS implementation in Causalnex)

Context

I love this package and I use it for small number of time series without issues.
However, recently I have been involved in a project with large number of time series, eg around 2500 time series to discover the causal relationships.

Possible Implementation

I suggest for the team look into the possibilities of implementing PySpark MLLib for Spark matrices and Spark dataframes.

Possible Alternatives

I have started implementing Granger Causal test using Pyspark Dataframe, but I still want to see Bayesian Network solution by Causalnex package.