[FEAT] Add support for exogenous variables in utils.aggregate

[KuriaMaingi]

This change to the utility function will assist in instances where you need to generate your summation and Y_df but also want to retain any exogenous vars required for your forecast.

You will need to pass in a dictionary containing your exogenous vars and the Pandas agg functions you want applied against them. The latter can either be a single string or a list of strings if you want to apply multiple aggfuncs to the same column.

Output column will have the format column_aggfunc.

Examples:
aggregate(df, spec, exog_vars = {'avg_price':'mean'}) -> Will return a new column called "avg_price_mean"
aggregate(df, spec, exog_vars = {'avg_price':['mean','sum']}) -> Will return 2 new columns called "avg_price_mean" & "avg_price_sum

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[elephaint]

This change to the utility function will assist in instances where you need to generate your summation and Y_df but also want to retain any exogenous vars required for your forecast.

You will need to pass in a dictionary containing your exogenous vars and the Pandas agg functions you want applied against them. The latter can either be a single string or a list of strings if you want to apply multiple aggfuncs to the same column.

Output column will have the format column_aggfunc.

Examples: aggregate(df, spec, exog_vars = {'avg_price':'mean'}) -> Will return a new column called "avg_price_mean" aggregate(df, spec, exog_vars = {'avg_price':['mean','sum']}) -> Will return 2 new columns called "avg_price_mean" & "avg_price_sum

This is a great and valuable contribution. I'd like to first merge #296 so that we can have all the correct tests running. In the mean time, I'll have a look at the PR

[elephaint]

Thanks - nice and simple!

Could you include a unit test at the bottom of utils.ipynb that verifies correct behaviour? For example, for a small toy dataset it returns the correct aggregations over a set of inputs?

[christophertitchen]

Thank you for implementing the recommendations from the old PR.

As @elephaint said, a simple unit test is all it needs.

Going forward, considering the potential scale of hierarchical forecasting problems, we should probably consider swapping out pandas in these sorts of tasks for a library like Polars, for single-node environments, or Ibis, to give us the flexibility to easily switch between something like Polars or DuckDB as execution engines in single-node environments and PySpark in distributed environments.

[KuriaMaingi]

Thanks @elephaint and @christophertitchen , I'll have some time later this week and wrap this up.

On Polars/DuckDB/PySpark, this would have to be a repository level change right? Or would we want to consider modular self-contained conversions for tasks that are likely to require a fair amount of compute e.g. aggregations?

[elephaint]

Thanks @elephaint and @christophertitchen , I'll have some time later this week and wrap this up.

On Polars/DuckDB/PySpark, this would have to be a repository level change right? Or would we want to consider modular self-contained conversions for tasks that are likely to require a fair amount of compute e.g. aggregations?

I'd first like to move towards Polars support on a repo level, that should be relatively doable. Distributed support seems harder because of all the computations involved with Numpy arrays at the moment.

[KuriaMaingi]

Apologies, the commit somehow ended up being very messy merging & syncing all the changes that had happened since. All I was resolving were:

• Adding the unit test
• Small change to enable the function signature to handle instances where the exog_var aggfunc was a single string instead of a list e.g.

aggregate( df = df, spec = spec, exog_vars = {'exog1':'mean','exog2':'sum'} )

Let me know if this inadvertently caused other issues

[elephaint]

Apologies, the commit somehow ended up being very messy merging & syncing all the changes that had happened since. All I was resolving were:

• Adding the unit test
• Small change to enable the function signature to handle instances where the exog_var aggfunc was a single string instead of a list e.g.

aggregate( df = df, spec = spec, exog_vars = {'exog1':'mean','exog2':'sum'} )

Let me know if this inadvertently caused other issues

Thanks, I think the PR is fine but I observed some merge conflict issues in utils.ipynb that must be removed; I'll remove those myself.

[elephaint]

@KuriaMaingi Thank you for your contribution! We appreciate it very much.