ODFVs raise a PerformanceWarning for very large sets of features

[roy651]

Expected Behavior

Feature views should support hundreds of features, regardless of the existence of ODFV in the configuration

Current Behavior

When defining (a few?) feature views with a large total number of features (roughly 100 in our case), Pandas raises a PerformanceWarning due to fragmentation. See output:

/Users/user/anaconda3/envs/feast_wo/lib/python3.10/site-packages/feast/on_demand_feature_view.py:216: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()` df[f"{feature.name}"] = pd.Series(dtype=dtype)

Steps to reproduce

Create a few FeatureViews (I believe it's not the contributing factor) with a total of roughly 100 Features and add a simple transformation inside an on_demand_feature_view configuration. Run feast apply and observe the warning

Specifications

• Version: 0.17.0
• Platform: MacOS 12.1
• Subsystem: Python 3.10, pandas 1.3.5

Possible Solution

As noted in the warning message above:
Consider joining all columns at once using pd.concat(axis=1) instead

[felixwang9817]

@roy651 thanks for reporting this issue! This might just be an issue with your specific environment: I just created a feature repo with 100 FVs and a single simple ODFV and didn't see any issues on feast apply. I'm on Feast 0.17, MacOS 11.4, Python 3.7, and pandas 1.1.5.

Is this currently blocking your usage of Feast, or is it just a warning?

[roy651]

@felixwang9817 it's possible that it's a local issue for me, although I doubt it, since the warning comes from within on_deman_feature_view.py right here and this code looks quite generic and not dependent on any local configuration.
BTW, If anything, I suggest testing it again with an updated Pandas/Python. This warning might be coming from a later version (my environment is Python 3.10 and Pandas 1.3.5)

Anyways - You're right. It's not blocking me right now but I thought I'll log it incase someone else encounters it in the future.
Feel free to close it.
Thanks!

[felixwang9817]

@roy651 got it - thanks for reporting this warning! I'll close the issue for now.