fix(udf): udf call with empty table and batch size #3604
Conversation
daft/udf.py
Outdated
| # Don't call UDF with empty tables | ||
| # TODO: fix assumption that all input lengths are the same | ||
| if len(evaluated_expressions[0]) == 0: | ||
| return Series.empty(return_dtype, name)._series |
There was a problem hiding this comment.
technically it's a little wrong to do this, since a UDF might want to have some special logic for when the incoming data is empty... For example aggregation UDFs.
Any thoughts on how this might affect usage of UDFs in map_groups?
There was a problem hiding this comment.
Oh you're right, although I don't think it affects map_groups since each group would have at least one element, otherwise it would not be included. However this is probably wrong behavior if we ever build some sort of global aggregation UDF.
I will handle the logic in another manner then.
There was a problem hiding this comment.
@jaychia went with a new solution, PTAL!
CodSpeed Performance ReportMerging #3604 will degrade performances by 14.42%Comparing Summary
Benchmarks breakdown
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3604 +/- ##
==========================================
- Coverage 77.80% 77.12% -0.68%
==========================================
Files 718 718
Lines 88176 89686 +1510
==========================================
+ Hits 68607 69174 +567
- Misses 19569 20512 +943
|
No description provided.