Add Series.all and Series.any #359
Conversation
| # Here we check if the count of `True`s is more than one in order to mimic `any`. | ||
| return sdf.select( | ||
| (F.count(F.when(col.cast('boolean'), 1).otherwise(None)) >= 1) | ||
| ).collect()[0][0] |
There was a problem hiding this comment.
@hvanhovell do you remember this discussion about rewriting for any and every at Apache Spark side? (IIRC you suggested max/min approach but the hole was None handling). Can you quickly check if this rewriting makes sense to you?
Here I need to mimic both here to mimic Pandas' with rewriting since any and every only exist in the Spark's master.
There was a problem hiding this comment.
I can't entirely remember that comment. I did found an e-mail thread about this, and there I state the opposite: ANY(col) can be safely rewritten into MAX(col) you may need to add a coalesce if we need to return false for an empty dataset.
Codecov Report
@@ Coverage Diff @@
## master #359 +/- ##
==========================================
- Coverage 94.4% 94.36% -0.04%
==========================================
Files 36 36
Lines 3646 3656 +10
==========================================
+ Hits 3442 3450 +8
- Misses 204 206 +2
Continue to review full report at Codecov.
|
|
All tests passed |
| # Here we check if the count of `True`s is more than one in order to mimic `any`. | ||
| return sdf.select( | ||
| (F.count(F.when(col.cast('boolean'), 1).otherwise(None)) >= 1) | ||
| ).collect()[0][0] |
There was a problem hiding this comment.
BTW, count(col) ignores null
|
Let me get this in. The output is as expected anyway. |
| # Note that we're ignoring `None`s here for now. | ||
| # Here we check if the count of `True`s is more than one in order to mimic `any`. | ||
| return sdf.select( | ||
| (F.count(F.when(col.cast('boolean'), 1).otherwise(None)) >= 1) |
There was a problem hiding this comment.
Two nits: otherwise(None) should be implied, and if you do a is not empty check why not check > 0?
There was a problem hiding this comment.
Yea, currently None is being simply ignored (that's what Pandas does by default - skipna argument). Yes .. > 0 looks better. Will fix it soon. thanks :D.
There was a problem hiding this comment.
also let me give a shot with min/max ones too
No description provided.