Errors in @filter() with multiple conditions

[christophscheuch]

As I'm compiling a basic set of introductory applications, I uncovered an issue with @filter(): I cannot use logical and / or in filtering the data:

using Tidier
df = DataFrame(a = repeat(1:10), b = repeat('a':'e', inner = 2))

@chain df begin
    @filter(a == 1 & b == "a")
end
# ERROR: MethodError: no method matching &(::Int64, ::Char)

@chain df begin
    @filter(a == 1 && b == "a")
end
# ERROR: TypeError: non-boolean (BitVector) used in boolean context

@chain df begin
    @filter(a == 1 | b == "a")
end
# ERROR: MethodError: no method matching &(::Int64, ::Char)

@chain df begin
    @filter(a == 1 || b == "a")
end
# ERROR: TypeError: non-boolean (BitVector) used in boolean context

Interestingly, the following chunk works, but produces non-meaningful results since it returns an empty data frame (while the result should be the first row).

@chain df begin
    @filter(a == 1, b == "a")
end

I did not find any hints in the documentation. Am I missing something?

[kdpsingh]

This may be a datatype issue. Will check when I get some time. Single quotes denote characters and double quotes denote strings. Would try b == 'a' instead of b == "a".

[christophscheuch]

Ah, I did not realize the difference - thanks! However, the errors are the same, regardless whether I use single or double quotes as you suggested.

To avoid any data type issues, I created another data frame with only integers and get other errors:

df2 = DataFrame(a = repeat(1:10), b = repeat(1:5, inner = 2))

@chain df2 begin
    @filter(a == 1 & b == 1)
end
# ERROR: UndefVarError: a not defined

@chain df2 begin
    @filter(a == 1 && b == 1)
end
# ERROR: TypeError: non-boolean (BitVector) used in boolean context

[kdpsingh]

Thank you for trying it. I will take a look!

[kdpsingh]

Ok, I figured out the issue, which has to do with a bug in Tidier.jl as well as an operator precedence issue in Julia.

The bug: && should be auto-vectorized by Tidier, but it wasn't because its abstract syntax tree looks different from & -- unlike &, && doesn't behave like a function.

The bug is now fixed in a PR. Once it passes tests, will merge.

The operator precedence issue: While == takes on higher precedence than &&, it actually has a lower precedence than &. This means that if using &, you have to write @filter((a == 1) & (b == 1)), which works correctly.

I've added a doctest and have updated the documentation with 3 different ways to handle this situation.

[kdpsingh]

Resolved by #96. See https://tidierorg.github.io/Tidier.jl/dev/examples/generated/UserGuide/filter/ for details.

[kdpsingh]

(the documentation is still re-building so may need to check again in a few mins in case looking now)