(NA == NA) should return true and (NA == 1) should return false

[sylvaticus]

Hello, I don't know if this issue will be solved with the moving of DataFrames to Nullable, but the fact that both (NA == NA) and (NA == 1) (ot whichever other value) return NA instead of the expected boolean value is currently at the root of several problems when querying DataFrames.

It makes extremely tedious filters using loops:

function filter(df,c)
    for r in eachrow(df)
        if (isna(c) && isna(r:[c])) || ( !isna(r[:c]) && r[:c] == c   )
        ...

and breaks select-like functionalities in DataFramesMeta.jl and Query.jl when NA values are present or requested for..

EDIT: I did read the nice answer of simonster on Stack Overflow.. still in some situations you want to get exactly the NA values or the values that are NOT NA, maybe introducing an operator explicitly for that instead of == that can be used in querying the data ?

[sylvaticus]

Sorry, you can close this issue.. I learned I can create my own version of "equal" and use it to query..:

function naeq(a,b)
    return (isna(a) && isna(b)) || ( !isna(a) && !isna(b) &&  a == b )
end

test = @where(df, naeq.(:a,"cc"), naeq.(:b,NA) )

[nalimilan]

That's a complex issue, but essentially we've decided to keep this behavior because that's the most correct and safest one. See JuliaData/Missings.jl#33 for references about these discussions.

You don't need your own equal function, just use isequal (which might get its own Unicode operator in Base at some point).