Add a @transform! mutating version

[jlperla]

In order to port over Stata and R code, it is nice to have succinct calculations of new columns based on existing values. Right now,

using DataFrames, DataFramesMeta
df = DataFrame(a=[1; 2; 3], b=[10; 12; 13])
f(a, b) = a + b
df =@transform(df, c = f.(:a, :b))

But it would be great if this could instead be:

using DataFrames, DataFramesMeta
df = DataFrame(a=[1; 2; 3], b=[10; 12; 13])
f(a, b) = a + b
@transform!(df, c = f.(:a, :b))

Is this a relatively easy change for someone to add a PR?

[nalimilan]

Yes, that sounds quite easy to do, given that transform calls copy to avoid mutating the input data frame:
https://github.com/JuliaStats/DataFramesMeta.jl/blob/07325ebd1e100945ae217af70529521d37ecb3af/src/DataFramesMeta.jl#L382

You just need to define transform in terms of a new transform! function which would use the existing code except the copy call. Then the corresponding macro will need to be added too.

There's a more tricky case when the argument is a GroupedDataFrame. Probably better throw an error in for that.

[jlperla]

Thanks for the idea. Actually, let me make it even better, since variadic makes a lot of sense here.

using DataFrames, DataFramesMeta
df = DataFrame(a=[1; 2; 3], b=[10; 12; 13])
f(a, b) = a + b
@transform!(df, c = f.(:a, :b), d = :a + :b)

instead of

using DataFrames, DataFramesMeta
df = DataFrame(a=[1; 2; 3], b=[10; 12; 13])
f(a, b) = a + b
df =@transform(df, c = f.(:a, :b), d = :a + :b)

EDIT: Looks like @transform already has the variadic functionality, which makes it even easier. I might see if an RA can work on this issue in the next month or so.

[pdeffebach]

One thing to note is that if you are doing a mutating transforms over and over again, @with is probably a more concise macro to use.

@with df begin 
    df.t = f.(:a, :b)
    df.t = :a + :b
end

[pdeffebach]

This would actually a bit tough for a grouped data frame. Because transform(g::GroupedDataFrame, args...) uses DataFrame(g) in the first line, it's always going to make a copy. Perhaps we could do

d = sort!(g.parent, g.cols)
..all the other stuff

[pdeffebach]

I just created a PR for this, but I think we should have more discussion about whether or not it is the right strategy.

It's nice that this is an opinionated package where there is one way of doing each thing. I might prefer to have just one, albeit copying, macro for each operation than face a splintered API.

Despite opening the PR I might wonder if we can convince users to get used to copying, and make sure that everything is as performant as possible.

[matthieugomez]

Having in-place versions of operations that otherwise do a deep copy of the dataframe would be particularly useful: it would make it easier to work with dataframes with a size close to the maximum amount of RAM. If I understand correctly, this would apply to @orderby and @where.

[pdeffebach]

Yes I agree. Though I haven't heard too many complaints to this effect with dplyr.

I just added a commit in #119 making some more progress. @orderby and @where are on the list

[pdeffebach]

Closed via #216