names(.SD) should work

[ColeMiller1]

Closes #795. Towards #3189.

This is the proposed implementation:

dt[, names(.SD) := lapply(.SD, '*', 5)]

• Update Reference Semantics Vignette
• Update .SD Usage Vignette
• Update assign help
• Include tests

TO DO after release:
Update StackOverflow post to mirror updated .SD vignette.

[jangorecki]

my vote is for only names(.SD) in LHS and not .SD
even if we decide to use just .SD on LHS then still names(.SD) should works as well

[ColeMiller1]

Thank you, Jan. I am working on better test cases that do not involve logicals. I will update tonight.

I also noticed that #4031 would be partially addressed:

dt = data.table(iris)
keep = c('Species', 'Sepal.Width')
dt[, .SD := NULL, .SDcols = !keep]

I will try to substitute names(.SD) with the sdvars - that would allow solutions like yours setdiff(names(.SD), keep) work and would provide the option of paste0(names(.SD), 'max') so you can update all of the columns without having to use .SDcols or an intermediate character vector.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.50%. Comparing base (958e3dd) to head (8fe60ee).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4163   +/-   ##
=======================================
  Coverage   97.50%   97.50%           
=======================================
  Files          80       80           
  Lines       14884    14884           
=======================================
  Hits        14513    14513           
  Misses        371      371           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ColeMiller1]

This should be good to go unless we want to support ncol(.SD) or length(.SD) which may be expected to work on the LHS of :=. I could see that being useful for lapply with multiple functions.

Regarding codecov, I am not sure what to do. There are 10 new tests to address 7 lines of new code plus all previous tests using lhs := rhs would also use this code.

[jangorecki]

Some minor comments.
Support for length(.SD), ncol(.SD) would be nice but that would generally require a different approach, probably an temporary object relating to subset of self. Current approach is fine for most of use cases.
Thing to decide IMO is if we really want to support .SD on LHS which breaks consistency about column names there. My vote goes for an extra names() to be necessary, so we maintain consistency in API.
We also have .SD related vignette so would be good to put some examples there, but that we can always do as a follow-up.

[ColeMiller1]

Should be good again. The .SD vignette could be updated with a few items such as patterns() and the ability of using functions. I am happy to update the .SD vignette with this PR, but am unsure since they were done outside of this PR.

But there are two examples that this and Michael's recent PR will work great with:

fkt_idx = which(sapply(Teams, is.factor))
Teams[, (fkt_idx) := lapply(.SD, as.character), .SDcols = fkt_idx]
## to
Teams[, names(.SD) := lapply(.SD, as.character), .SDcols = is.factor]

##From 
team_idx = grep('team', names(Teams), value = TRUE)
Teams[, (team_idx) := lapply(.SD, factor), .SDcols = team_idx]
## To
Teams[, names(.SD) := lapply(.SD, factor), .SDcols = patterns('team')]

[MichaelChirico]

Friendly ping :) would be great to get this merged!

[ColeMiller1]

I will make changes tonight or by the end of tomorrow night. Thanks for going through everything.

Cole

[ColeMiller1]

I will give the vignette's another look tomorrow and give you a final ping when I'm done.

[MichaelChirico]

A simpler implementation might be the following:

e <- copyenv(parent.frame()) # pseudocode
e$.SD <- setNames(logical(length(sdvars)), sdvars) # or vector("list"), or even vector("raw") to really scrimp on storage
lhs = eval(lhs, e, e)

WDYT?

[MichaelChirico]

Something I can imagine will happen soon is a user trying:

x[, names(.SD)[1:5] := ...]

Does that work already? If so, please add a test. If not, no need to handle it until it's requested later unless you see an easy fix.

[MichaelChirico]

Looks great! Last thing is checking if we can simplify implementation / possibly make it much more general.

[ColeMiller1]

I attempted to make it more general per your suggestion. I copied the implementation below. All the tests pass and it supports something like dt[, names(.SD)[1:5] :=...]. After the commit, I checked to see if base::names(.SD) works and it would as well.

It's easy to revert back or to go down a route you suggested as well. Let me know and I add a base::names(.SD) test, too, depending on how we go.

# i.e lhs is names(.SD) || setdiff(names(.SD), cols) || (cols)

lhs = eval(lhs, list(.SD = setNames(logical(length(sdvars)), sdvars)), parent.frame())

[TysonStanley]

I'm a personal fan of the ability to use dt[, names(.SD)[1:5] :=...]. Would be a really nice feature to have that flexibility.

[MichaelChirico]

New version looks great, awesome!

[MichaelChirico]

Almost 10 years later, it's a one-line change 😎

Great stuff!