compute() no longer works as expected from dbplyr 2.3.3

[nviets]

Issue originally reporting at: sparklyr/sparklyr#3375 and redirected to dbplyr. Prior to 2.3.3 compute() worked in the sparklyr code below:

library(dplyr)
library(dbplyr)
library(arrow)
library(sparklyr)
library(DBI)

spark_disconnect_all()
conf <- spark_config()
conf$spark.yarn.queue <- "myqueue"
cn <- spark_connect(master = "yarn-client", config = conf, app_name = "snap_test")

tst <- tbl(cn, "mydb.mytable") %>%
head(200) %>%
compute()

Error in db_save_query.DBIConnection():
! Can't save query to "dbplyr_001".
Caused by error:
! org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE OR REPLACE TEMPORARY VIEW \n"dbplyr_001"'(line 2, pos 0)
...

Full error log and session details are available in the original issue.

@edgararuiz suggested we need in_schema() in the tbl() statement for the table name to parse correctly, but this was not necessary prior to 2.3.3. The shorthand in tbl(cn, "mydb.mytable") is convenient and widely used. Could dbplyr please support it going forward?

@Zhuk66

[nviets]

Minimal example:

iris_spark <- copy_to(sc, iris, "iris", overwrite = T)
tmp1 <- iris_spark %>% tally() %>% collect # works
tmp2 <- iris_spark %>% tally() %>% compute("tmp2") # fails

Error in `db_save_query.DBIConnection()`:
! Can't save query to "tmp2".
Caused by error:
! org.apache.spark.sql.catalyst.parser.ParseException: 
no viable alternative at input 'CREATE OR REPLACE TEMPORARY VIEW \n"tmp2"'(line 2, pos 0)

== SQL ==
CREATE OR REPLACE TEMPORARY VIEW 
"tmp2" AS SELECT COUNT(*) AS `n`
^^^
FROM `iris`
...

[hadley]

I think this should be fixed by #1379.

[nviets]

Thank you! This one was painful. Was #1379 included in 2.4.0? I don't see it in the release notes.

[hadley]

It was — it's mentioned as "Preliminary databricks Sqark SQL backend".