Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compute() no longer works as expected from dbplyr 2.3.3 #1372

Closed
nviets opened this issue Oct 3, 2023 · 4 comments
Closed

compute() no longer works as expected from dbplyr 2.3.3 #1372

nviets opened this issue Oct 3, 2023 · 4 comments

Comments

@nviets
Copy link

nviets commented Oct 3, 2023

Issue originally reporting at: sparklyr/sparklyr#3375 and redirected to dbplyr. Prior to 2.3.3 compute() worked in the sparklyr code below:

library(dplyr)
library(dbplyr)
library(arrow)
library(sparklyr)
library(DBI)

spark_disconnect_all()
conf <- spark_config()
conf$spark.yarn.queue <- "myqueue"
cn <- spark_connect(master = "yarn-client", config = conf, app_name = "snap_test")

tst <- tbl(cn, "mydb.mytable") %>%
head(200) %>%
compute()
Error in db_save_query.DBIConnection():
! Can't save query to "dbplyr_001".
Caused by error:
! org.apache.spark.sql.catalyst.parser.ParseException:
no viable alternative at input 'CREATE OR REPLACE TEMPORARY VIEW \n"dbplyr_001"'(line 2, pos 0)
...

Full error log and session details are available in the original issue.

@edgararuiz suggested we need in_schema() in the tbl() statement for the table name to parse correctly, but this was not necessary prior to 2.3.3. The shorthand in tbl(cn, "mydb.mytable") is convenient and widely used. Could dbplyr please support it going forward?

@Zhuk66

@nviets
Copy link
Author

nviets commented Oct 3, 2023

Minimal example:

iris_spark <- copy_to(sc, iris, "iris", overwrite = T)
tmp1 <- iris_spark %>% tally() %>% collect # works
tmp2 <- iris_spark %>% tally() %>% compute("tmp2") # fails
Error in `db_save_query.DBIConnection()`:
! Can't save query to "tmp2".
Caused by error:
! org.apache.spark.sql.catalyst.parser.ParseException: 
no viable alternative at input 'CREATE OR REPLACE TEMPORARY VIEW \n"tmp2"'(line 2, pos 0)

== SQL ==
CREATE OR REPLACE TEMPORARY VIEW 
"tmp2" AS SELECT COUNT(*) AS `n`
^^^
FROM `iris`
...

@hadley
Copy link
Member

hadley commented Nov 2, 2023

I think this should be fixed by #1379.

@hadley hadley closed this as completed Nov 2, 2023
@nviets
Copy link
Author

nviets commented Nov 2, 2023

Thank you! This one was painful. Was #1379 included in 2.4.0? I don't see it in the release notes.

@hadley
Copy link
Member

hadley commented Nov 3, 2023

It was — it's mentioned as "Preliminary databricks Sqark SQL backend".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants