Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbplyr 2.4.0 fails to read CSV from Spark with regexpr error #1387

Closed
tomshafer opened this issue Oct 31, 2023 · 3 comments
Closed

dbplyr 2.4.0 fails to read CSV from Spark with regexpr error #1387

tomshafer opened this issue Oct 31, 2023 · 3 comments
Labels
reprex needs a minimal reproducible example

Comments

@tomshafer
Copy link

tomshafer commented Oct 31, 2023

In conjunction with sparklyr:

dbplyr 2.4.0:

sc <- sparklyr::spark_connect(method = "databricks")
sparklyr::spark_read_csv(sc, path)
Warning: `src_sql()` was deprecated in dbplyr 1.4.0.
ℹ The deprecated feature was likely used in the sparklyr package.
  Please report the issue at <https://github.com/sparklyr/sparklyr/issues>.
Error in purrr::map(clauses, sql_format_clause, lvl = lvl, con = con) : 
  ℹ In index: 2.
Caused by error in `` regexpr("`", x)[[1]] ``:
! subscript out of bounds

dbplyr 2.3.4:

remotes::install_version("dbplyr", version = "2.3.4")

sc <- sparklyr::spark_connect(method = "databricks")
sparklyr::spark_read_csv(sc, path)

(Works fine.)

@hadley
Copy link
Member

hadley commented Nov 2, 2023

cc @edgararuiz

@edgararuiz
Copy link
Collaborator

Hi @tomshafer , what version of sparklyr are you running, I believe that was addressed in version 1.8.4

@hadley hadley added the reprex needs a minimal reproducible example label Nov 2, 2023
@tomshafer
Copy link
Author

Thanks, @edgararuiz! At the time I think it was 1.8.3 as the newer one wasn’t available from CRAN. All seems OK now, with 1.8.4 installed, so I’ll close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
reprex needs a minimal reproducible example
Projects
None yet
Development

No branches or pull requests

3 participants