Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help misuse of options argument. #1402

Merged
merged 2 commits into from
Apr 8, 2020
Merged

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Apr 7, 2020

I have sometimes seen misuse of a keyword argument options in read_xxx/to_xxx functions. E.g.,

kdf = ks.read_csv(..., options={ ... })

In this case, the argument options is actually {'options': { ... }}, which is not what the user wants to do.

We can help those cases by getting the 'options' value.

@ueshin ueshin requested a review from HyukjinKwon April 7, 2020 21:32
@itholic
Copy link
Contributor

itholic commented Apr 7, 2020

This way is not currently supported in pandas, right?

>>> pd.read_csv('test.csv', options={'engine': None})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: parser_f() got an unexpected keyword argument 'options'

But It's LGTM though since I think It will not not harm the experience of the existing pandas users.

@HyukjinKwon
Copy link
Member

It's not supported in pandas because options here means PySpark specific options.

@HyukjinKwon HyukjinKwon merged commit fab9f6f into databricks:master Apr 8, 2020
@ueshin ueshin deleted the options branch April 8, 2020 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants