Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix as_spark_type to not support "bigint". #2011

Merged
merged 2 commits into from
Jan 19, 2021
Merged

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented Jan 16, 2021

Fix as_spark_type to not support "bigint".

The string "bigint" is not recognizable by np.dtype and it causes an unexpected error:

>>> import numpy as np
>>> from databricks.koalas.typedef import as_spark_type
>>> as_spark_type(np.dtype("datetime64[ns]"))
Traceback (most recent call last):
...
TypeError: data type "bigint" not understood

Also, it doesn't work in pandas:

>>> pd.Series([1, 2, 3], dtype="bigint")
Traceback (most recent call last):
...
TypeError: data type "bigint" not understood

@codecov-io
Copy link

codecov-io commented Jan 16, 2021

Codecov Report

Merging #2011 (f78c76f) into master (26e5a2d) will decrease coverage by 0.04%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2011      +/-   ##
==========================================
- Coverage   94.58%   94.54%   -0.05%     
==========================================
  Files          50       50              
  Lines       11192    11200       +8     
==========================================
+ Hits        10586    10589       +3     
- Misses        606      611       +5     
Impacted Files Coverage Δ
databricks/koalas/typedef/typehints.py 90.32% <44.44%> (-3.65%) ⬇️
databricks/koalas/base.py 97.20% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26e5a2d...f78c76f. Read the comment docs.

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@itholic itholic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ueshin ueshin changed the title Fix as_spark_type to handle bigint properly and make it deprecated. Fix as_spark_type to not support "bigint". Jan 19, 2021
@ueshin
Copy link
Collaborator Author

ueshin commented Jan 19, 2021

Thanks! merging.

@ueshin ueshin merged commit 0e39097 into databricks:master Jan 19, 2021
@ueshin ueshin deleted the bigint branch January 19, 2021 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants