Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong datatype for timestamp fields #449

Open
Khelied opened this issue Nov 7, 2023 · 3 comments
Open

Wrong datatype for timestamp fields #449

Khelied opened this issue Nov 7, 2023 · 3 comments

Comments

@Khelied
Copy link

Khelied commented Nov 7, 2023

I use Mosaic to load a geopackage into a Spark Dataframe. When loading timestamp fields, something goes wrong and I get the date 1899-11-30 everywhere. Mosaic automatically makes this a "datetime" while this should be a "timestamp". There is no option to load everything as a "string" or define the schema.

This is wat I get:
image

And this is what it need to be:
image

Thank you

@milos-colic
Copy link
Contributor

@Khelied thank you for raising the issue, could you share the code snippet please so that I can reproduce.
I will open a PR to address this for the next release.

@Khelied
Copy link
Author

Khelied commented Nov 7, 2023

@milos-colic hereby my code:

import mosaic as mos
mos.enable_mosaic(spark, dbutils)       # Enable Mosaic
mos.enable_gdal(spark)                  # Enable GDAL
import fiona
import datetime
import os
from pyspark.sql.functions import *
from pyspark.sql.types import *

FilePath = f'/mnt/my_geopackage.gpkg'
gpkg_layer = 'layer_one'

#layers = fiona.listlayers("/dbfs"+FilePath)
#print(layers)

path, filename = os.path.split(FilePath)
print(filename)
print(path)

spark_gpkg_df = (mos.read().format("multi_read_ogr")
    .option("driverName", "GPKG")
    .option("layerName", gpkg_layer)
    .option("asWKB", "true")
    .option("chunkSize", "500000")
    .load(FilePath)
)

@Khelied
Copy link
Author

Khelied commented Dec 14, 2023

@milos-colic Is there an indication when this will be resolved? Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants