Batch Import, feature with datetime format issue

[budi]

Expected Behavior

• The format used in python different with the one used in import job

Current Behavior

• The sdk is using csv format as staging file before ingesting to feast
• It's using different zone offset format for the timestamp column than the one used in the import job

Steps to reproduce

Using the quickstart:

df = pd.read_csv('./train_head.csv', index_col=False)
df['pickup_datetime'] = pd.to_datetime(df['pickup_datetime'])  # 'pickup_datetime' is a proper datetime column
...

importer = Importer.from_df(df, 
                           entity='test_entity', 
                           owner='[email protected]',  
                           staging_location=STAGING_LOCATION,
                           id_column='entity_id',
                           timestamp_column='pickup_datetime')
fs.run(importer, apply_features=True, apply_entity=True)

Specifications

• Version: latest
• Platform: as per suggested by install guide
• Subsystem: as per suggested by install guide

Possible Solution

The current workaround is:

df['pickup_datetime'] = pd.to_datetime(df['pickup_datetime']).apply(lambda d: d.replace(tzinfo=None))

@romanwozniak

[romanwozniak]

The proper fix would be to convert the timestamp column of the df during a staging into CSV into the format, compatible with the importer job datetime format.