Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how pandas deals with missing values #1748

Merged
merged 1 commit into from
Jul 30, 2021
Merged

Document how pandas deals with missing values #1748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions sdk/python/feast/infra/offline_stores/file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ def evaluate_historical_retrieval():
feature_view.batch_source.created_timestamp_column
)

# Read offline parquet data in pyarrow format
# Read offline parquet data in pyarrow format.
table = pyarrow.parquet.read_table(feature_view.batch_source.path)

# Rename columns by the field mapping dictionary if it exists
Expand All @@ -120,7 +120,11 @@ def evaluate_historical_retrieval():
table, feature_view.batch_source.field_mapping
)

# Convert pyarrow table to pandas dataframe
# Convert pyarrow table to pandas dataframe. Note, if the underlying data has missing values,
# pandas will convert those values to np.nan if the dtypes are numerical (floats, ints, etc.) or boolean
# If the dtype is 'object', then missing values are inferred as python `None`s.
# More details at:
# https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missing
df_to_join = table.to_pandas()

# Make sure all timestamp fields are tz-aware. We default tz-naive fields to UTC
Expand Down
7 changes: 6 additions & 1 deletion sdk/python/feast/infra/utils/aws_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,12 @@ def upload_df_to_redshift(
# Drop the index so that we dont have unnecessary columns
df.reset_index(drop=True, inplace=True)

# Convert Pandas DataFrame into PyArrow table and compile the Redshift table schema
# Convert Pandas DataFrame into PyArrow table and compile the Redshift table schema.
# Note, if the underlying data has missing values,
# pandas will convert those values to np.nan if the dtypes are numerical (floats, ints, etc.) or boolean.
# If the dtype is 'object', then missing values are inferred as python `None`s.
# More details at:
# https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missing
table = pa.Table.from_pandas(df)
column_names, column_types = [], []
for field in table.schema:
Expand Down