Skip to content

Commit

Permalink
update conftest for backwards compat and new api for to parquet call (#…
Browse files Browse the repository at this point in the history
…1784)

Co-authored-by: Karl Higley <[email protected]>
  • Loading branch information
jperez999 and karlhigley authored Mar 17, 2023
1 parent 65bbe86 commit f726194
Showing 1 changed file with 19 additions and 2 deletions.
21 changes: 19 additions & 2 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,25 @@ def datasets(tmpdir_factory):
half = int(len(df) // 2)

# Write Parquet Dataset
df.iloc[:half].to_parquet(str(datadir["parquet"].join("dataset-0.parquet")), chunk_size=1000)
df.iloc[half:].to_parquet(str(datadir["parquet"].join("dataset-1.parquet")), chunk_size=1000)
cudf_version = 0
if cudf:
cudf_version = cudf.__version__.split(".")[:2]
cudf_version = float(".".join(cudf_version))

if cudf_version > 22.10:
df.iloc[:half].to_parquet(
str(datadir["parquet"].join("dataset-0.parquet")), row_group_size_rows=5000
)
df.iloc[half:].to_parquet(
str(datadir["parquet"].join("dataset-1.parquet")), row_group_size_rows=5000
)
else:
df.iloc[:half].to_parquet(
str(datadir["parquet"].join("dataset-0.parquet")), chunk_size=1000
)
df.iloc[half:].to_parquet(
str(datadir["parquet"].join("dataset-1.parquet")), chunk_size=1000
)

# Write CSV Dataset (Leave out categorical column)
df = df[allcols_csv] # Set deterministic column order before write
Expand Down

0 comments on commit f726194

Please sign in to comment.