New streaming engine incorrectly sets sorted flag for lexical categorical in partition_by

[coastalwhite]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import os
import polars as pl

os.environ['POLARS_AUTO_NEW_STREAMING'] = '1'

df = pl.DataFrame({"x": [0, 1, 1], "y": ["B", "B", "A"]}).with_columns(
    pl.col("y").cast(pl.Categorical(ordering="lexical"))
)

second_part = df.partition_by("x")[1]
print(second_part._to_metadata('y', 'sorted_asc'))
print(second_part)

Log output

shape: (1, 2)
┌─────────────┬────────────┐
│ column_name ┆ sorted_asc │
│ ---         ┆ ---        │
│ str         ┆ bool       │
╞═════════════╪════════════╡
│ y           ┆ true       │
└─────────────┴────────────┘
shape: (2, 2)
┌─────┬─────┐
│ x   ┆ y   │
│ --- ┆ --- │
│ i64 ┆ cat │
╞═════╪═════╡
│ 1   ┆ B   │
│ 1   ┆ A   │
└─────┴─────┘

Issue description

Sorted flag is incorrectly set.

Expected behavior

Sorted flag should not be set.

Installed versions

Replace this line with the output of pl.show_versions(). Leave the backticks in place.

[coastalwhite]

Another possibly related issue.

import os
import polars as pl

os.environ['POLARS_AUTO_NEW_STREAMING'] = '1'

df = pl.DataFrame(
    {"cats": ["z", "z", "k", "a", "b"], "vals": [3, 1, 2, 2, 3]}
).with_columns(
    pl.col("cats").cast(pl.Categorical("lexical")),
)

out = df.sort(["cats"])
print(out)

shape: (5, 2)
┌──────┬──────┐
│ cats ┆ vals │
│ ---  ┆ ---  │
│ cat  ┆ i64  │
╞══════╪══════╡
│ z    ┆ 3    │
│ z    ┆ 1    │
│ k    ┆ 2    │
│ a    ┆ 2    │
│ b    ┆ 3    │
└──────┴──────┘

[siddharth-vi]

Related issue - #19900

[etiennebacher]

This seems fixed on main:

CategoricalRemappingWarning: Local categoricals have different encodings, expensive re-encoding is done to perform this merge operation. Consider using a StringCache or an Enum type if the categories are known in advance

shape: (1, 2)
┌─────────────┬────────────┐
│ column_name ┆ sorted_asc │
│ ---         ┆ ---        │
│ str         ┆ bool       │
╞═════════════╪════════════╡
│ y           ┆ false      │
└─────────────┴────────────┘
shape: (2, 2)
┌─────┬─────┐
│ x   ┆ y   │
│ --- ┆ --- │
│ i64 ┆ cat │
╞═════╪═════╡
│ 1   ┆ B   │
│ 1   ┆ A   │
└─────┴─────┘

Note there's a TODO in tests/unit/operations/namespaces/test_categorical.py with a link to this issue and a pytest.mark.may_fail_auto_streaming.

Edit: also one pytest.mark in tests/unit/datatypes/test_categorical.py

[siddharth-vi]

Seems like it got fixed with this #20414