BUG: jupyter loading of serialized modin objects from ray tasks converts to pandas objects

[eyadgaran]

Modin version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest released version of Modin.

• I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

import ray
ray.init()

@ray.remote
def test():
    import modin.pandas as pd

    df = pd.DataFrame({
        'A': [1, 2, 3],
        'B': [4, 5, 6]
    })

    print(type(df)) # <class 'modin.pandas.dataframe.DataFrame'>
    return df

df = ray.get(test.remote())
print(type(df)) # <class 'pandas.core.frame.DataFrame'>

Issue Description

Hi! i am trying to integrate modin with jupyter environments and noticed unexpectedly that when dataframes are passed to/from ray tasks, they serialize to pandas objects. While this is fine for toy examples, I am concerned about unexpected OOMs popping up during interactive work with large datasets.

Expected Behavior

My expectation is the objects would deserialize to the same objects with whatever cluster refs to continue manipulating the underlying data.

Error Logs

No errors. Just inline type changing.

Installed Versions

INSTALLED VERSIONS

commit : 1c4d173
python : 3.11.9
python-bits : 64
OS : Darwin
OS-release : 23.6.0
Version : Darwin Kernel Version 23.6.0: Wed Jul 31 20:49:39 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_ARM64_T6000
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

Modin dependencies

modin : 0.32.0+6.g1c4d173d
ray : 2.35.0
dask : 2024.10.0
distributed : 2024.10.0

pandas dependencies

pandas : 2.2.3
numpy : 2.1.3
pytz : 2024.2
dateutil : 2.9.0.post0
pip : 24.2
Cython : None
sphinx : None
IPython : 8.29.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.10.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 18.0.0
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.2
qtpy : None
pyqt5 : None

[smit-kiri]

Running into the same issue. This also happens if you pass in a modin dataframe to a ray task, like

import modin.pandas as pd
import ray

@ray.remote
def test(df: pd.DataFrame) -> None:
    print(type(df))    # <class 'pandas.core.frame.DataFrame'>

df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6]
})

print(type(df)) # <class 'modin.pandas.dataframe.DataFrame'>
test.remote(df)