[BUG] Error on als_movielens.ipynb

[Seyoung9304]

Description

Hello :)
I found two bugs on ALS Tutorial.

1. Error at data = movielens.load_spark_df(spark, size=MOVIELENS_DATA_SIZE, schema=schema)

This occurs when I use pyspark==2.3.1.
I used pyspark==2.3.1 because tutorial was based on this version,
but it fails.
But when I use pyspark==3.2.0, it works.

2. Join operation fail at

dfs_pred_exclude_train = dfs_pred.alias("pred").join(
        train.alias("train"),
        (dfs_pred[COL_USER] == train[COL_USER]) & (dfs_pred[COL_ITEM] == train[COL_ITEM]),
        how='outer'
    )

As version 2.3.1 fails, I tried this with pyspark==3.2.0.
However, AnalysisException occurs.

I think this is also a version issue.

In which platform does it happen?

• Google Colab
• System version: 3.7.12 (GCC 7.5.0)
• Spark version: 2.3.1, 3.2.0

How do we replicate the issue?

Run https://github.com/microsoft/recommenders/blob/main/examples/00_quick_start/als_movielens.ipynb on Colab

Expected behavior (i.e. solution)

Version issues should be resolved.
Join operations should be done without error.

Other Comments

Thank you in advance! :)

[miguelgfierro]

@Seyoung9304 it might be related to this fix we did #1608?

Can you try staging branch and run again?

FYI @simonzhaoms

[Seyoung9304]

Hello!
Now it is working! Thank you so much 👍👍👍
Have a great day :)