Upgrade azureml-pipeline (and others) pyarrow dependency to at least >2.0.0

[brunocous]

Currently, the azureml-pipeline pip package (version 1.23.0) and others requires pyarrow to be (>=0.17.0,<2.0.0).

Now, for my application I require pyarrow features which are available only after 2.0.0.

pip install --upgrade pyarrow after installing azureml-pipeline results in an failure resolving the versions.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
azureml-pipeline 1.23.0 requires pyarrow<2.0.0,>=0.17.0, but you have pyarrow 3.0.0 which is incompatible.

The fix is: updating the dependency list of the affected azureml packages.

[yikei]

Hi @brunocous - Thank you for your feedback, we will look into loosening the upper bound of pyarrow after conducting thorough validation on whether newer versions have breaking changes on AzureML SDK or not.

In the meantime, even though pip shows an error about the incompatibility, the pip install --upgrade pyarrow should still succeed with Successfully installed pyarrow-3.0.0. Does this unblock your scenario, or do you run into other issues after upgrading pyarrow?

[brunocous]

There are indeed workarounds. Forcing the install of a recent pyarrow works in some setups (notebooks etc).

Our use-case consists of defining an Azure ML Pipeline with multiple PythonScriptSteps. In order to specify the dependencies, a RunConfiguration object is created with a CondaDependencies object to point to a conda_env.yaml file. Azure ML builds a container image with Python dependencies with whatever is in that file.
The problem is that conda is very strict in the way it resolves dependency conflicts. In our case, the building process is freezes after a while, resulting in a failed build.
A solution could be to built our own image, but then we need to manage our own containers, losing the "managed" experience of Azure ML.

[yikei]

Hi @brunocous , thanks for the additional information. Totally makes sense that resolving conda dependencies from an env file would encounter issues. We will look into updating our dependencies safely, and reply here once we have more updates on that.

[yikei]

Hi @brunocous , in our recent release, we have increased the pyarrow dependency upper bound to allow pyarrow < 4.0.0. Thanks!

[brunocous]

Yes, I noticed it! Thank you @yikei and the Azure Dev team :)

[poojithag554]

I was trying to import transformers in AzureML designer pipeline, it says for importing transformers and datasets the version of pyarrow needs to >=3.0.0, but then after upgrading pyarrow's version to 3.0.0 and importing transformers pyarrow version is reset to original version of 0.16.0. attaching few error samples. please have a look.

Got exception when invoking script: 'RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):To use datasets, the module pyarrow>=3.0.0 is required, and the current version of pyarrow doesn't match this condition.If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of pyarrow.' azureml-designer-core 0.0.68 requires pyarrow==0.16.0, but you'll have pyarrow 3.0.0 which is incompatible.

[yikei]

Hi @poojithag554 , I recommend opening a new issue to get attention from the right people. This seems to be a dependency specified by azureml-designer-core, which is different from the dependency originally raised in this issue.

[sushmit86]

@yikei I am having similar conflicts with snowflake[pandas] and azureml-dataset-runtime==1.40.0? What should be the right place to raise the issue?

[yikei]

Hi @sushmit86 , it looks like there is an open issue here: #1698
But unfortunately it hasn't gotten attention. I'll try to direct someone to look at that issue.