[Docker] Update Dockerfile to force-use transformers main

[younesbelkada]

What does this PR do?

Optimum pins transformers to <4.35.0, leading our docker images to have transformers==4.34.1, which is not intended since we want to test PEFT against transformers and accelerate main

This PR fixes it, there is no need to update the cpu docker file as optimum is not installed there

cc @pacman100 @BenjaminBossan

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[BenjaminBossan]

Making this change should be fine. We are, however, changing what exactly we're testing here, going from latest transformers release to transformers main. We could, theoretically, also do a pip install -U transformers after installing optimum to get the latest transformers release, ignoring the pin from optimum.

If we do opt to go with transformers main, we should show the commit hash to allow us to debug potential failures, e.g. by running pip freeze | grep transformers.

[younesbelkada]

@BenjaminBossan as far as I know we've been always testing on transformers main: https://github.com/huggingface/peft/blob/main/docker/peft-gpu/Dockerfile#L38 except until the latest optimum release

[BenjaminBossan]

Oh, so this is more of a problem of the order in which the packages are installed. Is there a reason why bitsandbytes optimum auto-gptq are installed in a separate block further down below? Maybe they should be installed in the first block, and then we add a second block to install every package that needs to be installed from source? Otherwise, we are prone to run into similar issues in the future if any dependency adds another pin to a version.

[younesbelkada]

Makes sense yes, let me update that as well

[BenjaminBossan]

Thanks for investigating this issue and providing a solution, looks good.

[younesbelkada]

Thanks @BenjaminBossan !