Upgrade to Python 3.8

[metrizable]

Upgrade to Python 3.8

To provide additional features from Python 3.8 to our users, and since Python 3.7 will no longer receive security updates as of 2023-06-27, Colab has upgraded to Python 3.8 (with plans to upgrade to more modern versions in the future). Additionally, this upgrade provides benefits for the packages we pre-install in our runtime, as many can only be upgraded with a newer Python version.

Colab’s fallback runtime version

Using the fallback runtime version temporarily allows access to the last version of the runtime before the upgrade described above, and is available until mid-December. Its purpose is to provide a temporary mechanism for users to more smoothly upgrade their notebooks to be compatible with Colab’s current runtime version. This is available from the Command Palette via the "Use fallback runtime version" command when connected to a runtime. Of note, this setting does not persist across sessions – the command will need to be invoked on each new session.

[CbroLab]

Hey, thanks for the update/upgrade, but some prebuild binary packages are no longer available like
!pip install talib-binary
import talib as ta
could you help to upgrade them asap for users smooth migration? Thx!

[metrizable]

Thanks for reaching out! We're excited to bring a more modern Python version into Colab's runtime.

I see that you are attempting to install talib-binary via pip from pypi. It appears that talib-binary has not had a recent package uploaded to pypi since 2019 and the last one published is an old version that doesn't support Python 3.8.

It appears more recent versions of the source package in github support up to Python 3.9. I would recommend reaching out to the package maintainers and/or file a github issue to see if they are able to publish a newer version of the binary version of the package to pypi.

In the meantime, you can try Colab's fallback runtime version mentioned in the OP, or try installing a more recent version of the TA-lib package in pypi that supports Python 3.8+ which the binary package appears to be based on. Following the install instructions from the README, you can download, make, and install ta-lib like:

%%shell
wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar -xzf ta-lib-0.4.0-src.tar.gz
cd ta-lib/
./configure --prefix=/usr
make
sudo make install

Then !pip install TA-lib.

I went ahead and tried this myself, and although YMMV, I was able to import talib without issue.

[CbroLab]

Really appreciate for the fast response, and also like your answer! The main reason I posted here is installing the binary wheel only takes 0.3s but doing all c make stuff takes over minutes. This library is a python c++ scientific calculation python wrapper, and I believe there are many similar packages and used by a lot of colab users and those package's maintainers wouldn't upgrade them pretty often. If colab users can select any python version by themselves it will be way better to upgrade everything everytime.

[francesco-ballarin]

Thanks @metrizable for the upgrade!

For future reference, can the Colab team work out a better way to announce such breaking changes?
The amount of issues that get opened weekly in this repo is so high that you cannot expect developers to be subscribed ("watch") to this repo with issue notification.

Since the change actually occured in googlecolab/backend-info@c43ed71, maybe the backend-info issue tracker (which has negligible activity) can be used for that?

[mco-gh]

@francesco-ballarin We've recently started a public blog. Announcements about new releases, feature updates, etc. will be publicized there and the posting frequency is low enough (1-2 article per week) that it won't swamp your inbox. I'd suggest subscribing to email updates from that blog to stay on top of things going forward.

[francesco-ballarin]

Thanks @mco-gh , that's exactly what I was looking for! :)

[LyWangPX]

In the current TPU runtime ( I am using pro+ ), I cannot find the option in Command Palette to use fallback.
In the meantime, all wheels of pytorch-xla only support python 3.7 on colab.
Is there any workaround on the colab side such as using a fallback version? ( current pytorch-xla's tpu vm python3.8 wheel needs higher version of Debian)

[metrizable]

@LyWangPX Thanks for the report and thank you for using Colab.

I understand that you're trying to install pytorch_xla using pip from a compiled wheel provided by the PyTorch team, but one for Colab is not available for Python 3.8. I see that there are several pre-compiled wheels in the typical location for tpu-pytorch, but none that suit your purpose (or they are marked "tmp" or "test*").

I'd recommend filing an upstream issue requesting publication of a pre-compiled wheel of PyTorch-XLA for Colab.

In the meantime, when connected to a runtime, "Use fallback runtime version" is available from the Command Palette. I just invoked it myself and have confirmed that I get the last runtime version using Python 3.7 on TPU:

On a related note, there is a forthcoming upgrade of the PyTorch libraries on Colab that can be followed here. Requests for pre-compiled PyTorch-XLA wheels would ideally also include a version compatible with these.

[LyWangPX]

Thanks for your explanation!
My bad, I did not pay attention to "when connected to a runtime". Now I can see the option.
There is already a related issue on xla's repo. For now, I will use the fallback version and wait for the upgrade of the wheel.

[Andrej730]

Is there way to still use Python 3.7? I tried to change kernelspec but had no success, colab is automatically switching to python 3.8.

"kernelspec": {
      "display_name": "Python 3.7",
      "name": "py37"
}

Also tried to

!sudo apt-get install python3.7
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 0
!sudo update-alternatives --config python3 # choose python 3.7 here

But colab is still using 3.8 as default.

[mco-gh]

Check the section on "Colab’s fallback runtime version" in this article.

[Andrej730]

Check the section on "Colab’s fallback runtime version" in this article.

it does work, thank you! Any ideas how to switch to python 3.7 after this command will revoked?

[g-i-o-r-g-i-o]

What kind of problems could arise from using a current python version? It seems a thing that only Debian would do.

[namurphy]

Thank you for doing this...I'm excited about this update!

Packages in the scientific pythoniverse often follow the deprecation schedule in NumPy Enhancement Proposal 29, which means a lot of packages will only support Python feature releases that are ≤42 months old. For example, releases of NumPy after April 2023 will drop support for Python 3.8. The development branch of these repos might end up dropping support ∼3–6 months earlier. In the future, is the plan for Colab to consistently use a version of Python that is ≲36 months old?

[metrizable]

Great question @namurphy . While we can't commit to a specific policy like the one you suggested, the intent is to continue to offer a contemporary, stable, and secure ecosystem for Colab users. This includes forthcoming upgrades to Python (3.9, 3.10, etc), the Linux distribution (Ubuntu 20.04 LTS, 22.04 LTS, etc.), drivers and libraries (Nvidia, CUDA, cuDNN, etc.), core packages (IPython, notebook, etc.), and user packages (NumPy, SciPy, TensorFlow, PyTorch, JAX, etc.). We're looking forward bringing these upgrades to our users in the future!

[EwoutH]

Thanks for both your effort and open communication @metrizable!

With NumPy and many other scientific packages dropping Python 3.8 in April 2023, do you think upgrading to Python 3.9 is doable before/around then?

[bsteiner-dandy]

Is the fallback runtime still available? I don't see it and am trying to run a notebook that does not work with python 3.8

[Andrej730]

Is the fallback runtime still available? I don't see it and am trying to run a notebook that does not work with python 3.8

according to this article fallback option was available only "until mid-December".

[bsteiner-dandy]

Are there any alternative methods to falling back to python 3.7? I tried the following:

!sudo apt-get update -y

!sudo apt-get install python3.7

!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1

!sudo update-alternatives --config python3

!apt-get install python3-pip

!python -m pip install --upgrade pip --user

But I am getting an error when installing installing an external project via pip:
python setup.py egg_info did not run successfully.

All of this was working with Python 3.7 a couple months ago when I last worked with it

[EwoutH]

@bsteiner-dandy What are you exactly using that works on 3.7 but doesn’t on 3.8?

Python 3.7 will be end-of-life in a few months by the way.

[bsteiner-dandy]

I am using a project with pretrained keras models that cannot be unmarshaled with a newer version of Keras

[rosecongou]

Is the fallback runtime still available? I don't see it and am trying to run a notebook that does not work with python 3.8

according to this article fallback option was available only "until mid-December".

This seems like an exceedingly short time, especially spanning the holidays, to leave the fallback version up.

[rosecongou]

Are there any alternative methods to falling back to python 3.7? I tried the following:

!sudo apt-get update -y

!sudo apt-get install python3.7

!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1

!sudo update-alternatives --config python3

!apt-get install python3-pip

!python -m pip install --upgrade pip --user

But I am getting an error when installing installing an external project via pip: python setup.py egg_info did not run successfully.

All of this was working with Python 3.7 a couple months ago when I last worked with it

Yes, would it be possible to do something about this?