Python 3.9 conda build checksums are incorrect for v0.10.0

[lmmx]

🐛 Bug

I just got a conda ChecksumMismatchError and thought I should report it (I've never seen this in all my times installing torchvision envs).

I downloaded and checked the checksums of all the builds on conda (using the JSON output from conda search --json -i torchvision -c pytorch -c nvidia) and found it only affects 3 builds, all of which are torchvision 0.10.0 and Python 3.9:

• py39_cpu
• py39_cu102
• py39_cu111

To Reproduce

conda create -n torchvisionbug
conda activate torchvisionbug
conda install -y "cudatoolkit<11.2" -c conda-forge # 11.1.1

⇣

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/louis/miniconda3/envs/torchvisionbug

  added / updated specs:
    - cudatoolkit[version='<11.2']


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _openmp_mutex-4.5          |            1_gnu          22 KB  conda-forge
    libgcc-ng-11.1.0           |       hc902ee8_8         908 KB  conda-forge
    libgomp-11.1.0             |       hc902ee8_8         427 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.3 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
  cudatoolkit        conda-forge/linux-64::cudatoolkit-11.1.1-h6406543_8
  libgcc-ng          conda-forge/linux-64::libgcc-ng-11.1.0-hc902ee8_8
  libgomp            conda-forge/linux-64::libgomp-11.1.0-hc902ee8_8
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.1.0-h56837e0_8



Downloading and Extracting Packages
libgcc-ng-11.1.0     | 908 KB    | ############################################### | 100% 
_openmp_mutex-4.5    | 22 KB     | ############################################### | 100% 
libgomp-11.1.0       | 427 KB    | ############################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: / By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

done

conda install -y pytorch torchvision -c pytorch # get Python 3.9

⇣

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/louis/miniconda3/envs/torchvisionbug

  added / updated specs:
    - pytorch
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    torchvision-0.10.0         |       py39_cu111         8.3 MB  pytorch
    ------------------------------------------------------------
                                           Total:         8.3 MB

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.7.5-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.5.30-py39h06a4308_0
  ffmpeg             pytorch/linux-64::ffmpeg-4.3-hf484d3e_0
  freetype           pkgs/main/linux-64::freetype-2.10.4-h5ab3b9f_0
  gmp                pkgs/main/linux-64::gmp-6.2.1-h2531618_2
  gnutls             pkgs/main/linux-64::gnutls-3.6.15-he1e5248_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.3.0-h06a4308_3350
  jpeg               pkgs/main/linux-64::jpeg-9b-h024ee3a_2
  lame               pkgs/main/linux-64::lame-3.100-h7b6447c_0
  lcms2              pkgs/main/linux-64::lcms2-2.12-h3be6417_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libiconv           pkgs/main/linux-64::libiconv-1.15-h63c8f33_5
  libidn2            pkgs/main/linux-64::libidn2-2.3.2-h7f8727e_0
  libpng             pkgs/main/linux-64::libpng-1.6.37-hbc83047_0
  libtasn1           pkgs/main/linux-64::libtasn1-4.16.0-h27cfd23_0
  libtiff            pkgs/main/linux-64::libtiff-4.2.0-h85742a9_0
  libunistring       pkgs/main/linux-64::libunistring-0.9.10-h27cfd23_0
  libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0
  libwebp-base       pkgs/main/linux-64::libwebp-base-1.2.0-h27cfd23_0
  lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1
  mkl                pkgs/main/linux-64::mkl-2021.3.0-h06a4308_520
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  nettle             pkgs/main/linux-64::nettle-3.7.3-hbbd107a_1
  ninja              pkgs/main/linux-64::ninja-1.10.2-hff7bd54_1
  olefile            pkgs/main/noarch::olefile-0.46-py_0
  openh264           pkgs/main/linux-64::openh264-2.1.0-hd408876_0
  openjpeg           pkgs/main/linux-64::openjpeg-2.3.0-h05c96fa_1
  openssl            pkgs/main/linux-64::openssl-1.1.1k-h27cfd23_0
  pillow             pkgs/main/linux-64::pillow-8.3.1-py39h2c7a002_0
  pip                pkgs/main/linux-64::pip-21.2.2-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.6-h12debd9_0
  pytorch            pytorch/linux-64::pytorch-1.9.0-py3.9_cuda11.1_cudnn8.0.5_0
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-52.0.0-py39h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.36.0-hc218d9a_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  torchvision        pytorch/linux-64::torchvision-0.10.0-py39_cu111
  typing_extensions  pkgs/main/noarch::typing_extensions-3.10.0.0-pyh06a4308_0
  tzdata             pkgs/main/noarch::tzdata-2021a-h52ac0ba_0
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
  zstd               pkgs/main/linux-64::zstd-1.4.9-haebb681_0



Downloading and Extracting Packages
torchvision-0.10.0   | 8.3 MB    | ############################################### | 100% 

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2'.
  download saved to: /home/louis/miniconda3/pkgs/torchvision-0.10.0-py39_cu111.tar.bz2
  expected md5: 78b4c927e54b06d7a6d18eec8b3f2d18
  actual md5: 69dd8411c573903db293535017742bd9

Expected behavior

The checksum should not be mismatched.

I can also reproduce manually:

wget https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2
md5sum torchvision-0.10.0-py39_cu111.tar.bz2

⇣

69dd8411c573903db293535017742bd9  torchvision-0.10.0-py39_cu111.tar.bz2

from subprocess import run
from json import loads
from hashlib import md5
import requests
command = "conda search --json -i torchvision -c pytorch -c nvidia".split(" ")
json_str = run(command, capture_output=True).stdout
json_dict = loads(json_str)
tv_builds = json_dict.get("torchvision")
builds = [d for d in tv_builds if d.get("build") == "py39_cu111"]
for b in builds:
    resp = requests.get(b["url"])
    md5_expected = b["md5"]
    build_name = b["build"]
    checksum = md5(resp.content)
    md5_hex = c.hexdigest()
    print(f"{build_name}: {md5_expected == md5_hex=}")

⇣

py39_cu111: md5_expected == md5_hex=True
py39_cu111: md5_expected == md5_hex=False
py39_cu111: md5_expected == md5_hex=False

Eventually re-ran it on every build and re-ran that with more info:

>>> pprint({b["version"]: (b["url"], b["verified_md5"]) for b in tv_builds if b["build"] == "py39_cu111"})
{'0.10.0': ('https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2',
            False),
 '0.9.0': ('https://conda.anaconda.org/pytorch/linux-64/torchvision-0.9.0-py39_cu111.tar.bz2',
           True),
 '0.9.1': ('https://conda.anaconda.org/pytorch/linux-64/torchvision-0.9.1-py39_cu111.tar.bz2',
           True)}

Essentially ran this:

for b in tv_builds:
    resp = requests.get(b["url"])
    md5_expected = b["md5"]
    build_name = b["build"]
    checksum = md5(resp.content)
    md5_hex = checksum.hexdigest()
    print(f"{build_name}: {md5_expected == md5_hex=}")

This takes an awfully long time though, so I rewrote it as async (but I'll leave it in this gist rather than paste the entire thing here)

bad_builds = [d for d in tv_builds if d.get("verified_md5") is False]
pprint({b["fn"]: {"version": b["version"], "build": b["build"], "url": b["url"], "correct_md5": b["verified_md5"]} for b in bad_builds})

⇣

{'torchvision-0.10.0-py39_cpu.tar.bz2': {'build': 'py39_cpu',
                                         'correct_md5': False,
                                         'url': 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cpu.tar.bz2',
                                         'version': '0.10.0'},
 'torchvision-0.10.0-py39_cu102.tar.bz2': {'build': 'py39_cu102',
                                           'correct_md5': False,
                                           'url': 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu102.tar.bz2',
                                           'version': '0.10.0'},
 'torchvision-0.10.0-py39_cu111.tar.bz2': {'build': 'py39_cu111',
                                           'correct_md5': False,
                                           'url': 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2',
                                           'version': '0.10.0'}}

So this can be safely considered a PyTorch 0.10.0 and Python 3.9 specific problem.

Hope this was not too much irrelevant info I just wanted to be sure it wasn't a broader issue since it effectively causes an outage (the package installation aborts)

[vadimkantorov]

I guess, this is continuation of hotfix for #4240 by @malfet ?

[seemethere]

I've gone ahead and deleted the old packages and the hash mismatch appears to have been resolved, feel free to re-open the issue if still persists, see pytorch/pytorch#63006 (comment)