Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torch.dot() produce wrong results #180

Open
1 task done
husisy opened this issue Aug 4, 2023 · 7 comments
Open
1 task done

torch.dot() produce wrong results #180

husisy opened this issue Aug 4, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@husisy
Copy link

husisy commented Aug 4, 2023

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

claim: pip install torch gives the correct results, so I think it should be an issue to be reported here.

claim: right now (2023-08-04) i can only reproduce this issue on macOS (I test both M1 and M2 silicon), cannot reproduce on ubuntu-22.04 with cuda.

minimum working example (MWE) to reproduce wrong results

## use conda/micromamba either ONE

conda create -y -n test00
conda install -y -n test00 -c conda-forge python pytorch
conda activate test00

## conda-forge is the default one (usually)
micromamba create -y -n test00
micromamba install -y -n test00 python pytorch
micromamba activate test00
# draft00.py
import torch
x = torch.tensor([2], dtype=torch.complex128)
print(torch.dot(x, x))
python draft00.py
# tensor(0.+0.j, dtype=torch.complex128) ##should be 4

MWE to reproduce correct results

# similar for micromamba
conda create -y -n test01
conda install -y -n test01 -c conda-forge python numpy
# if no numpy, UserWarning is raised
conda activate test01
pip install torch
# draft00.py
import torch
x = torch.tensor([2], dtype=torch.complex128)
print(torch.dot(x, x))
python draft00.py
# tensor(4.+0.j, dtype=torch.complex128)

Installed packages

# packages in environment at /Users/zhangc/miniconda3/envs/test00:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2023.7.22            hf0a4a13_0    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
gmp                       6.2.1                h9f76cd9_0    conda-forge
gmpy2                     2.1.2           py311h2ba9262_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
libblas                   3.9.0           17_osxarm64_openblas    conda-forge
libcblas                  3.9.0           17_osxarm64_openblas    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           12_3_0_hd922786_1    conda-forge
libgfortran5              12.3.0               ha3a6a3e_1    conda-forge
liblapack                 3.9.0           17_osxarm64_openblas    conda-forge
libopenblas               0.3.23          openmp_hc731615_0    conda-forge
libprotobuf               3.21.12              hb5ab8b9_0    conda-forge
libsqlite                 3.42.0               hb31c410_0    conda-forge
libuv                     1.44.2               hb547adb_1    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               16.0.6               h1c12783_0    conda-forge
markupsafe                2.1.3           py311heffc1b2_0    conda-forge
mpc                       1.3.1                h91ba8db_0    conda-forge
mpfr                      4.2.0                he09a6ba_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numpy                     1.25.2          py311hb8f3215_0    conda-forge
openssl                   3.1.2                h53f4e23_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.4          h47c9636_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytorch                   2.0.0           cpu_generic_py311ha38e979_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
sleef                     3.5.1                h156473d_2    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

Environment info

active environment : test00
    active env location : /Users/zhangc/miniconda3/envs/test00
            shell level : 1
       user config file : /Users/zhangc/.condarc
 populated config files : /Users/zhangc/.condarc
          conda version : 23.7.2
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.4.1=0
                          __unix=0=0
       base environment : /Users/zhangc/miniconda3  (writable)
      conda av data dir : /Users/zhangc/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/zhangc/miniconda3/pkgs
                          /Users/zhangc/.conda/pkgs
       envs directories : /Users/zhangc/miniconda3/envs
                          /Users/zhangc/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.7.2 requests/2.31.0 CPython/3.10.12 Darwin/22.5.0 OSX/13.4.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False
@husisy husisy added the bug Something isn't working label Aug 4, 2023
@h-vetinari
Copy link
Member

Thanks for the detailed report. To try to narrow down the problem (in this case: away from OpenBLAS, which does a lot of the internals behind matrix/tensor multiplication), could you try modifying (or recreating) your test00 environment with:

conda install libblas=*=*netlib

and try again?

@husisy
Copy link
Author

husisy commented Aug 4, 2023

thx for the instant reply. i name this environment test02 and the results are still not correct.

conda create -y -n test02
conda install -y -n test02 -c conda-forge python pytorch "libblas=*=*netlib"
# libblas[build=*netlib]
conda activate test02
python draft00.py
# tensor(0.+0.j, dtype=torch.complex128) ##should be 4
❯ conda list
# packages in environment at /Users/zhangc/miniconda3/envs/test02:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2023.7.22            hf0a4a13_0    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
gmp                       6.2.1                h9f76cd9_0    conda-forge
gmpy2                     2.1.2           py311h2ba9262_1    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
libblas                   3.9.0           5_h880f123_netlib    conda-forge
libcblas                  3.9.0           5_h880f123_netlib    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           12_3_0_hd922786_1    conda-forge
libgfortran5              12.3.0               ha3a6a3e_1    conda-forge
liblapack                 3.9.0           5_h880f123_netlib    conda-forge
libprotobuf               3.21.12              hb5ab8b9_0    conda-forge
libsqlite                 3.42.0               hb31c410_0    conda-forge
libuv                     1.44.2               hb547adb_1    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               16.0.6               h1c12783_0    conda-forge
markupsafe                2.1.3           py311heffc1b2_0    conda-forge
mpc                       1.3.1                h91ba8db_0    conda-forge
mpfr                      4.2.0                he09a6ba_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numpy                     1.25.2          py311hb8f3215_0    conda-forge
openssl                   3.1.2                h53f4e23_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.4          h47c9636_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytorch                   2.0.0           cpu_generic_py311ha38e979_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
sleef                     3.5.1                h156473d_2    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

@ngam
Copy link
Contributor

ngam commented Aug 14, 2023 •
edited
Loading

Could you please print torch.__config__.show()? This is likely something wrong in our linking somewhere... perhaps the metal piece. I will check how they're updated things upstream. Wrong.

@husisy
Copy link
Author

husisy commented Aug 14, 2023 

❯ python -c "import torch; print(torch.__config__.show())"
PyTorch built with:
  - GCC 4.2
  - C++ Version: 201703
  - clang 15.0.7
  - OpenMP 201811
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CXX_COMPILER=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1690825982320/_build_env/bin/arm64-apple-darwin20.0.0-clang++, CXX_FLAGS=-ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++  -fmessage-length=0 -isystem /Users/runner/miniforge3/conda-bld/pytorch-recipe_1690825982320/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1690825982320/work=/usr/local/src/conda/pytorch-2.0.0 -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1690825982320/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac=/usr/local/src/conda-prefix -Wno-deprecated-declarations -Wno-unknown-warning-option -Wno-error=unused-command-line-argument -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Winconsistent-missing-override -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wsuggest-override -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-aligned-allocation-unavailable -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-constexpr-not-const -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKLDNN=0, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

@ngam
Copy link
Contributor

ngam commented Aug 14, 2023

Thanks. I can reproduce locally. Similar to this. pytorch/pytorch#64156

@ngam
Copy link
Contributor

ngam commented Aug 14, 2023

Also, pytorch from the pytorch channel has the right results

@ngam
Copy link
Contributor

ngam commented Aug 14, 2023

Both .dot and .matmul are affected, but only for complex-valued numbers in my testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@husisy @h-vetinari @ngam