Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New debug logs present in 3.28.0 #5095

Closed
2 tasks done
JeanChristopheMorinPerso opened this issue Dec 5, 2023 · 1 comment
Closed
2 tasks done

New debug logs present in 3.28.0 #5095

JeanChristopheMorinPerso opened this issue Dec 5, 2023 · 1 comment
Labels
locked [bot] locked due to inactivity source::anaconda created by members of Anaconda, Inc. type::bug describes erroneous operation, use severity::* to classify the type

Comments

@JeanChristopheMorinPerso
Copy link
Contributor

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

conda-build 3.28.0 fixed a problem where there was debug logs like DEBUG:conda_index.index were printed to stdout.

Unfortunately, 3.28.0 introduces new debug logs.

(base) jcmorin-mac-anac:aggregate jcmorin$ conda-build jcmorin/transformers-feedstock --python 3.11
/Users/jcmorin/miniconda/bin/conda-build:11: DeprecationWarning: conda_build.cli.main_build.main is deprecated and will be removed in 24.1.0. Use `conda build` instead.
  sys.exit(main())
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Adding in variants from /Users/jcmorin/anaenv/aggregate/conda_build_config.yaml
Adding in variants from config.variant
Attempting to finalize metadata for transformers
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
BUILD START: ['transformers-4.31.0-py311hca03da5_0.tar.bz2']
DEBUG:chardet.charsetprober:SHIFT_JIS Japanese prober hit error at byte 2124
DEBUG:chardet.charsetprober:EUC-JP Japanese prober hit error at byte 1956
DEBUG:chardet.charsetprober:GB2312 Chinese prober hit error at byte 2127
DEBUG:chardet.charsetprober:EUC-KR Korean prober hit error at byte 1956
DEBUG:chardet.charsetprober:CP949 Korean prober hit error at byte 1956
DEBUG:chardet.charsetprober:Big5 Chinese prober hit error at byte 1957
DEBUG:chardet.charsetprober:EUC-TW Taiwan prober hit error at byte 1956
DEBUG:chardet.charsetprober:windows-1251 Russian confidence = 0.01
DEBUG:chardet.charsetprober:KOI8-R Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-5 Russian confidence = 0.0
DEBUG:chardet.charsetprober:MacCyrillic Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM866 Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM855 Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-7 Greek confidence = 0.0
DEBUG:chardet.charsetprober:windows-1253 Greek confidence = 0.0
DEBUG:chardet.charsetprober:ISO-8859-5 Bulgarian confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Bulgarian confidence = 0.01
DEBUG:chardet.charsetprober:TIS-620 Thai confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-9 Turkish confidence = 0.5409487892112278
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Russian confidence = 0.01
DEBUG:chardet.charsetprober:KOI8-R Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-5 Russian confidence = 0.0
DEBUG:chardet.charsetprober:MacCyrillic Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM866 Russian confidence = 0.0
DEBUG:chardet.charsetprober:IBM855 Russian confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-7 Greek confidence = 0.0
DEBUG:chardet.charsetprober:windows-1253 Greek confidence = 0.0
DEBUG:chardet.charsetprober:ISO-8859-5 Bulgarian confidence = 0.0
DEBUG:chardet.charsetprober:windows-1251 Bulgarian confidence = 0.01
DEBUG:chardet.charsetprober:TIS-620 Thai confidence = 0.01
DEBUG:chardet.charsetprober:ISO-8859-9 Turkish confidence = 0.5409487892112278
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
DEBUG:chardet.charsetprober:windows-1255 Hebrew confidence = 0.0
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done

And for a reason that seems obscure to me, they don't appear on all feedstocks...

I'm currently getting these with AnacondaRecipes/transformers-feedstock#9 but I'm not seeing them on other feedstocks.

Conda Info

active environment : base
    active env location : /Users/jcmorin/miniconda
            shell level : 1
       user config file : /Users/jcmorin/.condarc
 populated config files : /Users/jcmorin/.condarc
          conda version : 23.11.0
    conda-build version : 3.28.0
         python version : 3.11.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m2
                          __conda=23.11.0=0
                          __osx=13.6=0
                          __unix=0=0
       base environment : /Users/jcmorin/miniconda  (writable)
      conda av data dir : /Users/jcmorin/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/jcmorin/miniconda/pkgs
                          /Users/jcmorin/.conda/pkgs
       envs directories : /Users/jcmorin/miniconda/envs
                          /Users/jcmorin/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Darwin/22.6.0 OSX/13.6 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/jcmorin/.condarc <==
report_errors: False
conda-build: {}

Conda list

# packages in environment at /Users/jcmorin/miniconda:
#
# Name                    Version                   Build  Channel
archspec                  0.2.1              pyhd3eb1b0_0    defaults
attrs                     23.1.0          py311hca03da5_0    defaults
backports                 1.1                pyhd3eb1b0_0    defaults
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0    defaults
backports.tempfile        1.0                pyhd3eb1b0_1    defaults
backports.weakref         1.0.post1                  py_1    defaults
beautifulsoup4            4.12.2          py311hca03da5_0    defaults
boltons                   23.0.0          py311hca03da5_0    defaults
brotli-python             1.0.9           py311h313beb8_7    defaults
brotlipy                  0.7.0           py311h80987f9_1002    defaults
bzip2                     1.0.8                h620ffc9_4    defaults
c-ares                    1.19.1               h80987f9_0    defaults
ca-certificates           2023.08.22           hca03da5_0    defaults
cctools                   949.0.1             hc179dcd_25    defaults
cctools_osx-arm64         949.0.1             h332cad3_25    defaults
certifi                   2023.11.17      py311hca03da5_0    defaults
cffi                      1.16.0          py311h80987f9_0    defaults
chardet                   4.0.0           py311hca03da5_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.1.7           py311hca03da5_0    defaults
conda                     23.11.0         py311hca03da5_0    defaults
conda-build               3.28.0          py311hca03da5_0    defaults
conda-content-trust       0.2.0           py311hca03da5_0    defaults
conda-index               0.3.0           py311hca03da5_0    defaults
conda-libmamba-solver     23.11.1         py311hca03da5_0    defaults
conda-package-handling    2.2.0           py311hca03da5_0    defaults
conda-package-streaming   0.9.0           py311hca03da5_0    defaults
conda-verify              3.4.2                      py_1    defaults
cryptography              41.0.3          py311hd4332d6_0    defaults
darts                     0.25.0                   pypi_0    pypi
distro                    1.8.0           py311hca03da5_0    defaults
filelock                  3.13.1          py311hca03da5_0    defaults
fmt                       9.1.0                h48ca7d4_0    defaults
future                    0.18.3          py311hca03da5_0    defaults
glob2                     0.7                pyhd3eb1b0_0    defaults
icu                       73.1                 h313beb8_0    defaults
idna                      3.4             py311hca03da5_0    defaults
jinja2                    3.1.2           py311hca03da5_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
jsonschema                4.19.2          py311hca03da5_0    defaults
jsonschema-specifications 2023.7.1        py311hca03da5_0    defaults
krb5                      1.20.1               hf3e1bf2_1    defaults
ld64                      530                 hb29bf3f_25    defaults
ld64_osx-arm64            530                 h001ce53_25    defaults
ldid                      2.1.5                h20b2a84_3    defaults
libarchive                3.6.2                h62fee54_2    defaults
libcurl                   8.4.0                h3e2b118_0    defaults
libcxx                    14.0.6               h848a8c0_0    defaults
libedit                   3.1.20221030         h80987f9_0    defaults
libev                     4.33                 h1a28f6b_1    defaults
libffi                    3.4.4                hca03da5_0    defaults
libiconv                  1.16                 h1a28f6b_2    defaults
liblief                   0.12.3               h313beb8_0    defaults
libllvm14                 14.0.6               h7ec7a93_3    defaults
libmamba                  1.5.3                h15e39b3_0    defaults
libmambapy                1.5.3           py311h1c5506f_0    defaults
libnghttp2                1.57.0               h62f6fdd_0    defaults
libsolv                   0.7.24               h514c7bf_0    defaults
libssh2                   1.10.0               h02f6b3c_2    defaults
libxml2                   2.10.4               h0dcf63f_1    defaults
lz4-c                     1.9.4                h313beb8_0    defaults
markupsafe                2.1.1           py311h80987f9_0    defaults
menuinst                  2.0.0           py311hca03da5_0    defaults
more-itertools            10.1.0          py311hca03da5_0    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   3.0.12               h1a28f6b_0    defaults
packaging                 23.1            py311hca03da5_0    defaults
patch                     2.7.6             h1a28f6b_1001    defaults
pcre2                     10.42                hb066dcc_0    defaults
pip                       23.3.1          py311hca03da5_0    defaults
pkginfo                   1.9.6           py311hca03da5_0    defaults
platformdirs              3.10.0          py311hca03da5_0    defaults
pluggy                    1.0.0           py311hca03da5_1    defaults
psutil                    5.9.0           py311h80987f9_0    defaults
py-lief                   0.12.3          py311h313beb8_0    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py311h80987f9_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 23.2.0          py311hca03da5_0    defaults
pysocks                   1.7.1           py311hca03da5_0    defaults
python                    3.11.5               hb885b13_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
python.app                3               py311h80987f9_0    defaults
pytz                      2023.3.post1    py311hca03da5_0    defaults
pyyaml                    6.0.1           py311h80987f9_0    defaults
readline                  8.2                  h1a28f6b_0    defaults
referencing               0.30.2          py311hca03da5_0    defaults
reproc                    14.2.4               hc377ac9_1    defaults
reproc-cpp                14.2.4               hc377ac9_1    defaults
requests                  2.31.0          py311hca03da5_0    defaults
rpds-py                   0.10.6          py311hf0e4da2_0    defaults
ruamel.yaml               0.17.21         py311h80987f9_0    defaults
ruamel.yaml.clib          0.2.7           py311h80987f9_0    defaults
setuptools                68.0.0          py311hca03da5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
soupsieve                 2.5             py311hca03da5_0    defaults
sqlite                    3.41.2               h80987f9_0    defaults
tapi                      1100.0.11            h8754e6a_1    defaults
tk                        8.6.12               hb8d0fd4_0    defaults
tomli                     2.0.1           py311hca03da5_0    defaults
toolz                     0.12.0          py311hca03da5_0    defaults
tqdm                      4.65.0          py311hb6e6a13_0    defaults
truststore                0.8.0           py311hca03da5_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.18         py311hca03da5_0    defaults
wheel                     0.41.2          py311hca03da5_0    defaults
xz                        5.4.2                h80987f9_0    defaults
yaml                      0.2.5                h1a28f6b_0    defaults
yaml-cpp                  0.8.0                h313beb8_0    defaults
zlib                      1.2.13               h5a0b063_0    defaults
zstandard                 0.19.0          py311h80987f9_0    defaults
zstd                      1.5.5                hd90d995_0    defaults

Additional Context

No response
@JeanChristopheMorinPerso JeanChristopheMorinPerso added the type::bug describes erroneous operation, use severity::* to classify the type label Dec 5, 2023
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Dec 5, 2023
@mbargull
Copy link
Member

mbargull commented Dec 5, 2023

And for a reason that seems obscure to me, they don't appear on all feedstocks...

I'm currently getting these with AnacondaRecipes/transformers-feedstock#9 but I'm not seeing them on other feedstocks.

That recipe has some emoji characters:

    - 📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.
    - 🖼️ Images, for tasks like image classification, object detection, and segmentation.
    - 🗣️ Audio, for tasks like speech recognition and audio classification.

highly likely that those triggered that.

Unfortunately, 3.28.0 introduces new debug logs.

I don't believe this is any new behavior; maybe you can try some older versions to confirm this really is new?

The origin from these log messages is probably in bs4.dammit which tries a bunch of encoding before trying UTF-8, including those from chardet's prober.
We probably want to replace all uses of UnicodeDammit with something that tries UTF-8 first (and then uses UnicodeDammit or something else if really necessary).

@travishathaway travishathaway added the source::anaconda created by members of Anaconda, Inc. label Dec 12, 2023
@JeanChristopheMorinPerso JeanChristopheMorinPerso closed this as not planned Won't fix, can't repro, duplicate, stale Jan 25, 2024
@github-project-automation github-project-automation bot moved this from 🆕 New to 🏁 Done in 🧭 Planning Jan 25, 2024
@github-actions github-actions bot added the locked [bot] locked due to inactivity label Aug 20, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
locked [bot] locked due to inactivity source::anaconda created by members of Anaconda, Inc. type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@travishathaway @mbargull @JeanChristopheMorinPerso