pip install hanlp failed

[neoedmund]

Describe the bug
pip install hanlp failed

Code to reproduce the issue

pipx install hanlp  

Describe the current behavior

Fatal error from pip prevented installation. Full pip output in file:
    /home/neoe/.local/pipx/logs/cmd_2023-10-12_20.44.33_pip_errors.log

pip failed to build package:
    tokenizers

Some possibly relevant errors from pip install:
    error: subprocess-exited-with-error
    error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
    error: could not compile `tokenizers` (lib) due to previous error; 3 warnings emitted
    error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101

Error installing hanlp.

Expected behavior
install ok

System information

• debian 12
• Python version: Python 3.11.2
• HanLP version: newest

Other info / logs

• I've completed this form and searched the web for solutions.

[neoedmund]

the rustc is current, so I guess the tokenizers is too old to pass the new rustc

[hankcs]

Hi, you could be right, the tokenizers version is pinned here: https://github.com/hankcs/HanLP/blob/doc-zh/setup.py#L64

However, even the lastest tokenizers doesn't release a Python3.11 wheel, indicating that there could be some issues with py3.11 compilation.

Also, HanLP doesn't claim support for Python3.11 so it's not a bug: https://github.com/hankcs/HanLP/blob/doc-zh/setup.py#L52 You can clone a copy and unpin the version by yourself.

To support 3.11, the latest tokenizers would force us to drop support for py3.6. Considering the popularity of py3.6, this is not a wise choice.

[neoedmund]

Thank you. I will try new version of tokenizers later. maybe rustc version related than python version.

[neoedmund]

I realized I install from pip not source, so maybe I cannot unpin anything. what the economics of rust is like, I just have a taste of it.