webui 只加载Ziya 13B，推理的时候报 RuntimeError: probability tensor contains either inf, nan or element < 0

[edisonzf2020]

错误：

Traceback (most recent call last):
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "/home/hysz/AI/LLaMA-Efficient-Tuning/src/web_demo.py", line 99, in predict
    generation_output = model.generate(input_ids=input_ids, **gen_kwargs)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/transformers/generation/utils.py", line 1568, in generate
    return self.sample(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/transformers/generation/utils.py", line 2651, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
06/03/2023 13:48:06 - INFO - httpx - HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 500 Internal Server Error"
06/03/2023 13:48:06 - INFO - httpx - HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"
06/03/2023 13:48:07 - INFO - httpx - HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 200 OK"
06/03/2023 13:48:07 - INFO - httpx - HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"

[edisonzf2020]

是int8 加载的，用Ziya官网推荐的代码推理是没有问题的

[hiyouga]

请更新仓库代码后重新尝试。

[edisonzf2020]

刚刚更新仓库代码还是这样，
启动命令如下：
python ./src/web_demo.py --model_name_or_path ../models/Ziya-LLaMA-13B --quantization_bit 8
出错信息如下：

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/transformers/generation/utils.py:1255: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
  warnings.warn(
Traceback (most recent call last):
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "/home/hysz/AI/LLaMA-Efficient-Tuning/./src/web_demo.py", line 112, in predict
    generation_output = model.generate(input_ids=input_ids, **gen_kwargs)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/transformers/generation/utils.py", line 1568, in generate
    return self.sample(
  File "/home/hysz/anaconda3/envs/qlora/lib/python3.10/site-packages/transformers/generation/utils.py", line 2651, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
06/03/2023 21:26:59 - INFO - httpx - HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 500 Internal Server Error"
06/03/2023 21:26:59 - INFO - httpx - HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"
06/03/2023 21:27:00 - INFO - httpx - HTTP Request: POST http://localhost:7860/api/predict "HTTP/1.1 200 OK"
06/03/2023 21:27:00 - INFO - httpx - HTTP Request: POST http://localhost:7860/reset "HTTP/1.1 200 OK"

[hiyouga]

运行 cli_demo 会出现相同问题吗？

[flaviadeutsch]

一样的

[hiyouga]

设置参数 num_beams 为 1 试试。

[flaviadeutsch]

更新代码可以了