Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ernie] failed to predict. (data_id=3 log_id=3) [ernie|0] Failed to process(batch: [3]): In user code: #2008

Open
lisdoo opened this issue Oct 25, 2024 · 1 comment
Assignees

Comments

@lisdoo
Copy link

lisdoo commented Oct 25, 2024

请问以下报错是什么原因?
[ernie] failed to predict. (data_id=3 log_id=3) [ernie|0] Failed to process(batch: [3]): In user code:

File "/home/aistudio/work/PaddleNLP/legacy/applications/neural_search/recall/in_batch_negative/export_model.py", line 56, in <module>
  paddle.jit.save(model, save_path)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/api.py", line 752, in wrapper
  func(layer, path, input_spec, **configs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/dygraph/base.py", line 75, in __impl__
  return func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/api.py", line 1043, in save
  static_func.concrete_program_specify_input_spec(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 709, in concrete_program_specify_input_spec
  concrete_program, _ = self.get_concrete_program(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 589, in get_concrete_program
  concrete_program, partial_program_layer = self._program_cache[
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1249, in __getitem__
  self._caches[item_id] = self._build_once(item)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1193, in _build_once
  concrete_program = ConcreteProgram.from_func_spec(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/dygraph/base.py", line 75, in __impl__
  return func(*args, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/program_translator.py", line 1063, in from_func_spec
  outputs = static_func(*inputs)
File "/home/aistudio/work/PaddleNLP/legacy/applications/neural_search/recall/in_batch_negative/base_model.py", line 111, in get_pooled_embedding
  _, cls_embedding = self.ptm(input_ids, token_type_ids, position_ids, attention_mask)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1256, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1235, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddlenlp/transformers/ernie/modeling.py", line 331, in forward
  if attention_mask is None:
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/convert_operators.py", line 352, in convert_ifelse
  out = _run_py_ifelse(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/jit/dy2static/convert_operators.py", line 429, in _run_py_ifelse
  py_outs = true_fn() if pred else false_fn()
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddlenlp/transformers/ernie/modeling.py", line 332, in forward
  attention_mask = paddle.unsqueeze(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/layers/math_op_patch.py", line 445, in __impl__
  current_block(self).append_op(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/framework.py", line 4013, in append_op
  op = Operator(
File "/opt/conda/envs/python35-paddle120-env/lib/python3.10/site-packages/paddle/fluid/framework.py", line 2781, in __init__
  for frame in traceback.extract_stack():

InvalidArgumentError: Axis should be less than 2, but received axis is 2.
  [Hint: Expected axis < max_dim, but received axis:2 >= max_dim:2.] (at /paddle/paddle/phi/kernels/funcs/common_shape.h:53)
  [operator < equal > error]. Please check the input dict and checkout PipelineServingLogs/pipeline.log for more details.

运行的环境是:

python -V

Python 3.8.10

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

pip list | grep paddle

paddle-bfloat 0.1.7
paddle-serving-app 0.9.0
paddle-serving-client 0.9.0
paddle-serving-server-gpu 0.9.0.post1028
paddle2onnx 1.2.11
paddlefsl 1.1.0
paddlenlp 2.4.2
paddlepaddle-gpu 2.4.2

按照这个例子训练了一个召回模型
https://github.com/PaddlePaddle/PaddleNLP/tree/develop/slm/applications/neural_search/recall/in_batch_negative

现在是部署推理阶段报错了

Copy link

Message that will be displayed on users' first issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants