Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BatchNorm1D eval模式报错CUDNN_STATUS_NOT_SUPPORTED #49718

Closed
justld opened this issue Jan 11, 2023 · 5 comments
Closed

BatchNorm1D eval模式报错CUDNN_STATUS_NOT_SUPPORTED #49718

justld opened this issue Jan 11, 2023 · 5 comments
Assignees

Comments

@justld
Copy link

justld commented Jan 11, 2023

bug描述 Describe the Bug

BatchNorm1D在eval模式,报错CUDNN_STATUS_NOT_SUPPORTED.

环境:
cuda 10.2
Tesla v100 32gb
paddlepaddle-gpu 2.4.1

复现代码:

import paddle
x = paddle.randn((69798, 32, 64))     # 怀疑是输入太大了,当输入维度是[49931, 32, 64]时没问题
norm = paddle.nn.BatchNorm1D(64, epsilon=1e-3, momentum=0.99)   
norm.eval()    
out = norm(x.transpose([0, 2, 1]))       

报错内容:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 948, in call
return self.forward(*inputs, **kwargs)
File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/nn/layer/norm.py", line 725, in forward
use_global_stats=self._use_global_stats,
File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/nn/functional/norm.py", line 233, in batch_norm
False,
OSError: (External) CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED.
[Hint: Please search for the error code(9) on website (https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnStatus_t) to get Nvidia's official solution and advice about CUDNN Error.] (at /paddle/paddle/phi/kernels/gpu/batch_norm_kernel.cu:849)

其他补充信息 Additional Supplementary Information

No response

@paddle-bot
Copy link

paddle-bot bot commented Jan 11, 2023

您好,我们已经收到了您的问题,会安排技术人员尽快解答您的问题,请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时,您也可以通过查看官网API文档常见问题历史IssueAI社区来寻求解答。祝您生活愉快~

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the APIFAQGithub Issue and AI community to get the answer.Have a nice day!

@zhangbo9674
Copy link
Contributor

你好,从报错栈上看,并未发现CUDNN_STATUS_NOT_SUPPORTED,报错显示:the shape of scale must equal to [32]But received: the shape of scale is [64],可以查看一下输入数据shape

@justld
Copy link
Author

justld commented Jan 11, 2023

不好意思,刚刚的代码少了个transpose,现在的代码可以复现了

@ceci3
Copy link
Contributor

ceci3 commented Jan 11, 2023

你好,应该是实现bug,我这边修复一下哈

Copy link

paddle-bot bot commented Jan 16, 2024

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复,我们将关闭这个issue/pr。
若问题未解决或有后续问题,请随时重新打开,我们会继续跟进。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants