Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"relu_layer.cu:29] Cuda kernel failed. Error: invalid configuration argument" on ilsvrc12 #282

Closed
caffecuda opened this issue Apr 1, 2014 · 3 comments
Closed

"relu_layer.cu:29] Cuda kernel failed. Error: invalid configuration argument" on ilsvrc12 #282

caffecuda opened this issue Apr 1, 2014 · 3 comments
Labels
downstream problem?

Comments

@caffecuda
Copy link

Hi,

We got the master branch on March 28, while the mnist and cifar10 demos worked fine on Tesla M2090, the ilsvrc12 training script gave the following error:

...
I0401 15:13:18.902433 3143 net.cpp:173] Collecting Learning Rate and Weight Decay.
I0401 15:13:18.902487 3143 net.cpp:166] Network initialization done.
I0401 15:13:18.902521 3143 net.cpp:167] Memory required for Data 210114408
I0401 15:13:18.902660 3143 solver.cpp:36] Solver scaffolding done.
I0401 15:13:18.902760 3143 solver.cpp:47] Solving CaffeNet
F0401 15:13:19.040458 3143 relu_layer.cu:29] Cuda kernel failed. Error: invalid configuration argument
*** Check failure stack trace: ***
@ 0x2b06c5dcbb7d google::LogMessage::Fail()
@ 0x2b06c5dcdc7f google::LogMessage::SendToLog()
@ 0x2b06c5dcb76c google::LogMessage::Flush()
@ 0x2b06c5dce51d google::LogMessageFatal::~LogMessageFatal()
@ 0x48188c caffe::ReLULayer<>::Forward_gpu()
@ 0x431ada caffe::Net<>::ForwardPrefilled()
@ 0x423cb8 caffe::Solver<>::Solve()
@ 0x40e645 main
@ 0x2b06c806176d (unknown)
@ 0x40fced (unknown)
Aborted (core dumped)
Done.

Is this related to the issue described in #12?
@caffecuda
Copy link
Author

@blackball @jeffdonahue
Hi,
Do you think this is related to #214? Many thanks.

@kloudkl
Copy link
Contributor

kloudkl commented Apr 3, 2014

https://github.com/BVLC/caffe/search?q=invalid+configuration+argument&ref=cmdform&type=Issues

@teknosains
Copy link

no answer yet...but its closed

its : . Error: invalid configuration argument

not invalid device function

@OpenHero OpenHero mentioned this issue Jul 16, 2014
Closed
OpenHero added a commit to OpenHero/caffe that referenced this issue Jul 16, 2014
@OpenHero
When the Compute Capability of CUDA is < 3.0, the Maximum x-dimension…
0469839 
… of a grid of thread blocks is 65535. So the Kernel will crash when the CAFFE_GET_BLOCKS is bigger than 65535. Like Fermi architecture GPUs.

The crash will happen in src/caffe/layers/relu_layer.cu line 29. as BVLC#282 BVLC#12
![image](https://cloud.githubusercontent.com/assets/5321224/3595637/4c27157c-0cb8-11e4-8009-c40c88ac1500.png)

Fixed it with
// CUDA: number of blocks for threads.
inline int CAFFE_GET_BLOCKS(const int N) {
	//return (N + CAFFE_CUDA_NUM_THREADS - 1) / CAFFE_CUDA_NUM_THREADS;
	int num_blocks = (N + CAFFE_CUDA_NUM_THREADS - 1) / CAFFE_CUDA_NUM_THREADS;
	return num_blocks > Caffe::cuProp().maxGridSize[0]? num_blocks : Caffe::cuProp().maxGridSize[0];
}
This was referenced Jul 16, 2014
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
downstream problem?
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@shelhamer @teknosains @kloudkl @caffecuda