参数初始化的问题？

[junranhe]

我用了以前在cxxnet上跑的数据测试mxnet（网络结构用了alexnet和inception），发现一个奇怪的现象，只要不用batchnorm，网络都不收敛，而且运行的结果看起来学不到任何东西，简单地在卷积层后加上bn，网络就收敛正常了，一开始怀疑是inplace优化的问题，后来禁止了inplace还是一样，之前我在cxxnet上做过类似的实验，这个数据集都是能很好收敛的，是不是初始化参数权重的地方要特别注意的地方？能不能提供像cxxnet那么精确参数设置？
例如：
momentum = 0.9
wmat:lr = 0.05
wmat:wd = 0.0001
bias:wd = 0.000
bias:lr = 0.1

[tqchen]

You should be able to change the optimizer behavior easily by https://github.com/dmlc/mxnet/blob/master/python/mxnet/optimizer.py#L85

Consider to submit a PR on this :)

[antinucleon]

Thanks for letting us know We figure out a bug in initialization and we are fixing it. To set params for model, you may find this doc to be useful. https://mxnet.readthedocs.org/en/latest/python/model.html

[tqchen]

Was caused by inconsistent scale xavier initialization. Batchnorm makes things scale invariant so it poses a less problem

[antinucleon]

I changed default Initializer, Now it works on many nets without BN. Could you try again?

[junranhe]

更新代码不收敛的现象没了，btw，model.py 里,eval_metric.update() 可不可以挪到 调用epoch_end_callback 前， 这样比较方便我测试的时候传入自己的eval_metric和epochcallback来检查迭代情况，否则容易第一个batch抛除零异常。
还有一个发现一个奇怪的现象，可能跟机器相关，用的都是cudnn-v3，alexnet的计算时间是比cxxnet上快了很多，之前我的780是每秒180张，现在时260张，但inception-bn的速度却跟之前cxxnet的一样，都是34张每秒，没发现明显提高

[antinucleon]

What is your batch size of Inception BN?

On Tue, Sep 29, 2015 at 21:47 junranhe [email protected] wrote:

更新代码不收敛的现象没了，btw，model.py 里,eval_metric.update() 可不可以挪到
调用epoch_end_callback 前，
这样比较方便我测试的时候传入自己的eval_metric和epochcallback来检查迭代情况，否则容易第一个batch抛除零异常。

还有一个发现一个奇怪的现象，可能跟机器相关，用的都是cudnn-v3，alexnet的计算时间是比cxxnet上快了很多，之前我的780是每秒180张，现在时260张，但inception-bn的速度却跟之前cxxnet的一样，都是34张每秒，没发现明显提高

—
Reply to this email directly or view it on GitHub
#182 (comment).

[junranhe]

two 780，mxnet is 64batch，so each gpu 32batch， cxxnet is 20 batch each（I modify the cxxnet so it can run 20batch on 3GB memory）

[tqchen]

Normally you cannot expect mxnet to be faster than cxxnet in all cases as cxxnet is already heavily optimized in c++. But you can expect they run at same performance, with mxnet more memory efficient and more flexible.

You proposal seems to be valid, please open a PR on this

Tianqi

[junranhe]

well, when I run the same inception-bn on purine，and it run 50 sample per sec

[tqchen]

hmm, maybe you can try set update_on_kvstore=True http://mxnet.readthedocs.org/en/latest/python/model.html#mxnet.model.FeedForward.fit which will change default behavior. But I do not have tried 780 before so don't know if that would really help :)

[tqchen]

Sorry update_on_kvstore=False

[junranhe]

Oh, I try update_on_kvstore both True and False, it has no effect, and I try to run mxnet on single gpu, and inception-bn is 65 sample per sec... compare with two gpu 68 sample per sec(so 34 sample/sec), It seem just like used one gpu, Is there some lock make it slowdown? (I use dmesg can not find any error message, and I test alexnet is no any problem like inception-bn),

[tqchen]

I see what you mean, there is no such lock that I am aware of. What you observed was interesting.
Could because the synchronization was caped somehow on your machine, mxnet indeed uses cpu heavily because we need to catch up with dynamic dependencies issued in python, maybe you can set the env variables in https://github.com/dmlc/mxnet/blob/master/doc/env_var.md related to NTHREADS to 1. and see if that helps.

[junranhe]

I try it , nthreads seems no effect, the main effect on speed are batch_size and mxnet_engine_type, and NaiveEngine is 30 per sec, EngineThread is 50 per sec, and Default is 68 per sec(all on two 780 gpu)

[tqchen]

Thanks. I have no great idea so far. Seems it was due to synchronization cost(maybe PCI bandwith or memory speed). You can try use the most recent version and set

kvstore_type = 'device'

This will do GPU to GPU copy instead of copy to mainmemory
You will need to assign more to GPU worker threads in this setting.

[junranhe]

well, kvstore_type = 'device' is work, and My mechine run 120 sample per sec now, thanks for you help:)

[antinucleon]

Guess it is caused by your PCI-E bandwith . Good to see it works.

[tqchen]

You may also want to set
MXNET_GPU_WORKER_NTHREADS to be two, which I observed would help in this setting. Your problem was caused by the PCI-E bandwith, so the device side kvstore works better on your setting.

Glad that things works out for you. Thanks for using MXNet in the very early stage and help us improve it

[junranhe]

Thanks to let me know, My GPU'memory is so poor, and out of memory tor more worker :(

[tqchen]

I see, I have created #204 so memory allocation algorithm is not strictly tied with GPU threads. You can try

MXNET_EXEC_NUM_TEMP=1
MXNET_GPU_WORKER_NTHREADS=2

after the PR is merged in

[junranhe]

Thanks, I have one more question, When I build the VGGNet 13layer, on 2 batch_size on 1 gpu used 1500M memory, but I set 3 batch_size, It raise out of memory(780 has 3000M), How the memory allocation algorighm work?

[tqchen]

The memory allocator will try to re-use the memory as much as possible. I do not know in particular what happens in your case though. This could due to the memory demand of workspace of CUDNN is nonlinear to batch size(for example, rounded to size of 4)

[junranhe]

Year, I try
MXNET_EXEC_NUM_TEMP=1
MXNET_GPU_WORKER_NTHREADS=2

and It is 5% faster on my machine

[tqchen]

Yeah, that is what it can get you