Enabling 1bit SGD

Enabling 1-bit Stochastic Gradient Descent for the Computational Network Toolkit (1bit-SGD)

1-bit Stochastic Gradient Descent (1bit-SGD) is an optional component of the Computational Network Toolkit (CNTK) which significantly improves the performance during deep neural network (DNN) training using a single server with multiple GPUs and/or multiple servers with a single or multiple GPUs. This is achieved by the implementing the Gradient Quantization approach that significantly reduces the communication costs during parallel computations involved in the DNN training process. You may read more about 1-bit Stochastic Gradient Descent in this article.

License difference between CNTK and 1bit-SGD

1bit-SDK is distributed under a different license than the rest of CNTK. Before considering usage of 1bit-SGD with CNTK please get yourself familiar with both licenses (see links below). Note, that you may find 1bit-SGD license more restrictive than CNTK license:

• CNTK License
• CNTK 1bit-SGD License

Setting up 1-bit Stochastic Gradient Descent for CNTK on your system

By using default setup procedures and parameters described in the main installation section your system will contain neither 1bit-SGD code nor the corresponding functionality in compiled binaries.

Please follow the procedure below to enable 1bit-SGD on your system:

Making aware of Licensing Differences

Please ensure you understand that 1bit-SGD is licensed under a different license than the rest of CNTK (see License difference between CNTK and 1bit-SGD)