Skip to content

Enabling 1bit SGD

Jump to bottom
Alexey Orlov edited this page Jan 22, 2016 · 23 revisions

Enabling 1-bit Stochastic Gradient Descent for the Computational Network Toolkit (1bit-SGD)

1-bit Stochastic Gradient Descent (1bit-SGD) is an optional component of the Computational Network Toolkit (CNTK) which significantly improves the performance during deep neural network (DNN) training using a single server with multiple GPUs and/or multiple servers with a single or multiple GPUs. This is achieved by the implementing the Gradient Quantization approach that significantly reduces the communication costs during parallel computations involved in the DNN training process. You may read more about 1-bit Stochastic Gradient Descent in this article.

License difference between CNTK and 1bit-SGD

1bit-SDK is distributed under a different license than the rest of CNTK. Before considering usage of 1bit-SGD with CNTK please get yourself familiar with both licenses (see links below). Note, that you may find 1bit-SGD license more restrictive than CNTK license:

Setting up 1-bit Stochastic Gradient Descent for CNTK on your system

By using default setup procedures and parameters described in the main installation section your system will contain neither 1bit-SGD code nor the corresponding functionality in compiled binaries.

Please follow the procedure below to enable 1bit-SGD on your system:

Making aware of Licensing Differences

Please ensure you understand that 1bit-SGD is licensed under a different license than the rest of CNTK (see License difference between CNTK and 1bit-SGD).

Set up prerequisites

Ensure you set up all required software prerequisites as described in the main installation section. You may skip the part of getting the CNTK code - this will be covered in the next section.

Getting 1bit-SGD source code (together with or in addition to the main CNTK code)

1bit-SGD source code is stored in a separate GitHub.com repository which is set as a submodule to the main CNTK repository. There are two methods of getting the code depending on your situation.

Making new CNTK repository clone with 1bit-SGD code

This method works if you want to make a new clone of CNTK repository.

IMPORTANT! Do not apply the command below to an existing repository clone - you will get an error message. If you already have a clone of CNTK repository you are working with and would like to extend it with 1bit-SGD code, proceed to the next section.

To get a new CNTK repository clone with 1bit-SGD source code on your machine you need to use git recursive clone:

git clone --recursive https://github.com/Microsoft/CNTK/

You will find 1bit-SGD code in Source/1BitSGD directory of the CNTK repository.

Adding 1bit-SGD code to an existing CNTK repository clone

If you already have a working clone of CNTK repository and would like to expand it with 1bit-SGD perform the following set of commands:

  • CD to to the root of CNTK repository on your machine, e.g. cd /usr/cntk-user/git/repos/CNTK or cd c:\code\git\repos\CNTK
  • Execute git submodule update --init --recursive

Building CNTK binaries with 1bit-SGD functionality

Having 1bit-SGD code on your machine does not automatically mean you have 1bit-SGD functionality in your binaries. To invoke the build with 1bit-SGD perform the following:

Windows

Add the Environment Variable CNTK_ENABLE_1BitSGD and set its value to true.

Linux

Pass the option --1bitsgd=yes to the configure command

Then perform the build as usual.

Next steps

New Documentation Site

Iteration Plans

Clone this wiki locally