Add cuQuantum-based TFQ native ops and layers.

[jaeyoo]

This has large changes and requires some deps error, rewrite documentations, and fixed internal bugs inside random ops, and so forth.

Please look at the PR descriptions in my repo to understand the procedures : https://github.com/jaeyoo/quantum/pulls?q=is%3Apr+is%3Aclosed

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[MichaelBroughton]

Would you be able to revert the files where there are little to no changes made to the actual code so that the diff view is easier to read ?

[QuantumJaeYoo]

Sure, let me update the PR soon :) thank you Michael.

[Sinestro38]

Breaking Changes

• Build, compilation, and packaging:
  • The TensorFlow dependency has been upgraded from 2.7.0 to 2.11.0:
  • TensorFlow Quantum is now compiled with _GLIBCXX_USE_CXX11_ABI=1. Downstream projects that encounter std::__cxx11 or [abi:cxx11] linker errors will need to adopt this compiler option. See the GNU C++ Library docs on Dual ABI.
  • TensorFlow Quantum is now compiled with -std=c++17, see install.md for build instructions.
• Cirq dependency has been upgraded from 0.13.1 to ~=1.0
  • cirq_google.XMON was deprecated : [cirq-google] Proposal: Remove XmonDevice, Bristlecone, Foxtail and rename optimize_for_xmon. quantumlib/Cirq#4856
  • QuantumEngineSampler was deprecated : Deprecate QuantumEngineSampler quantumlib/Cirq#5371
    • So, we need ProcessorSampler() for testing
  • cirq.CNOT interface was changed.
    • https://quantumai.google/reference/python/cirq/CNOT
    • No more control, target argument.
  • cirq.SingleQubitGate was deprecated.
    • For testing, use cirq.testing.SingleQubitGate : https://github.com/quantumlib/Cirq/pull/5272/files
    • For implementation, use cirq.Gate.

Major Features and Improvements

• Significant performance improvements by introducing cuQuantum support for circuit execution on Nvidia GPUs:
  • TensorFlow Quantum Keras layers can now be executed on GPU by setting the optional arguement use_cuquantum=True at layer instantiation. Examples:
    • tfq.layers.Expectation(use_cuquantum=True)
    • tfq.layers.SampledExpectation(use_cuquantum=True) (note that cuQuantum runtime is unsupported for any noisy circuit operations
    • tfq.layers.State(use_cuquantum=True)
    • tfq.layers.Sample(use_cuquantum=True)
    • tfq.layers.PQC(model_circuit, operators, use_cuquantum=True)
    • tfq.layers.ControlledPQC(model_circuit, operators, use_cuquantum=True)
  • Important notes:
    • CuQuantum execution is currently only supported for source distributions meaning that the user must build TensorFlow Quantum & tensorFlow-cpu from source following the instructions in install.md.
      • Ensure that the first entry is "N" in the configure.sh script at this step of building. This ensures that you build upon tensorflow-cpu as tensorflow-gpu is unnecessary for CuQuantum support in TensorFlow Quantum.
    • The cuQuantum SDK must be installed locally. See installation instructions for details. As part of the installation process, ensure that the CUQUANTUM_ROOT environment variable is set (referred to in the installation instructions). If not set, bazel will attempt to automatically locate the folder containing the cuQuantum installation upon running configure.sh at this step.
      • Tested on Titan, Ampere and Volta Nvidia GPU architectures. Note that Pascal GPU architectures are not supported, see documentation to check whether your GPU is compatible with cuQuantum
    • Quantum concurrency (global context option) should be turned off when use_cuquantum=True. This can be done by running: tfq.python.quantum_context.set_quantum_concurrent_op_mode(False)

Source: https://github.com/Sinestro38/quantum/blob/master/tensorflow_quantum/release.md
cc: @MichaelBroughton @QuantumJaeYoo

[jccalvojackson]

Hi I tried all the above and when ran the tests, these failed:

//tensorflow_quantum/core/ops:tfq_adj_grad_op_cuquantum_test             FAILED in 8.9s
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/testlogs/tensorflow_quantum/core/ops/tfq_adj_grad_op_cuquantum_test/test.log
//tensorflow_quantum/core/ops:tfq_simulate_ops_cuquantum_test            FAILED in 7.6s
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/testlogs/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test/test.log
//tensorflow_quantum/python/layers/circuit_executors:sampled_expectation_test FAILED in 59.1s
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/testlogs/tensorflow_quantum/python/layers/circuit_executors/sampled_expectation_test/test.log
//tensorflow_quantum/python/optimizers:spsa_minimizer_test               FAILED in 24.6s
  /home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/testlogs/tensorflow_quantum/python/optimizers/spsa_minimizer_test/test.log

the output of the second test log being

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //tensorflow_quantum/core/ops:tfq_simulate_ops_cuquantum_test
-----------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/load_module.py", line 42, in load_module
    return load_library.load_op_library(path)
  File "/home/ubuntu/quantum_nvidia/quantum/quantum_env/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libcublas.so.12: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.py", line 23, in <module>
    from tensorflow_quantum.core.ops import tfq_simulate_ops_cuquantum
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum.py", line 19, in <module>
    SIM_OP_MODULE = load_module("_tfq_simulate_ops_cuquantum.so")
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/load_module.py", line 46, in load_module
    return load_library.load_op_library(path)
  File "/home/ubuntu/quantum_nvidia/quantum/quantum_env/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/ubuntu/quantum_nvidia/quantum/quantum_env/lib/python3.8/site-packages/tensorflow_quantum/core/ops/_tfq_simulate_ops_cuquantum.so: cannot open shared object file: No such file or directory

[Sinestro38]

@jccalvojackson

• Have you added the lib directory of your cuQuantum AND cuBLAS installations to your LD_LIBRARY_PATH environment variable? For example, after setting CUQUANTUM_ROOT env var, I would run:

export LD_LIBRARY_PATH=${CUQUANTUM_ROOT}/lib:${LD_LIBRARY_PATH}

• Your last error appears to be due to a build issue in compiling the cuda code or building the package wheel. I would recommend you follow the instructions to build from source as stated exactly in the docs/install.md doc of this fork. Specifically this section to enable GPU support: https://github.com/jaeyoo/quantum/blob/master/docs/install.md#7-build-the-tensorflow-quantum-pip-package-for-gpu

[jccalvojackson]

iI followed those, I didn't get any error during the build instructions.

when running configure I do get the message

cuQuantum library is detected here: CUQUANTUM_ROOT=<my path>

I tried again and see this error in one of the tests

Traceback (most recent call last):
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/load_module.py", line 42, in load_module
    return load_library.load_op_library(path)
  File "/home/ubuntu/quantum_nvidia/quantum/quantum_env/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libcublas.so.12: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.py", line 23, in <module>
    from tensorflow_quantum.core.ops import tfq_simulate_ops_cuquantum
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum.py", line 19, in <module>
    SIM_OP_MODULE = load_module("_tfq_simulate_ops_cuquantum.so")
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/3a577e6722a9311ddc77692e6a730328/execroot/__main__/bazel-out/k8-opt/bin/tensorflow_quantum/core/ops/tfq_simulate_ops_cuquantum_test.runfiles/__main__/tensorflow_quantum/core/ops/load_module.py", line 46, in load_module
    return load_library.load_op_library(path)
  File "/home/ubuntu/quantum_nvidia/quantum/quantum_env/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libcublas.so.12: cannot open shared object file: No such file or directory

I see that I have /usr/local/cuda/lib64/libcublas.so.11 but not libcublas.so.12 how do I make it use 11 instead of 12?

when running configure it says

Please specify the CUDA SDK major version you want to use. [Leave empty to default to CUDA 11]:

which I do

[jccalvojackson]

managed to make it work (though is not using gpus significantly) but that's for later. Thank you!