Kokkos: disable failing CUDA+DEBUG test #2494

ibaned · 2018-04-02T16:43:35Z

@trilinos/kokkos
@trilinos/framework

Description

Disable instances of a test which request a hardcoded number of
32 CUDA threads per warp, because with debugging
enabled the CUDA kernel uses too many registers
and can only run on 16 threads per warp max.

Motivation and Context

This test is failing in key ATDM build configurations

Related Issues

kokkos/kokkos#1514, kokkos/kokkos#1513, #2471

This test requests a hardcoded number of 32 CUDA threads per warp, but with debugging enabled the CUDA kernel uses too many registers and can only run on 16 threads per warp max. [kokkos/kokkos#1514, kokkos/kokkos#1513, #2471]

trilinos-autotester · 2018-04-02T16:58:05Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2018-04-02T17:00:11Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 418
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`ca2465c`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 135
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`ca2465c`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ibaned

bartlettroscoe

Not sure how that if() test works but if it disables this on unit test then the atdm platforms then that is good.

Does this disable the test for all CUDA builds? If so, i think we can disable this one unit test for just the atdm platforms. I just need a var to set that will trigger the disable.

trilinos-autotester · 2018-04-02T17:10:12Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 4:00:00. If a change to the Pull Request source branch occurs, the testing will be attempted again.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 418
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`ca2465c`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 135
Status: FAILED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`ca2465c`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

ibaned · 2018-04-02T17:10:25Z

@bartlettroscoe this will disable those unit tests only if KOKKOS_ENABLE_DEBUG=ON and it is running on CUDA. I think its right to disable this for all GPU platforms, because so far all the ones we've looked at cannot run this configuration due to running out of registers.

ibaned · 2018-04-02T17:10:46Z

@allevin can we get details on why testing failed for this pull request?

bartlettroscoe · 2018-04-02T17:15:21Z

Ok. Sounds like this is the right thing to do then.

allevin · 2018-04-02T17:28:07Z

@ibaned

I believe the data is on CDash at https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos#!#Pull_Request

From the build listing 418 on gcc 4.9.3:

In file included from /scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/packages/kokkos/core/unit_test/serial/TestSerial_Team.cpp:73:0:
/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/packages/kokkos/core/unit_test/TestTeamVector.hpp: In member function ‘virtual void Test::serial_triple_nested_parallelism_Test::TestBody()’:
/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/packages/kokkos/core/unit_test/TestTeamVector.hpp:907:37: error: ‘Cuda’ is not a member of ‘Kokkos’
if (!std::is_same<TEST_EXECSPACE, Kokkos::Cuda>::value) {
^
/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/packages/kokkos/core/unit_test/TestTeamVector.hpp:907:37: error: ‘Cuda’ is not a member of ‘Kokkos’
/scratch/trilinos/workspace/trilinos-folder/Trilinos_pullrequest_gcc_4.9.3/Trilinos/packages/kokkos/core/unit_test/TestTeamVector.hpp:907:49: error: template argument 2 is invalid
if (!std::is_same<TEST_EXECSPACE, Kokkos::Cuda>::value) {
^
[ 32%] Built target teuchos_xml_pl_test_helpers

ibaned · 2018-04-02T17:29:22Z

Thanks @allevin !

jwillenbring · 2018-04-02T17:29:55Z

@ibaned You can find the 4.8.4 results here:

https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3409296

You can locate this page by going to the new CDash dashboard, going to the correct day, then scrolling down to the "Pull Request section" and matching your PR number with PR-XXXX at the beginning of the job name.

The 4.9.3 build info isn't available yet via CDash.

ibaned · 2018-04-02T17:31:37Z

@jwillenbring thank you! I'll try using that directly next time I run into a PR failure.

trilinos-autotester · 2018-04-02T17:38:42Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2018-04-02T17:40:49Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 419
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`3c09c73`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 136
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`3c09c73`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Using Repos:

Repo: TRILINOS (trilinos/Trilinos)

Pull Request Author: ibaned

trilinos-autotester · 2018-04-02T20:29:16Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 419
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`3c09c73`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 136
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	2494
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	kokkos-debug-cuda-fix
TRILINOS_SOURCE_REPO	https://github.com/trilinos/Trilinos
TRILINOS_SOURCE_SHA	`3c09c73`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`d191960`

trilinos-autotester · 2018-04-02T20:29:35Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

trilinos-autotester · 2018-04-02T20:29:42Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

ibaned · 2018-04-02T20:40:40Z

@bartlettroscoe approved the main changes, and the only difference since then was introduced to workaround GCC 4.8.4 not being C++11 compliant, should not affect visible behavior.

Kokkos: disable failing CUDA+DEBUG test

ca2465c

This test requests a hardcoded number of 32 CUDA threads per warp, but with debugging enabled the CUDA kernel uses too many registers and can only run on 16 threads per warp max. [kokkos/kokkos#1514, kokkos/kokkos#1513, #2471]

ibaned added type: bug The primary issue is a bug in Trilinos code or tests pkg: Kokkos labels Apr 2, 2018

ibaned self-assigned this Apr 2, 2018

ibaned requested review from bartlettroscoe and crtrott April 2, 2018 16:43

ibaned mentioned this pull request Apr 2, 2018

New failing tests in ATDM debug builds of Trilinos due to KOKKOS_ENABLE_DEBUG=ON being set #2471

Closed

bartlettroscoe approved these changes Apr 2, 2018

View reviewed changes

Kokkos: fix compilation for GCC 4.8.4

3c09c73

ibaned merged commit 100aaf3 into develop Apr 2, 2018

ibaned deleted the kokkos-debug-cuda-fix branch April 2, 2018 20:40

trilinos-autotester mentioned this pull request Dec 18, 2018

Tpetra: Import/Export operations for StaticProfile CrsGraph #4070

Merged

Kokkos: disable failing CUDA+DEBUG test #2494

Kokkos: disable failing CUDA+DEBUG test #2494

Conversation

ibaned commented Apr 2, 2018

Description

Motivation and Context

Related Issues

trilinos-autotester commented Apr 2, 2018

trilinos-autotester commented Apr 2, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Using Repos:

bartlettroscoe left a comment

Choose a reason for hiding this comment

trilinos-autotester commented Apr 2, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

ibaned commented Apr 2, 2018

ibaned commented Apr 2, 2018

bartlettroscoe commented Apr 2, 2018

allevin commented Apr 2, 2018

ibaned commented Apr 2, 2018

jwillenbring commented Apr 2, 2018

ibaned commented Apr 2, 2018

trilinos-autotester commented Apr 2, 2018

trilinos-autotester commented Apr 2, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Using Repos:

trilinos-autotester commented Apr 2, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

trilinos-autotester commented Apr 2, 2018

trilinos-autotester commented Apr 2, 2018

ibaned commented Apr 2, 2018