Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] ASAN issue: ODR violation with Clang 15, mentioning global variable alignment #14672

Closed
mbautin opened this issue Oct 26, 2022 · 2 comments · Fixed by yugabyte/yugabyte-db-thirdparty#150
Closed

[DocDB] ASAN issue: ODR violation with Clang 15, mentioning global variable alignment #14672

mbautin opened this issue Oct 26, 2022 · 2 comments · Fixed by yugabyte/yugabyte-db-thirdparty#150
Assignees
@mbautin @NatashaSerebryanaya
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue

Comments

@mbautin
Copy link
Collaborator

mbautin commented Oct 26, 2022 •
edited by yugabyte-ci
Loading

Jira Link: DB-4034

Description

This was in fact observed with Clang 14 too.

==7617==The following global variable is not properly aligned.
==7617==This may happen if another global with the same name
==7617==resides in another non-instrumented module.
==7617==Or the global comes from a C file built w/o -fno-common.
==7617==In either case this is likely an ODR violation bug,
==7617==but AddressSanitizer can not provide more details.
=================================================================
==7617==ERROR: AddressSanitizer: odr-violation (0x5606e1b7ac1a):
  [1] size=28 'typeinfo name for std::__1::bad_function_call' /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013127-1aec6dda08-almalinux8-x86_64-clang15/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
  [2] size=28 'typeinfo name for std::__1::bad_function_call' /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013127-1aec6dda08-almalinux8-x86_64-clang15/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
These globals were registered at these points:
  [1]:
    #0 0x7f2d436981fa  (/opt/yb-build/llvm/yb-llvm-v15.0.2-yb-2-1665242997-b19371b8-almalinux8-x86_64/lib/clang/15.0.2/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x9c1fa)
    #1 0x7f2d43472a8e  (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013127-1aec6dda08-almalinux8-x86_64-clang15/installed/asan/libcxx/lib/libc++.so.1+0x1b3a8e)
    #2 0x7f2d44107e09  (/lib64/ld-linux-x86-64.so.2+0x14e09) (BuildId: 67aa0f1504abcfc4befb2b61a329a30a9984e0db)

  [2]:
    #0 0x7f2d436981fa  (/opt/yb-build/llvm/yb-llvm-v15.0.2-yb-2-1665242997-b19371b8-almalinux8-x86_64/lib/clang/15.0.2/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so+0x9c1fa)
    #1 0x7f2d43472a8e  (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013127-1aec6dda08-almalinux8-x86_64-clang15/installed/asan/libcxx/lib/libc++.so.1+0x1b3a8e)
    #2 0x7f2d44107e09  (/lib64/ld-linux-x86-64.so.2+0x14e09) (BuildId: 67aa0f1504abcfc4befb2b61a329a30a9984e0db)

==7617==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'typeinfo name for std::__1::bad_function_call' at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013127-1aec6dda08-almalinux8-x86_64-clang15/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
==7617==ABORTING

https://gist.githubusercontent.com/mbautin/a42e30005058fd905fb1baf41e082de4/raw
@mbautin mbautin added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Oct 26, 2022
@mbautin mbautin self-assigned this Oct 26, 2022
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Oct 26, 2022
@mbautin
Copy link
Collaborator Author

mbautin commented Oct 26, 2022

When rebuilding without -shared-libasan (both third-party dependencies and YugabyteDB), the error is slightly different:

$  ./tests-integration-tests/linked_list-test  --gtest_list_tests
==1065547==The following global variable is not properly aligned.
==1065547==This may happen if another global with the same name
==1065547==resides in another non-instrumented module.
==1065547==Or the global comes from a C file built w/o -fno-common.
==1065547==In either case this is likely an ODR violation bug,
==1065547==but AddressSanitizer can not provide more details.
=================================================================
==1065547==ERROR: AddressSanitizer: odr-violation (0x55c52222dc1a):
  [1] size=28 'typeinfo name for std::__1::bad_function_call' /home/mbautin/code/yugabyte-db-thirdparty2/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
  [2] size=28 'typeinfo name for std::__1::bad_function_call' /home/mbautin/code/yugabyte-db-thirdparty2/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
These globals were registered at these points:
  [1]:
    #0 0x55c52226ecea  (/home/mbautin/code/yugabyte-db/build/asan-clang15-dynamic-ninja/tests-integration-tests/linked_list-test+0xb8cea)
    #1 0x7f6d4f327a0e  (/home/mbautin/code/yugabyte-db/build/asan-clang15-dynamic-ninja/tests-integration-tests/../../../../yugabyte-db-thirdparty2/installed/asan/libcxx/lib/libc++.so.1+0x1b4a0e)
    #2 0x7f6d4f4c5e09  (/lib64/ld-linux-x86-64.so.2+0x14e09) (BuildId: 67aa0f1504abcfc4befb2b61a329a30a9984e0db)

  [2]:
    #0 0x55c52226ecea  (/home/mbautin/code/yugabyte-db/build/asan-clang15-dynamic-ninja/tests-integration-tests/linked_list-test+0xb8cea)
    #1 0x7f6d4f327a0e  (/home/mbautin/code/yugabyte-db/build/asan-clang15-dynamic-ninja/tests-integration-tests/../../../../yugabyte-db-thirdparty2/installed/asan/libcxx/lib/libc++.so.1+0x1b4a0e)
    #2 0x7f6d4f4c5e09  (/lib64/ld-linux-x86-64.so.2+0x14e09) (BuildId: 67aa0f1504abcfc4befb2b61a329a30a9984e0db)

==1065547==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'typeinfo name for std::__1::bad_function_call' at /home/mbautin/code/yugabyte-db-thirdparty2/src/llvm-15.0.2-yb-2/libcxx/src/functional.cpp
==1065547==ABORTING

mbautin added a commit to yugabyte/llvm-project that referenced this issue Oct 27, 2022
@mbautin
Temporarily remove the check for global variables being properly alig…
b99cfaa 
…ned.

See yugabyte/yugabyte-db#14672 for details.
@NatashaSerebryanaya
Copy link
Contributor

The main open issue is google/sanitizers#1017
Currently available workarounds -
add -mllvm -asan-use-private-alias=1 in compile time (as in llvm/llvm-project@dfc24b8)
or runtime ASAN_OPTIONS=report_globals=0

mbautin added a commit to yugabyte/llvm-project that referenced this issue Oct 29, 2022
@mbautin
Temporarily remove the check for global variables being properly alig…
0b8d118 
…ned.

See yugabyte/yugabyte-db#14672 for details.
@mbautin mbautin mentioned this issue Oct 29, 2022
Merged
@mbautin mbautin reopened this Oct 29, 2022
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Nov 1, 2022
mbautin added a commit that referenced this issue Nov 4, 2022
@mbautin
[#14672] Use Clang 15 for ASAN
c4e45a1 
Summary:
Use Clang 15 for ASAN (instead of Clang 13 used prior to this diff). This requires using an updated version of LLVM which suppresses an ODR violation error (#14672) due to issues with symbol alignment reported by ASAN in Clang 14 or later ( our custom patch is yugabyte/llvm-project@0b8d118 ), as well as specifying special flags (`-mllvm -asan-use-private-alias=1`). Thanks to @NatashaSerebryanaya for investigating the root cause of the false-positive ODR violation warning. Once we upgrade to LLVM version using the fix at llvm/llvm-project@1ada819, these workarounds might not be necessary.

Other improvements:
- Fix log rewriting for the `std::__1:: prefix`, to avoid the extra curly braces.
- Replace the test host name with `{test_host_name}` when rewriting the log.
- thirdparty_tool should be able to read the GitHub token from a file specified by an environment variable (`YB_GITHUB_TOKEN_FILE_PATH`).
- Update llvm-installer Python module version to pick up LLVM archives where clang and lld executables have been built with LTO for increased compilation speed.

Test Plan: Jenkins

Reviewers: nserebryanaya, steve.varnau

Reviewed By: nserebryanaya, steve.varnau

Differential Revision: https://phabricator.dev.yugabyte.com/D20487
mbautin added a commit to yugabyte/llvm-project that referenced this issue Nov 30, 2022
@mbautin
Temporarily remove the check for global variables being properly alig…
7ab3224 
…ned.

See yugabyte/yugabyte-db#14672 for details.
jayant07-yb pushed a commit to jayant07-yb/yugabyte-db that referenced this issue Dec 7, 2022
@mbautin
[yugabyte#14672] Use Clang 15 for ASAN
7007f13 
Summary:
Use Clang 15 for ASAN (instead of Clang 13 used prior to this diff). This requires using an updated version of LLVM which suppresses an ODR violation error (yugabyte#14672) due to issues with symbol alignment reported by ASAN in Clang 14 or later ( our custom patch is yugabyte/llvm-project@0b8d118 ), as well as specifying special flags (`-mllvm -asan-use-private-alias=1`). Thanks to @NatashaSerebryanaya for investigating the root cause of the false-positive ODR violation warning. Once we upgrade to LLVM version using the fix at llvm/llvm-project@1ada819, these workarounds might not be necessary.

Other improvements:
- Fix log rewriting for the `std::__1:: prefix`, to avoid the extra curly braces.
- Replace the test host name with `{test_host_name}` when rewriting the log.
- thirdparty_tool should be able to read the GitHub token from a file specified by an environment variable (`YB_GITHUB_TOKEN_FILE_PATH`).
- Update llvm-installer Python module version to pick up LLVM archives where clang and lld executables have been built with LTO for increased compilation speed.

Test Plan: Jenkins

Reviewers: nserebryanaya, steve.varnau

Reviewed By: nserebryanaya, steve.varnau

Differential Revision: https://phabricator.dev.yugabyte.com/D20487
mbautin added a commit to yugabyte/llvm-project that referenced this issue Mar 28, 2023
@mbautin
Temporarily remove the check for global variables being properly alig…
6b9d30d 
…ned.

See yugabyte/yugabyte-db#14672 for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mbautin mbautin

@NatashaSerebryanaya NatashaSerebryanaya

Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
3 participants
@mbautin @yugabyte-ci @NatashaSerebryanaya