[RUNTIME] Add libbacktrace for backtraces with line numbers #7153

tkonolige · 2020-12-22T23:32:15Z

Added libbacktrace to 3rdparty
Changed build settings to give absolute paths in debug symbols
Move CHECK and LOG to tvm/support/logging.h
Rename tvm::Error to tvm::CompileError
Create new tvm::Error that contains structured information including a backtrace.
Replace dmlc::Error with new tvm::Error.
Unify dmlc headers to include/tvm/support/dmlc.h and hide LOG and CHECK macros from dmlc.
Rename ICHECK, CHECK, and LOG to TVM_ICHECK, TVM_CHECK, TVM_LOG.

This branch is not up to date with main because the changes to check hit a lot of places in the codebase. I'll update it when we are in agreement to merge it.

RFC: https://discuss.tvm.apache.org/t/rfc-line-numbers-in-backtraces/8711

@areusch @junrushao1994 @u99127

junrushao · 2020-12-22T23:59:43Z

Thank you Tristan for the hard work! Would love to ask some questions just for clarification on the forum first :-)

Would you like to also copy your bullet points to the forum as well? Thanks!

junrushao

Let's split the renaming to another PR so that this one could be more reviewable...

include/tvm/target/tag.h

include/tvm/support/with.h

CMakeLists.txt

cmake/modules/Libbacktrace.cmake

include/tvm/support/logging.h

cmake/modules/Libbacktrace.cmake

junrushao · 2020-12-23T19:30:41Z

I am not the best guy to review the cmake scripts and the cross-platform part (especially windows). I know a really expert @rkimball - would you like to take a second look if you have time? Thanks a lot!

include/tvm/ir/attrs.h

src/support/logging.cc

src/tir/transforms/lower_thread_allreduce.cc

tkonolige · 2021-01-22T19:05:34Z

@tqchen @junrushao1994 @antinucleon @areusch dmlc-core has merged the changes needed for the PR. I've update this PR to inject logging macros into dmlc-core, so we now use our new logging macros everywhere. I think this PR is ready for a new round of reviews when you get a chance.

src/runtime/c_runtime_api.cc

include/tvm/runtime/logging.h

tkonolige · 2021-01-25T17:18:00Z

Right now I've put the Error types in the tvm::runtime namespace and then re-exported them from the tvm namespace. Should I just put them in the tvm namespace instead?

tkonolige · 2021-02-08T17:20:11Z

@junrushao1994 @tqchen @areusch CI is all green now. Could I get a re-review?

areusch · 2021-02-08T17:49:56Z

include/tvm/ir/diagnostic.h

@@ -37,6 +37,15 @@ namespace tvm {
 using tvm::parser::SourceMap;
 using tvm::runtime::TypedPackedFunc;

+/*! \brief The diagnostic level, controls the printing of the message. */
+enum class DiagnosticLevel : int {
+  kBug = 10,


should we use terminology that reflects logging levels here?

These are levels for feedback to the user from compiling a relay expression. I'm not sure if it makes sense to merge them as these are user-facing while logging is more for developers.

I think logging will be for users when it is possible to adjust the c log level at runtime

So you want LOG(BUG), LOG(ERROR), LOG(WARNING), LOG(NOTE), LOG(HELP)? Our current levels are INFO, WARNING, ERROR, FATAL. We cause alias INFO to NOTE and then add BUG and HELP?

I actually don't mind these labels for indicating what a help message contains, but it would be helpful to comment the translation to log levels here so I don't have to look that up. I imagine people will try to work with these labels but will be approaching this from "the build log is too spammy"

There isn't a translation to log messages here. DiagnosticLevel is used within relay and TIR to determine how to print the message (these messages are never hidden or filtered).

cmake/config.cmake

include/tvm/runtime/logging.h

tutorials/auto_scheduler/tune_network_x86.py

tkonolige · 2021-03-17T17:22:51Z

I've switched this to using a submodule.

junrushao · 2021-03-18T06:39:25Z

I built and tested locally the backtrace using some existing testcases, and here is the output:

▶ python tests/python/unittest/test_target_target.py
Traceback (most recent call last):
  File "tests/python/unittest/test_target_target.py", line 229, in <module>
    test_target_host_warning()
  File "tests/python/unittest/test_target_target.py", line 218, in test_target_host_warning
    tgt = tvm.target.Target("cuda --host nvidia/jetson-nano", "llvm")
  File "/home/jrshao/Projects/tvm-dev/python/tvm/target/target.py", line 100, in __init__
    self.__init_handle_by_constructor__(
  File "/home/jrshao/Projects/tvm-dev/python/tvm/_ffi/_ctypes/object.py", line 136, in __init_handle_by_constructor__
    handle = __init_by_constructor__(fconstructor, args)
  File "/home/jrshao/Projects/tvm-dev/python/tvm/_ffi/_ctypes/packed_func.py", line 260, in __init_handle_by_constructor__
    raise get_last_ffi_error()
ValueError: Traceback (most recent call last):
  7: TVMFuncCall
        at /home/jrshao/Projects/tvm-dev/src/runtime/c_runtime_api.cc:481
  6: tvm::runtime::PackedFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /home/jrshao/Projects/tvm-dev/include/tvm/runtime/packed_func.h:1158
  5: std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /usr/include/c++/10/bits/std_function.h:622
  4: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), void (*)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /usr/include/c++/10/bits/std_function.h:291
  3: std::enable_if<std::__and_<std::is_void<void>, std::__is_invocable<void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*> >::value, void>::type std::__invoke_r<void, void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /usr/include/c++/10/bits/invoke.h:153
  2: void std::__invoke_impl<void, void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(std::__invoke_other, void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /usr/include/c++/10/bits/invoke.h:60
  1: tvm::TargetInternal::ConstructorDispatcher(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
        at /home/jrshao/Projects/tvm-dev/src/target/target.cc:471
  0: tvm::Target::Target(tvm::Target, tvm::Target)
        at /home/jrshao/Projects/tvm-dev/src/target/target.cc:377
  File "/home/jrshao/Projects/tvm-dev/src/target/target.cc", line 377
ValueError: Check failed: !n->host.defined() == false: Adding a host to a target whose host field has been defined

Have to say it is really neat! Thank you so much Tristan!

CC: @comaniac @zhiics @yzhliu @icemelon9 @kevinthesun

junrushao

Looks good to me.

Some potential improvements in the future:

Use cmake for libbacktrace in the future. There is no official support but an ongoing PR on the upstream: CMake support ianlancetaylor/libbacktrace#19. Note that it seems to support windows :-)
Not sure if it is a good idea, but perhaps we can strip some unreadable frames in the STL, like this one:

  3: std::enable_if<std::__and_<std::is_void<void>, std::__is_invocable<void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*> >::value, void>::type std::__invoke_r<void, void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(void (*&)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /usr/include/c++/10/bits/invoke.h:153

junrushao · 2021-03-18T06:54:47Z

I googled around and found some related cmake support for libbacktrace, like libgo: https://go.googlesource.com/gollvm/+/refs/heads/master/cmake/modules/LibbacktraceUtils.cmake

@tkonolige do you think we should switch libbacktrace to using cmake, as the tvm project is built with cmake?

tqchen · 2021-03-18T13:08:33Z

Thanks @tkonolige @rkimball @junrushao1994 @areusch @antinucleon . This PR is now merged

tkonolige · 2021-03-18T18:48:36Z

Thanks everyone for putting the time in to review this. I know it was a big PR.

apivovarov · 2021-03-31T00:23:00Z

@tkonolige Looks like libbacktrace build does not look at CMAKE_CXX_COMPILER and CMAKE_C_COMPILER flags.
Regardless of their values it builds libbacktrace for host platform (x86_64)

$ cmake .. \
-DCMAKE_C_COMPILER=mips-linux-uclibc-gnu-gcc \
-DCMAKE_CXX_COMPILER=mips-linux-uclibc-gnu-g++ \
-DCMAKE_C_FLAGS="-march=mips32r2 -DDMLC_LOG_STACK_TRACE=0" \
-DCMAKE_CXX_FLAGS="-march=mips32r2 -D_GLIBCXX_USE_C99 -DDMLC_LOG_STACK_TRACE=0"
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/mips-linux-uclibc-gnu-gcc
-- Check for working C compiler: /home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/mips-linux-uclibc-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/mips-linux-uclibc-gnu-g++
-- Check for working CXX compiler: /home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/mips-linux-uclibc-gnu-g++ -- works
...
[  7%] Performing configure step for 'project_libbacktrace'
cd /home/alex/workspace/tvm/build-wyze/libbacktrace && /home/alex/workspace/tvm/cmake/libs/../../3rdparty/libbacktrace/configure --prefix=/home/alex/workspace/tvm/build-wyze/libbacktrace --with-pic
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
...
/bin/bash ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I/home/alex/workspace/tvm/cmake/libs/../../3rdparty/libbacktrace    -funwind-tables -frandom-seed=atomic.lo -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wmissing-format-attribute -Wcast-qual  -g -O2 -c -o atomic.lo /home/alex/workspace/tvm/cmake/libs/../../3rdparty/libbacktrace/atomic.c
...
[ 69%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/graph/graph_runtime.cc.o
/home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/mips-linux-uclibc-gnu-g++  -DDMLC_USE_FOPEN64=0 -DDMLC_USE_LOGGING_LIBRARY="<tvm/runtime/logging.h>" -DNDEBUG -DNDEBUG=1 -DTVM_INDEX_DEFAULT_I64=1 -DTVM_THREADPOOL_USE_OPENMP=0 -DTVM_USE_LIBBACKTRACE=1 -DUSE_FALLBACK_STL_MAP=0 -DTVM_GRAPH_RUNTIME_DEBUG -I/home/alex/workspace/tvm/include -isystem /home/alex/workspace/tvm/3rdparty/dlpack/include -isystem /home/alex/workspace/tvm/3rdparty/dmlc-core/include -isystem /home/alex/workspace/tvm/3rdparty/rang/include -isystem /home/alex/workspace/tvm/3rdparty/compiler-rt -isystem /home/alex/workspace/tvm/3rdparty/picojson -I/home/alex/workspace/tvm/build-wyze/libbacktrace/include -I/home/alex/workspace/tvm/3rdparty/libcrc/include  -std=c++14 -O2 -Wall -fPIC -march=mips32r2 -D_GLIBCXX_USE_C99 -DDMLC_LOG_STACK_TRACE=0   -o CMakeFiles/tvm_runtime_objs.dir/src/runtime/graph/graph_runtime.cc.o -c /home/alex/workspace/tvm/src/runtime/graph/graph_runtime.cc
...
/home/alex/workspace/mips-gcc540-glibc222-64bit-r3.3.0/bin/../lib/gcc/mips-linux-gnu/5.4.0/../../../../mips-linux-gnu/bin/ld: libbacktrace/lib/libbacktrace.a(fileline.o): Relocations in generic ELF (EM: 62)
libbacktrace/lib/libbacktrace.a: error adding symbols: File in wrong format

At least it can be disabled -DUSE_LIBBACKTRACE=0...

junrushao · 2021-03-31T00:54:25Z

@apivovarov I believe Tristan is working on this: #7705 (comment)

) * [RUNTIME] Add libbacktrace for backtraces with line numbers Co-authored-by: Robert Kimball <[email protected]>

junrushao reviewed Dec 23, 2020

View reviewed changes

include/tvm/target/tag.h Outdated Show resolved Hide resolved

include/tvm/support/with.h Outdated Show resolved Hide resolved

CMakeLists.txt Outdated Show resolved Hide resolved

areusch reviewed Dec 23, 2020

View reviewed changes

cmake/modules/Libbacktrace.cmake Show resolved Hide resolved

include/tvm/support/logging.h Outdated Show resolved Hide resolved

antinucleon suggested changes Dec 23, 2020

View reviewed changes

cmake/modules/Libbacktrace.cmake Show resolved Hide resolved

cmake/modules/Libbacktrace.cmake Show resolved Hide resolved

tqchen assigned junrushao Dec 23, 2020

tqchen added the status: need review label Dec 23, 2020

tqchen requested changes Dec 23, 2020

View reviewed changes

include/tvm/ir/attrs.h Outdated Show resolved Hide resolved

tqchen assigned tqchen and junrushao and unassigned junrushao Dec 23, 2020

tqchen added the status: need update need update based on feedbacks label Dec 26, 2020

tqchen reviewed Jan 6, 2021

View reviewed changes

src/support/logging.cc Outdated Show resolved Hide resolved

tqchen reviewed Jan 6, 2021

View reviewed changes

src/tir/transforms/lower_thread_allreduce.cc Outdated Show resolved Hide resolved

tkonolige force-pushed the libbacktrace branch 2 times, most recently from 20b743f to c96b076 Compare January 22, 2021 19:02

tqchen requested changes Jan 23, 2021

View reviewed changes

tkonolige force-pushed the libbacktrace branch 3 times, most recently from f7f94b1 to be07941 Compare February 1, 2021 23:35

tkonolige force-pushed the libbacktrace branch 2 times, most recently from 08ee22a to f5b083c Compare February 5, 2021 17:20

areusch reviewed Feb 8, 2021

View reviewed changes

cmake/config.cmake Show resolved Hide resolved

areusch reviewed Feb 8, 2021

View reviewed changes

include/tvm/runtime/logging.h Show resolved Hide resolved

areusch reviewed Feb 8, 2021

View reviewed changes

include/tvm/runtime/logging.h Outdated Show resolved Hide resolved

areusch reviewed Feb 8, 2021

View reviewed changes

tutorials/auto_scheduler/tune_network_x86.py Outdated Show resolved Hide resolved

tkonolige added 9 commits March 16, 2021 10:17

fix workload

c5b9b12

Accidentally droped dmlc dep

7c68da1

different ulimit change

ccbe212

finally fix stack overflow

481461b

one global backtrace object

f73b7e7

rename

2bc5dd1

formatting

3ef265f

formatting

70e8d61

move defines into header for jni

1cd3306

tkonolige force-pushed the libbacktrace branch from 9ac75f1 to 1106a2e Compare March 16, 2021 18:28

switch to submodule

0999ce1

tkonolige force-pushed the libbacktrace branch from 1106a2e to 0999ce1 Compare March 16, 2021 18:50

Trigger CI

4711201

junrushao approved these changes Mar 18, 2021

View reviewed changes

tqchen approved these changes Mar 18, 2021

View reviewed changes

tqchen merged commit c976a07 into apache:main Mar 18, 2021

tqchen added status: accepted and removed status: need review status: need update need update based on feedbacks labels Mar 18, 2021

tkonolige mentioned this pull request Mar 19, 2021

Use dmlc-core as a cmake project #7701

Closed

comaniac mentioned this pull request Mar 24, 2021

[Bug] Line backtrace results in stack overflow #7729

Closed

trevor-m pushed a commit to trevor-m/tvm that referenced this pull request May 6, 2021

[RUNTIME] Add libbacktrace for backtraces with line numbers (apache#7153

14e8449

) * [RUNTIME] Add libbacktrace for backtraces with line numbers Co-authored-by: Robert Kimball <[email protected]>

trevor-m pushed a commit to neo-ai/tvm that referenced this pull request May 11, 2021

[RUNTIME] Add libbacktrace for backtraces with line numbers (apache#7153

4930b19

) * [RUNTIME] Add libbacktrace for backtraces with line numbers Co-authored-by: Robert Kimball <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RUNTIME] Add libbacktrace for backtraces with line numbers #7153

[RUNTIME] Add libbacktrace for backtraces with line numbers #7153

tkonolige commented Dec 22, 2020

junrushao commented Dec 22, 2020 •

edited

Loading

junrushao left a comment

junrushao commented Dec 23, 2020

tkonolige commented Jan 22, 2021

tkonolige commented Jan 25, 2021

tkonolige commented Feb 8, 2021

areusch Feb 8, 2021

junrushao Feb 8, 2021

tkonolige Feb 8, 2021

areusch Feb 23, 2021

tkonolige Feb 23, 2021

areusch Feb 23, 2021

tkonolige Feb 23, 2021

tkonolige commented Mar 17, 2021

junrushao commented Mar 18, 2021

junrushao left a comment •

edited

Loading

junrushao commented Mar 18, 2021

tqchen commented Mar 18, 2021

tkonolige commented Mar 18, 2021

apivovarov commented Mar 31, 2021 •

edited

Loading

junrushao commented Mar 31, 2021

[RUNTIME] Add libbacktrace for backtraces with line numbers #7153

[RUNTIME] Add libbacktrace for backtraces with line numbers #7153

Conversation

tkonolige commented Dec 22, 2020

junrushao commented Dec 22, 2020 • edited Loading

junrushao left a comment

Choose a reason for hiding this comment

junrushao commented Dec 23, 2020

tkonolige commented Jan 22, 2021

tkonolige commented Jan 25, 2021

tkonolige commented Feb 8, 2021

areusch Feb 8, 2021

Choose a reason for hiding this comment

junrushao Feb 8, 2021

Choose a reason for hiding this comment

tkonolige Feb 8, 2021

Choose a reason for hiding this comment

areusch Feb 23, 2021

Choose a reason for hiding this comment

tkonolige Feb 23, 2021

Choose a reason for hiding this comment

areusch Feb 23, 2021

Choose a reason for hiding this comment

tkonolige Feb 23, 2021

Choose a reason for hiding this comment

tkonolige commented Mar 17, 2021

junrushao commented Mar 18, 2021

junrushao left a comment • edited Loading

Choose a reason for hiding this comment

junrushao commented Mar 18, 2021

tqchen commented Mar 18, 2021

tkonolige commented Mar 18, 2021

apivovarov commented Mar 31, 2021 • edited Loading

junrushao commented Mar 31, 2021

junrushao commented Dec 22, 2020 •

edited

Loading

junrushao left a comment •

edited

Loading

apivovarov commented Mar 31, 2021 •

edited

Loading