-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[cleanup] Log compile errors for AOT tests #10214
Conversation
6d4ae98
to
65324a8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @driazati , it will make debugging the mysterious CI failures much easier!
7ad65a1
to
8325ee2
Compare
proc.wait() | ||
if proc.returncode != 0: | ||
raise RuntimeError( | ||
f"Subprocess failed: {cmd}\nstdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last run on CI luckily hit a flaky test, although the Runtime Error doesn't really give many details currently (other than the command that failed):
RuntimeError: Subprocess failed: make -f /workspace/tests/python/relay/aot/corstone300.mk build_dir=/tmp/tmpqlaqvgf3/test/build CFLAGS='-DTVM_RUNTIME_ALLOC_ALIGNMENT_BYTES=8 ' TVM_ROOT=/workspace/tests/python/relay/aot/../../../.. AOT_TEST_ROOT=/workspace/tests/python/relay/aot CODEGEN_ROOT=/tmp/tmpqlaqvgf3/test/codegen STANDALONE_CRT_DIR=/workspace/build/standalone_crt FVP_DIR=/opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4/ run
E stdout:
E <_io.TextIOWrapper name=7 encoding='utf-8'>
E stderr:
E None
I tried to change proc.stdout
to something like proc.stdout.read()
but this just returns an empty string, presumably because the lines are already read above. There will also be no stderr since this is passed to stdout in the subprocess declaration.
We could make the error visible by adding the contents of the log file to the error message, although I don't think this is very elegant. Any other ideas?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added a side channel variable to capture the same info, seems to work locally
* removed the encode of msg that is already str Co-authored-by: lhutton1 <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
* [cleanup] Log compile errors for AOT tests See apache#10213 * Update tests/python/relay/aot/aot_test_utils.py * removed the encode of msg that is already str Co-authored-by: lhutton1 <[email protected]> Co-authored-by: driazati <[email protected]> Co-authored-by: Manupa Karunaratne <[email protected]> Co-authored-by: lhutton1 <[email protected]>
See #10213