Need a good way to read/find failed job logs #3004
Comments
|
We sadly don't have anything to filter down the Toil logs to just the good parts right now. But we ought to come up with something. It's a common usability complaint that the logs are very verbose and hard to find the actual problem in, between all the stuff Toil prints at WARNING level, the long prefixes it adds to job log lines, and the stuff at the end where it just complains nonspecifically that jobs failed. The logs ought to still be in the job store AFAIK, if you didn't say |
adamnovak
changed the title
|
Thanks @adamnovak. I found that I can get the logs using I can create a PR to fix this bytes conversion issue if you would like. |
|
That would be great! We're still tracking down places where we missed
string/bytes issues in the Python 3 switch. Some of these utilities
aren't under CI sadly.
…On 3/9/20, John Bradley ***@***.***> wrote:
Thanks @adamnovak. I found that I can get the logs using `toil status
--printLogs <jobstore_path>` when using python 2. With python 3 I get an
error:
```
Traceback (most recent call last):
File "/Users/jpb67/Documents/work/toil_log_fun/venv/bin/toil", line 10, in
<module>
sys.exit(main())
File
"/Users/jpb67/Documents/work/toil_log_fun/venv/lib/python3.7/site-packages/toil/utils/toilMain.py",
line 35, in main
module.main()
File
"/Users/jpb67/Documents/work/toil_log_fun/venv/lib/python3.7/site-packages/toil/utils/toilStatus.py",
line 365, in main
status.printJobLog()
File
"/Users/jpb67/Documents/work/toil_log_fun/venv/lib/python3.7/site-packages/toil/utils/toilStatus.py",
line 83, in printJobLog
msg += fH.read()
TypeError: can only concatenate str (not "bytes") to str
```
I can create a PR to fix this bytes conversion issue if you would like.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#3004 (comment)
|
Test to reproduce `TypeError: can only concatenate str (not "bytes") to str` error.
Fix for problem raised in issue DataBiosphere#3004
Test to reproduce `TypeError: can only concatenate str (not "bytes") to str` error.
Fix for problem raised in issue #3004
|
It turns out that the I'm going to hook it back up and also change how logs are dumped in the middle of the run to look nicer. |
When a CWL workflow fails it can be difficult to find the error messages generated by the failing command. Is there any guidance on how this data could be pulled out from larger logs?
As a simple example here is a simple CWL tool that has an invalid argument:
You can run it with toil like so:
toil-cwl-runner wc-tool.cwl --input_file wc-tool.cwlThe following output is printed:
Github does a pretty good job lining up the above text, but in the terminal it is harder to read due the line wrapping.
Are there any commands or scripts that might parse output logs or other data and provide easier to read logs?
Thanks.
┆Issue is synchronized with this Jira Task
┆Issue Number: TOIL-524
The text was updated successfully, but these errors were encountered: