cylc clean improvements #4440
cylc clean improvements #4440
Conversation
Traceback occurred when using --remote option and reg was parent of 1 run dir
- Use segregated streams (stdout and stderr) instead of just stderr, allowing more useful info logged during remote clean - Log which files are being deleted in single `-v` (verbose) mode - Tidy
Do not try to clean in parent dir for `cylc clean foo --rm something` where foo contains 1 run dir (i.e. only clean in the run dir)
Fixes possible typo in 44802cb
MetRonnie
requested review from
oliver-sanders
and removed request for
hjoliver
cylc/flow/pathutil.py
Outdated
| @@ -260,24 +261,24 @@ def remove_dir_and_target(path: Union[Path, str]) -> None: | |||
| Args: | |||
| path: the absolute path of the directory to delete. | |||
| """ | |||
| log = LOG.info if cylc.flow.flags.verbosity == 1 else LOG.debug | |||
There was a problem hiding this comment.
I think these should stay at LOG.debug, the visibility of the messages being decided by the level of the log handlers.
Is there a reason that approach won't work here?
There was a problem hiding this comment.
Ideally there would be a log level between INFO and DEBUG for the -v level, I suppose. Or are you saying we can manually set the CYLC_LOG handler level to DEBUG for cylc clean -v, inside the init_clean function?
There was a problem hiding this comment.
Ok, got it, DEBUG messages don't show in VERBOSE mode so you made them INFO level to make them visible.
VERBOSE mode seems a bit pointless ATM, I'm trying to remember what the original idea was...
Options:
- We could create a custom log level for VERBOSE (
loggingsupports this).- However, do we really need it? Would it really be helpful to separate log messages like this?
- (related) We have talked about creating a TASK log level (at ~ the same level as info) to help separate task and scheduler events in the log.
- We could set the log level to DEBUG in verbose mode
- I've forgotten what the original idea was, but this would make sense.
With (2) the levels would become:
QUIET
Shows WARNING or higher.
INFO
Shows INFO or higher.
VERBOSE
Shows DEBUG or higher.
DEBUG
Shows DEBUG or higher with full traceback for debug purposes.
There was a problem hiding this comment.
I've decided to make the cylc.flow.pathutil functions for deleting files/dirs/symlinks log at INFO level so that it's clear what's being deleted. This undoes 6066648. I hope this is fair - it only logs the bare minimum of deleted things (if it's removing a directory, it only logs that, not any constituent files; if a glob matches children of a matching parent, it only logs the parent etc). The only places where the logging may get a bit much are when the workflow has run on many different install targets, or when doing --rm 'log/job/*' instead of --rm log/job for a workflow with many cycle points.
- Make '-v' log at DEBUG level (so only difference with '-vv' is that turns on traceback) - Make file deletion log at INFO level during cylc clean
Otherwise daemonised scheduler hangs in quiet mode because it tries to parse this info from the log file
|
Test failures seem to be flakiness only |
This is a series of changes with no associated Issue.
cylc clean myflow --remotewhere~/cylc-run/myflowcontains exactly 1 run dirINFOlevelcylc cleanonly: Make everything below logging levelWARNINGgo to stdout instead of stderr; this allows stdout to be logged during remote clean so the above 2 bullet points can work for remote clean--debug/-vvmode, not-v(fixes possible typo in 44802cb#diff-e64f339efa9420b39fef7ef5bdee04a49751d7ed6b2446aa048ca1b8a7fa5e4b)-vmode logDEBUGlevel stuff (no traceback), because otherwise it's pretty pointlesscylc clean myflow/runNto infer the run numbercylc clean myflow --rm something(where~/cylc-run/myflowcontains exactly 1 run dir) from trying to delete~/cylc-run/myflow/something(i.e. it will only delete~/cylc-run/myflow/run1/something)cylc clean myflow --rm '..'from deleting the~/cylc-rundirectory!Requirements check-list
CONTRIBUTING.mdand added my name as a Code Contributor.setup.pyandconda-environment.yml.