Log message severity isn't properly reported in stackdriver

[jlewi]

Below are the TFJob operator logs fetched with the following query.

gcloud --project=${PROJECT} logging read --format="table(timestamp, severity, resource.labels.container_name, jsonPayload.level, jsonPayload.msg)" --freshness=1h      --order asc         "resource.type=\"container\" resource.labels.cluster_name=\"${CLUSTER}\"  resource.labels.pod_id=\"${POD}\"  jsonPayload.job=\"kubeflow.demo-trainer-11-05-dist-gpu\"" > ~/tmp/trainer-logs-1106.txt

The log prints out the severity as reported in stackdriver as well as the level in the json message. The level indicates they are info level log messages but they are logged as severity error in stackdriver.

trainer-logs-1106.txt

[cyprnose]

I had similar problem. By default stackdriver assigns severity based on target stream (so for example: logs captured from stdout get severity "info" and ones from stderr get "error"). Python logging module's default handler sends all logs to stderr (even those with INFO level), that's why all of them has "error" severity in stackdriver.
Here is an example how to split logging between stdout and stderr: https://stackoverflow.com/questions/16061641/python-logging-split-between-stdout-and-stderr

[gaocegege]

We are not using python logging package in tf-operator, I think.

We are using "github.com/sirupsen/logrus" and its default output is STDERR. It should be easy to fix.

// Creates a new logger. Configuration should be set by changing `Formatter`,
// `Out` and `Hooks` directly on the default logger instance. You can also just
// instantiate your own:
//
//    var log = &Logger{
//      Out: os.Stderr,
//      Formatter: new(JSONFormatter),
//      Hooks: make(LevelHooks),
//      Level: logrus.DebugLevel,
//    }
//
// It's recommended to make this a global instance called `log`.
func New() *Logger {
	return &Logger{
		Out:          os.Stderr,
		Formatter:    new(TextFormatter),
		Hooks:        make(LevelHooks),
		Level:        InfoLevel,
		ExitFunc:     os.Exit,
		ReportCaller: false,
	}
}

[gaocegege]

After #1088 is merged, I can have a look.

[jtfogarty]

/kind bug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.