Structured logging

[serathius]

Enhancement Description

Define standard structure for Kubernetes log messages, add methods to klog to enforce this structure, add ability to configure Kubernetes components to produce logs in JSON format and initiate migration to structured logging.

• Kubernetes Enhancement Proposal: Structured Logging KEP
• Primary contact (assignee): @serathius
• Responsible SIGs: sig-instrumentation
• Enhancement target (which target equals to which milestone):
  • Alpha release target 1.19
  • Beta release target 1.23
  • GA release target 1.27

Alpha

• Implement text format by adding klog.InfoS, klog.ErrorS functions to k8s.io/klog (Implement InfoS and ErrorS functions klog#125, @yuzhiquan)

• Implement klog.KObj, klog.KRef to k8s.io/klog (Implement klog.KObj klog.KRef functions klog#127, @tahsinrahman)

• Upgrade klog to v2 (Upgrade to klogv2 kubernetes#89961. @dims)

• Implement --logging-format flag in k8s.io/component-base (Implement --logging-format flag in k8s.io/component-base kubernetes#89190, @yuzhiquan)

• Prepare migration instructions (@serathius, Add instructions for structured logging migration community#4793)

• Migrate selected log messages to klog.InfoSusing Migration Instructions

  • staging/src/k8s.io/apiserver (httplog.go:161, get.go:251, watcher.go:363, watcher.go:256) (@AkashSarda, Structured logging for Apiserver [Issue #1602 Structured logging] kubernetes#91833)
  • pkg/controller/ (garbagecollector.go:404, garbagecollector.go:517, controller_utils.go:603, replica_set.go:561, deployment_controller.go:484, deployment_controller.go:575, endpoints_controller.go:340, cloud_cidr_allocator.go:233, ttl_controller.go:271, replica_set.go:597, replica_set.go:225) (@tahsinrahman, Migrate to log calls to klog.InfoS and klog.ErroS for pkg/controller kubernetes#91576)
  • /plugin/pkg/auth/authorizer (node_authorizer.go:197) (@ankeesler, Migrate a single node_authorizer.go klog.Infof call to klog.InfoS kubernetes#91591)
  • pkg/scheduler (scheduler.go:604, factory.go:431, scheduler.go:715) (@tanjunchen, Migrate to log calls to klog.ErroS for pkg/scheduler kubernetes#91679)
  • pkg/kubelet (kubelet_getters.go:173) (@hase1128, Migrate klog to klog.InfoS in pkg/kubelet kubernetes#91624)
  • staging/src/k8s.io/client-go (event.go:278) (@KobayashiD27, Migrate log to klog.InfoS for staging/src/k8s.io/client-go kubernetes#91712)

• Implement json log format (@yuzhiquan, Implement json log format  kubernetes#91490)

• Update flag help to list both format options (@serathius, Update logging format flag help kubernetes#92177)

• Add --logging-format flag to kubernetes components

  • kube-apiserver (Add --logging-format flag for kube-apiserver kubernetes#91501, @tahsinrahman)
  • kube-controller-manager (Add '--logging-format' flag to kube-controller-manager kubernetes#91521, @SataQiu)
  • kubelet (add --logging-format flag to kubelet kubernetes#91532, @afrouzMashaykhi)
  • kube-scheduler (Add '--logging-format' flag to kube-scheduler kubernetes#91522, @SataQiu)

• Write documentation for new logging format (Document Structured Logging Enhancement website#21202, @serathius)

Follow-up from main tasks:

• ErrorS(nil, ...) should call loggr.Error(nil, ...) instead of loggr.Info(...) klog#153 (@tahsinrahman)
• Fix InfoS serializing Stringer subfields klog#156 (@yuzhiquan)
• InfoS uses %#s for errors instead of %q klog#158 (@tahsinrahman)
• Update klog in k/k to collect bug fixes above (@serathius, Upgrade klog to v2.1.0 kubernetes#91792)
• Structured logging behaviour changed from v2.0.0 to v2.1.0 klog#165 (@physcat)
• Update klog to collect bug fixes above (@serathius, Update k8s.io/klog to v2.2.0 kubernetes#92554)
• Remove locking from logFormatRegistry as it's not needed (@rahulchheda, Removed mutex thread locking and unlocking from logging Format registry kubernetes#92347)
• Validate if incompatible klog flags were used (@rahulchheda, (feat) Add validation for Unsupported Loggings Flags for component-base kubernetes#92394)
• Fix JSON logger verbosity (@yuzhiquan, Fix JSON logger verbose kubernetes#92788)

Beta

• Directories can be marked as migrated, preventing regression via verify script in hack. (Prevent regressions after structured logging migration kubernetes#98975, @adisky)
• Kubernetes Logging documentation is updated for structured logging. (Update Logging Conventions doc with structured logging community#5593)
• Structured logging interface design is verify by migrating one whole component (Kubelet) to structured logging. (Kubelet migration to structured logs kubernetes#98976)
• Log volume (i.e. size of output) will be benchmarked and documented for kubelet in non-JSON and JSON format. (Benchmark kubelet log sizes before and after structured log migration kubernetes#99803)
• Dedicated periodic e2e test scenario with Json enabled (Add an e2e test scenario with json logging enabled for kubelet kubernetes#99802)
• Refactor Options to use LoggingConfiguration (Move Logs options to component-base/config kubernetes#99267)
• Json format should support same set of feature flags as text format, minus those we will decide to deprecte. (Json format should support same set of feature flags as klog kubernetes#99270)
  • Add source code location to JSON logs (Add source code location information to JSON logs kubernetes#102353)
• Move klog flags to Logs flag group (Move not-deprecated klog flags to Logs flag group kubernetes#99265)
• Enabling Json format should work from component start (Kubelet logs start with unstructured logs then transition to structured logs kubernetes#100152)

Follow ups:

• Migrate trace.go)

Please let me know if someone is interested in working on one of the tasks.

[serathius]

/sig instrumentation

[serathius]

/help

[k8s-ci-robot]

@serathius:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yuzhiquan]

interested in

[tanjunchen]

/cc

[serathius]

/milestone v1.19

[k8s-ci-robot]

@serathius: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

/milestone v1.19

[hase1128]

/cc

[serathius]

KEP was marked as "implementable", so we can start working on it.
I have created first two subtasks within specific repositories. I will add more as work progresses and new tasks are unblocked.

If your interested in contributing please volunteer directly on subtasks or ask for specific task here.

[gavinfish]

/cc

[serathius]

After discussing offline with @dims we concluded that upgrading klog will be non-trivial as we need to upgrade all references at once. Upgrade cannot be done partially as it would break klog configuration.

We considered two options:

• [Preferred] Properly update of all dependencies
• Aliasing in go modules

As we have a lot of time before 1.19 cut we would like to do it properly. I will create subtask with upgrade plan.

[fenggw-fnst]

/cc

[johnbelamaric]

@serathius Enhancements shadow for v1.19 here. Do you see this as moving to alpha in 1.19?

[serathius]

Yes, I think we are on track. We are finishing klog upgrade, after that next milestones will be really fast to implement.

[harshanarayana]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale

[logicalhan]

@pohly is this dependent on contextual logging?

[pohly]

Yes. We are now doing both together.

[gnufied]

Are there any plans to migrate logs inside client-go to structured logs? https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reflector.go#L295

[pohly]

Yes, most likely. Now that interaction with slog in Go 1.21 is clear(er), we can continue like we did so far based on klog+go-logr. I'm hoping to get confirmation from others for that plan in #4181.

[dashpole]

@pohly @mengjiao-liu Based on #1602 (comment), I assume this will not progress to GA in 1.30?

[pohly]

/unassign @serathius
/assign

@dashpole: yes, let's keep it in beta. We keep chucking away at the code, but the conversion is not complete yet.

At some point we also need to find our whether JSON output really is being used. I'll need help from the SIG for this.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale