Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failing e2e tests before 1.28 code freeze #2904

Closed
neolit123 opened this issue Jul 14, 2023 · 4 comments
Closed

failing e2e tests before 1.28 code freeze #2904

neolit123 opened this issue Jul 14, 2023 · 4 comments
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone

Comments

@neolit123
Copy link
Member

we have a number of failures at testgrid right now.

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-kubelet-1-26-on-latest
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-kubelet-1-26-on-latest/1679579540906577920/artifacts/kinder-xony-control-plane-1/kubelet.log

Jul 13 19:57:22 kinder-xony-control-plane-1 kubelet[1227]: E0713 19:57:22.733672    1227 run.go:74] "command failed" err="failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to decode: json: cannot unmarshal string into Go struct field LoggingConfiguration.logging.flushFrequency of type time.Duration, path: /var/lib/kubelet/config.yaml"
Jul 13 19:57:22 kinder-xony-control-plane-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

same error in:
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-latest-on-1-27
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-kubelet-1-27-on-latest

we haven't touched the kubeletconfiguration that kubeadm prepares for the kubelet, which means that something changed in the kubelet or the LoggingConfiguration, which i believe comes from . strange, either way.

need to investigate.

@neolit123 neolit123 added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. labels Jul 14, 2023
@neolit123 neolit123 added this to the v1.28 milestone Jul 14, 2023
@neolit123
Copy link
Member Author

neolit123 commented Jul 14, 2023

this changed last month:
https://github.com/kubernetes/component-base/blob/master/logs/api/v1/types.go#L48

here is what we produce from kubeadm at master:

$ kubeadm config print init-defaults --component-configs=KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0 # <-----------------------
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

	// Maximum time between log flushes.
	// If a string, parsed as a duration (i.e. "1s")
	// If an int, the maximum number of nanoseconds (i.e. 1s = 1000000000).
	// Ignored if the selected logging backend writes log messages without buffering.
	FlushFrequency TimeOrMetaDuration `json:"flushFrequency"`

so our 0 should be compatible.

ok, looks like this merged yesterday.
kubernetes/kubernetes#117800

i will log a k/k issue.

@chendave
Copy link
Member

/cc

@neolit123
Copy link
Member Author

more details here:
kubernetes/kubernetes#119325

@neolit123
Copy link
Member Author

tests are green now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

No branches or pull requests

2 participants