Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky auto-instrumentation multi-container test #959

Closed
pavolloffay opened this issue Jun 30, 2022 · 0 comments · Fixed by #1072
Closed

Fix flaky auto-instrumentation multi-container test #959

pavolloffay opened this issue Jun 30, 2022 · 0 comments · Fixed by #1072
Labels
area:auto-instrumentation Issues for auto-instrumentation

Comments

@pavolloffay
Copy link
Member

e.g. https://github.com/open-telemetry/opentelemetry-operator/runs/7127161280?check_suite_focus=true

=== CONT  kuttl/harness/instrumentation-python-multicontainer
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer/1-install-app | test step failed 1-install-app
    case.go:361: failed in step 1-install-app
    case.go:363: --- Pod:kuttl-test-prepared-chigger/
        +++ Pod:kuttl-test-prepared-chigger/my-deployment-with-sidecar-7b877dc6f-7qbsf
        @@ -5,66 +5,208 @@
             instrumentation.opentelemetry.io/container-names: myapp,myrabbit
             instrumentation.opentelemetry.io/inject-python: "true"
             sidecar.opentelemetry.io/inject: "true"
        +  generateName: my-deployment-with-sidecar-7b877dc6f-
           labels:
             app: my-pod-with-sidecar
        +    pod-template-hash: 7b877dc6f
        +  managedFields:
        +  - apiVersion: v1
        +    fieldsType: FieldsV1
        +    fieldsV1:
        +      f:metadata:
        +        f:annotations:
        +          .: {}
        +          f:instrumentation.opentelemetry.io/container-names: {}
        +          f:instrumentation.opentelemetry.io/inject-python: {}
        +          f:sidecar.opentelemetry.io/inject: {}
        +        f:generateName: {}
        +        f:labels:
        +          .: {}
        +          f:app: {}
        +          f:pod-template-hash: {}
        +        f:ownerReferences:
        +          .: {}
        +          k:{"uid":"ad611419-f7c2-428f-a121-d0609a20fe04"}: {}
        +      f:spec:
        +        f:containers:
        +          k:{"name":"myapp"}:
        +            .: {}
        +            f:image: {}
        +            f:imagePullPolicy: {}
        +            f:name: {}
        +            f:resources: {}
        +            f:terminationMessagePath: {}
        +            f:terminationMessagePolicy: {}
        +          k:{"name":"myrabbit"}:
        +            .: {}
        +            f:image: {}
        +            f:imagePullPolicy: {}
        +            f:name: {}
        +            f:resources: {}
        +            f:terminationMessagePath: {}
        +            f:terminationMessagePolicy: {}
        +        f:dnsPolicy: {}
        +        f:enableServiceLinks: {}
        +        f:restartPolicy: {}
        +        f:schedulerName: {}
        +        f:securityContext: {}
        +        f:terminationGracePeriodSeconds: {}
        +    manager: kube-controller-manager
        +    operation: Update
        +    time: "2022-06-30T08:12:05Z"
        +  - apiVersion: v1
        +    fieldsType: FieldsV1
        +    fieldsV1:
        +      f:status:
        +        f:conditions:
        +          k:{"type":"ContainersReady"}:
        +            .: {}
        +            f:lastProbeTime: {}
        +            f:lastTransitionTime: {}
        +            f:status: {}
        +            f:type: {}
        +          k:{"type":"Initialized"}:
        +            .: {}
        +            f:lastProbeTime: {}
        +            f:lastTransitionTime: {}
        +            f:status: {}
        +            f:type: {}
        +          k:{"type":"Ready"}:
        +            .: {}
        +            f:lastProbeTime: {}
        +            f:lastTransitionTime: {}
        +            f:status: {}
        +            f:type: {}
        +        f:containerStatuses: {}
        +        f:hostIP: {}
        +        f:phase: {}
        +        f:podIP: {}
        +        f:podIPs:
        +          .: {}
        +          k:{"ip":"10.244.0.13"}:
        +            .: {}
        +            f:ip: {}
        +        f:startTime: {}
        +    manager: kubelet
        +    operation: Update
        +    subresource: status
        +    time: "2022-06-30T08:12:20Z"
        +  name: my-deployment-with-sidecar-7b877dc6f-7qbsf
           namespace: kuttl-test-prepared-chigger
        +  ownerReferences:
        +  - apiVersion: apps/v1
        +    blockOwnerDeletion: true
        +    controller: true
        +    kind: ReplicaSet
        +    name: my-deployment-with-sidecar-7b877dc6f
        +    uid: ad611419-f7c2-428f-a121-d0609a20fe04
         spec:
           containers:
        -  - env:
        -    - name: PYTHONPATH
        -      value: /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation
        -    - name: OTEL_TRACES_EXPORTER
        -      value: otlp_proto_http
        -    - name: OTEL_EXPORTER_OTLP_ENDPOINT
        -      value: http://localhost:4317
        -    - name: OTEL_EXPORTER_OTLP_TIMEOUT
        -      value: "20"
        -    - name: OTEL_TRACES_SAMPLER
        -      value: parentbased_traceidratio
        -    - name: OTEL_TRACES_SAMPLER_ARG
        -      value: "0.85"
        -    - name: SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
        -      value: "true"
        -    - name: OTEL_SERVICE_NAME
        -      value: my-deployment-with-sidecar
        -    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
        -    - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
        -    - name: OTEL_PROPAGATORS
        -      value: jaeger,b3
        -    - name: OTEL_RESOURCE_ATTRIBUTES
        +  - image: ghcr.io/anuraaga/flask-hello-world:latest
        +    imagePullPolicy: Always
             name: myapp
        +    resources: {}
        +    terminationMessagePath: /dev/termination-log
        +    terminationMessagePolicy: File
             volumeMounts:
             - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        -    - mountPath: /otel-auto-instrumentation
        -      name: opentelemetry-auto-instrumentation
        -  - env:
        -    - name: PYTHONPATH
        -      value: /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation
        -    - name: OTEL_TRACES_EXPORTER
        -      value: otlp_proto_http
        -    - name: OTEL_EXPORTER_OTLP_ENDPOINT
        -      value: http://localhost:4317
        -    - name: OTEL_EXPORTER_OTLP_TIMEOUT
        -      value: "20"
        -    - name: OTEL_TRACES_SAMPLER
        -      value: parentbased_traceidratio
        -    - name: OTEL_TRACES_SAMPLER_ARG
        -      value: "0.85"
        -    - name: SPLUNK_TRACE_RESPONSE_HEADER_ENABLED
        -      value: "true"
        -    - name: OTEL_SERVICE_NAME
        -      value: my-deployment-with-sidecar
        -    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
        -    - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
        -    - name: OTEL_PROPAGATORS
        -      value: jaeger,b3
        -    - name: OTEL_RESOURCE_ATTRIBUTES
        +      name: kube-api-access-7xn2m
        +      readOnly: true
        +  - image: rabbitmq
        +    imagePullPolicy: Always
             name: myrabbit
        +    resources: {}
        +    terminationMessagePath: /dev/termination-log
        +    terminationMessagePolicy: File
             volumeMounts:
             - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        -    - mountPath: /otel-auto-instrumentation
        -      name: opentelemetry-auto-instrumentation
        -  - name: otc-container
        +      name: kube-api-access-7xn2m
        +      readOnly: true
        +  dnsPolicy: ClusterFirst
        +  enableServiceLinks: true
        +  nodeName: kind-control-plane
        +  preemptionPolicy: PreemptLowerPriority
        +  priority: 0
        +  restartPolicy: Always
        +  schedulerName: default-scheduler
        +  securityContext: {}
        +  serviceAccount: default
        +  serviceAccountName: default
        +  terminationGracePeriodSeconds: 30
        +  tolerations:
        +  - effect: NoExecute
        +    key: node.kubernetes.io/not-ready
        +    operator: Exists
        +    tolerationSeconds: 300
        +  - effect: NoExecute
        +    key: node.kubernetes.io/unreachable
        +    operator: Exists
        +    tolerationSeconds: 300
        +  volumes:
        +  - name: kube-api-access-7xn2m
        +    projected:
        +      defaultMode: 420
        +      sources:
        +      - serviceAccountToken:
        +          expirationSeconds: 3607
        +          path: token
        +      - configMap:
        +          items:
        +          - key: ca.crt
        +            path: ca.crt
        +          name: kube-root-ca.crt
        +      - downwardAPI:
        +          items:
        +          - fieldRef:
        +              apiVersion: v1
        +              fieldPath: metadata.namespace
        +            path: namespace
         status:
        +  conditions:
        +  - lastProbeTime: null
        +    lastTransitionTime: "2022-06-30T08:12:05Z"
        +    status: "True"
        +    type: Initialized
        +  - lastProbeTime: null
        +    lastTransitionTime: "2022-06-30T08:12:16Z"
        +    status: "True"
        +    type: Ready
        +  - lastProbeTime: null
        +    lastTransitionTime: "2022-06-30T08:12:16Z"
        +    status: "True"
        +    type: ContainersReady
        +  - lastProbeTime: null
        +    lastTransitionTime: "2022-06-30T08:12:05Z"
        +    status: "True"
        +    type: PodScheduled
        +  containerStatuses:
        +  - containerID: containerd://4959f31b5ccd4dca0872b68bdea27ac6f4c7228af20d60e70c69bf73902672dd
        +    image: ghcr.io/anuraaga/flask-hello-world:latest
        +    imageID: ghcr.io/anuraaga/flask-hello-world@sha256:fdf1776b9924d1795554c9e4c341818f77e634ae3487b06c3efdcd1a853de55d
        +    lastState: {}
        +    name: myapp
        +    ready: true
        +    restartCount: 0
        +    started: true
        +    state:
        +      running:
        +        startedAt: "2022-06-30T08:12:10Z"
        +  - containerID: containerd://8ab70d9022ffde893d70bd019af58bd6ae8d29d2f623521b8e3ae41e5be61835
        +    image: docker.io/library/rabbitmq:latest
        +    imageID: docker.io/library/rabbitmq@sha256:ce332751d4da118e9c06b43dd74027b2aa697322606f76598aa5eccdfd117885
        +    lastState: {}
        +    name: myrabbit
        +    ready: true
        +    restartCount: 0
        +    started: true
        +    state:
        +      running:
        +        startedAt: "2022-06-30T08:12:16Z"
        +  hostIP: 172.18.0.2
           phase: Running
        +  podIP: 10.244.0.13
        +  podIPs:
        +  - ip: 10.244.0.13
        +  qosClass: BestEffort
        +  startTime: "2022-06-30T08:12:05Z"
    case.go:363: resource Pod:kuttl-test-prepared-chigger/: .spec.containers: slice length mismatch: 3 != 2
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | instrumentation-python-multicontainer events from ns kuttl-test-prepared-chigger:
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:05 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf		Scheduled	Successfully assigned kuttl-test-prepared-chigger/my-deployment-with-sidecar-7b877dc6f-7qbsf to kind-control-plane		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:05 +0000 UTC	Normal	ReplicaSet.apps my-deployment-with-sidecar-7b877dc6f		SuccessfulCreate	Created pod: my-deployment-with-sidecar-7b877dc6f-7qbsf		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:05 +0000 UTC	Normal	Deployment.apps my-deployment-with-sidecar		ScalingReplicaSet	Scaled up replica set my-deployment-with-sidecar-7b877dc6f to 1		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:07 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myapp}		Pulling	Pulling image "ghcr.io/anuraaga/flask-hello-world:latest"		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:10 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myapp}		Pulled	Successfully pulled image "ghcr.io/anuraaga/flask-hello-world:latest" in 2.648590422s		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:10 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myapp}		Created	Created container myapp		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:11 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myapp}		Started	Started container myapp		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:11 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myrabbit}		Pulling	Pulling image "rabbitmq"		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:16 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myrabbit}		Pulled	Successfully pulled image "rabbitmq" in 5.6[949](https://github.com/open-telemetry/opentelemetry-operator/runs/7127161280?check_suite_focus=true#step:5:950)43545s		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:16 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myrabbit}		Created	Created container myrabbit		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | 2022-06-30 08:12:16 +0000 UTC	Normal	Pod my-deployment-with-sidecar-7b877dc6f-7qbsf.spec.containers{myrabbit}		Started	Started container myrabbit		
    logger.go:42: 08:14:36 | instrumentation-python-multicontainer | Deleting namespace: kuttl-test-prepared-chigger
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:auto-instrumentation Issues for auto-instrumentation
Projects
None yet
1 participant