Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky health trait test #4922

Closed
rinaldodev opened this issue Nov 20, 2023 · 7 comments
Closed

Fix flaky health trait test #4922

rinaldodev opened this issue Nov 20, 2023 · 7 comments

Comments

@rinaldodev
Copy link
Contributor

rinaldodev commented Nov 20, 2023

What happened?

While running this test,
I noticed the Integration's Ready condition is true for a while before becoming false,
which in turns makes this assertion fail sometimes,
making the test flaky.

- firstTruthyTime: "2023-11-20T13:02:00Z"
  lastTransitionTime: "2023-11-20T13:02:00Z"
  lastUpdateTime: "2023-11-20T13:02:00Z"
  message: 1/1 ready replicas
  reason: DeploymentReady
  status: "True"
  type: Ready

See comment #4922 (comment)

Steps to reproduce

Run this e2e test a few times until it fails.

Relevant log output

apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  annotations:
    camel.apache.org/operator.id: test-305d00bf-726e-4415-9567-2573502fa0bf
  creationTimestamp: "2023-11-20T12:49:24Z"
  generation: 1
  name: never-readypnryt
  namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  resourceVersion: "19430"
  uid: aa15f5f4-1343-4d20-b61c-48cc8f187298
spec:
  sources:
  - content: |
      /*
       * Licensed to the Apache Software Foundation (ASF) under one or more
       * contributor license agreements.  See the NOTICE file distributed with
       * this work for additional information regarding copyright ownership.
       * The ASF licenses this file to You under the Apache License, Version 2.0
       * (the "License"); you may not use this file except in compliance with
       * the License.  You may obtain a copy of the License at
       *
       *      http://www.apache.org/licenses/LICENSE-2.0
       *
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */

      import org.apache.camel.builder.RouteBuilder;

      public class NeverReady extends RouteBuilder {
          @Override
          public void configure() throws Exception {
              from("timer:tick").id("never-ready")
                  .to("controlbus:route?routeId=never-ready&action=stop&async=true")
                  .setHeader("m").constant("string!")
                  .setBody().simple("Magic${header.m}")
                  .log("${body}");
          }
      }
    name: NeverReady.java
  traits:
    health:
      enabled: true
      livenessProbeEnabled: false
      livenessScheme: HTTP
      readinessProbeEnabled: true
      readinessScheme: HTTP
      startupProbeEnabled: false
      startupScheme: HTTP
status:
  conditions:
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: test-305d00bf-726e-4415-9567-2573502fa0bf/test-305d00bf-726e-4415-9567-2573502fa0bf
    reason: IntegrationPlatformAvailable
    status: "True"
    type: IntegrationPlatformAvailable
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: explicitly disabled by the user
    reason: masterTraitConfiguration
    status: "True"
    type: TraitInfo
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: kit-cldlbl2hqaes73e5b8j0
    reason: IntegrationKitAvailable
    status: "True"
    type: IntegrationKitAvailable
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: deployment name is never-readypnryt
    reason: DeploymentAvailable
    status: "True"
    type: DeploymentAvailable
  - firstTruthyTime: "2023-11-20T13:02:00Z"
    lastTransitionTime: "2023-11-20T13:02:00Z"
    lastUpdateTime: "2023-11-20T13:02:00Z"
    message: 1/1 ready replicas
    reason: DeploymentReady
    status: "True"
    type: Ready
  dependencies:
  - camel:controlbus
  - camel:timer
  - mvn:org.apache.camel.k:camel-k-runtime
  - mvn:org.apache.camel.quarkus:camel-quarkus-java-joor-dsl
  - mvn:org.apache.camel.quarkus:camel-quarkus-microprofile-health
  digest: vZi_hPRbPLDXMUuj2SPoFJLW2OPdK7PcY7M79Rh_eGB8
  image: 10.98.173.76/test-305d00bf-726e-4415-9567-2573502fa0bf/camel-k-kit-cldlbl2hqaes73e5b8j0@sha256:15df1691ea5f08295324eb828228c69335bfe56ef4c544645e7813ad829539e5
  integrationKit:
    name: kit-cldlbl2hqaes73e5b8j0
    namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  lastInitTimestamp: "2023-11-20T12:49:24Z"
  observedGeneration: 1
  phase: Running
  platform: test-305d00bf-726e-4415-9567-2573502fa0bf
  profile: Kubernetes
  replicas: 1
  runtimeProvider: quarkus
  runtimeVersion: 3.2.0
  selector: camel.apache.org/integration=never-readypnryt
  version: 2.2.0-SNAPSHOT

Camel K version

2.1.0

@rinaldodev rinaldodev added the kind/bug Something isn't working label Nov 20, 2023
@rinaldodev
Copy link
Contributor Author

Would someone please update the labels on this issue?

@squakez
Copy link
Contributor

squakez commented Nov 21, 2023

@rinaldodev thanks for reporting. What labels do you think it is required? Please also let me know if you are going to work on this so I can assign to you.

@rinaldodev
Copy link
Contributor Author

@squakez I believe that one is fine. Yes, you can assign it to me, thanks.

@rinaldodev
Copy link
Contributor Author

To bring more clarity to this issue: this appears to be caused by the camel-routes health check reporting as Ready before the routes had some time to load. See https://issues.apache.org/jira/browse/CAMEL-20242.

@squakez squakez added area/test/flakiness and removed kind/bug Something isn't working labels Apr 10, 2024
@squakez
Copy link
Contributor

squakez commented Apr 10, 2024

@rinaldodev are you planning to work on this for next 2.4 release? Should I add it to the milestone?

@christophd
Copy link
Contributor

Some more analysis on this flaky test can be found in #5345 which is a duplicate of this one

christophd added a commit to christophd/camel-k that referenced this issue Apr 10, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status
christophd added a commit to christophd/camel-k that referenced this issue Apr 10, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status
christophd added a commit that referenced this issue Apr 10, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status
christophd added a commit to christophd/camel-k that referenced this issue Apr 11, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status
christophd added a commit that referenced this issue Apr 11, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status
squakez pushed a commit to jboss-fuse/camel-k that referenced this issue Apr 11, 2024
- Avoid failing assertion on condition status ready=false due to temporary deployment ready condition status

(cherry picked from commit apache/camel-k@1c40a392f)
@christophd
Copy link
Contributor

Fixed with #5346

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants