TelemetryMultipleEnginesTest fails for 7.15 on CE daily pipeline

[ThorbenLindhauer]

This issue was imported from JIRA:

Field	Value
JIRA Link	CAM-13760
Reporter	@tasso94
Has restricted visibility comments	false

---

Acceptance Criteria (Required on creation):

org.camunda.bpm.engine.test.api.mgmt.telemetry.TelemetryMultipleEnginesTest#shouldReportInitialTelemetryOnce fails on:

• 7.15
• CE daily pipeline
• Stage: old-engine-postgresql-122

Hints (Optional):

Error log <^telemetry-error.txt>

Links:

Tasks

Preview Give feedback

1. chore(qa): disable telemetry by default for all test setups #3908
   ci:postgresql ci:rolling-update +2
   
2. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1103
3. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1104
4. https://github.com/camunda/camunda-bpm-platform-maintenance/pull/1105

[ThorbenLindhauer]

This comment was imported from JIRA and written by user @tasso94

---

Reorccured:

• 7.15
• CE daily pipeline
• Stages
  • old-engine-postgres-1013
  • old-engine-postgres-132

[tasso94]

Failed again on 7.17 sidetrack pipeline in old-engine-cockroachdb-201 stage

[danielkelemen]

Failed again on 7.21 daily in old-engine-postgresql-150 stage:

https://ci-pipeline.cambpm.camunda.cloud/job/7.21/job/cambpm-ce/job/cambpm-daily/job/master/10/

Details

[2023-10-21T03:20:35.510Z] [ERROR] Failures: [2023-10-21T03:20:35.510Z] [ERROR] TelemetryMultipleEnginesTest.shouldReportMetricsPerEngine:137 [2023-10-21T03:20:35.510Z] Expected size:<2> but was:<3> in:

[danielkelemen]

Hi @mboskamp, Can you take a look at this build failure?
Applying the CI Failure rotation.

[mboskamp]

Findings:

• The latest failure is due to the test expecting two requests to the /pings telemetry endpoint but receiving three.
• Of the three requests the endpoint receives, the two that were expected have a data-collection-start-date of roughly the same date the test ran. The third one has a data-collection-start-date which is around 5 minutes before the other two before the test even started. I checked the log, and at that time, the following step in the maven reactor was executed: Building Camunda Platform - Wildfly 26 Subsystem 7.21.0-SNAPSHOT [114/115]
• The failing test is executed after the maven build is done (old-engine-test).

Payload of the three requests for easy reference:

unexpected request

{
  "body": {
    "installation": "1439ed95-7843-495f-9b5c-77cb44828e38",
    "product": {
      "name": "Camunda BPM Runtime",
      "version": "7.20.0",
      "edition": "community",
      "internals": {
        "database": {
          "vendor": "PostgreSQL",
          "version": "15.0"
        },
        "application-server": null,
        "license-key": null,
        "camunda-integration": [],
        "data-collection-start-date": "Oct 21, 2023, 4:56:53 AM",
        "commands": {},
        "metrics": {
          "root-process-instance-start": {
            "count": 0
          },
          "activity-instance-start": {
            "count": 0
          },
          "executed-decision-elements": {
            "count": 0
          },
          "flow-node-instances": {
            "count": 0
          },
          "executed-decision-instances": {
            "count": 0
          },
          "decision-instances": {
            "count": 0
          },
          "process-instances": {
            "count": 0
          }
        },
        "webapps": [],
        "jdk": {
          "version": "11",
          "vendor": "Oracle Corporation"
        },
        "telemetry-enabled": true
      }
    }
  }
}

expected first request

{
  "body": {
    "installation": "1439ed95-7843-495f-9b5c-77cb44828e38",
    "product": {
      "name": "Camunda BPM Runtime",
      "version": "7.20.0",
      "edition": "community",
      "internals": {
        "database": {
          "vendor": "PostgreSQL",
          "version": "15.0"
        },
        "application-server": null,
        "license-key": null,
        "camunda-integration": [],
        "data-collection-start-date": "Oct 21, 2023, 5:01:53 AM",
        "commands": {
          "TelemetryConfigureCmd": {
            "count": 1
          },
          "MetricsCollectionTask_MetricsCollectionCmd": {
            "count": 1
          },
          "IsTelemetryEnabledCmd": {
            "count": 1
          },
          "GetNextIdBlockCmd": {
            "count": 1
          }
        },
        "metrics": {
          "root-process-instance-start": {
            "count": 0
          },
          "activity-instance-start": {
            "count": 0
          },
          "executed-decision-elements": {
            "count": 0
          },
          "flow-node-instances": {
            "count": 0
          },
          "executed-decision-instances": {
            "count": 1
          },
          "decision-instances": {
            "count": 1
          },
          "process-instances": {
            "count": 0
          }
        },
        "webapps": [],
        "jdk": {
          "version": "11",
          "vendor": "Oracle Corporation"
        },
        "telemetry-enabled": true
      }
    }
  }
}

expected second request

{
  "body": {
    "installation": "1439ed95-7843-495f-9b5c-77cb44828e38",
    "product": {
      "name": "Camunda BPM Runtime",
      "version": "7.20.0",
      "edition": "community",
      "internals": {
        "database": {
          "vendor": "PostgreSQL",
          "version": "15.0"
        },
        "application-server": null,
        "license-key": null,
        "camunda-integration": [],
        "data-collection-start-date": "Oct 21, 2023, 5:01:53 AM",
        "commands": {
          "TelemetryConfigureCmd": {
            "count": 1
          },
          "MetricsCollectionTask_MetricsCollectionCmd": {
            "count": 1
          },
          "IsTelemetryEnabledCmd": {
            "count": 1
          },
          "GetNextIdBlockCmd": {
            "count": 1
          }
        },
        "metrics": {
          "root-process-instance-start": {
            "count": 1
          },
          "activity-instance-start": {
            "count": 0
          },
          "executed-decision-elements": {
            "count": 0
          },
          "flow-node-instances": {
            "count": 0
          },
          "executed-decision-instances": {
            "count": 0
          },
          "decision-instances": {
            "count": 0
          },
          "process-instances": {
            "count": 1
          }
        },
        "webapps": [],
        "jdk": {
          "version": "11",
          "vendor": "Oracle Corporation"
        },
        "telemetry-enabled": true
      }
    }
  }
}

Conclusion:

• The third request is not expected, and it is unclear why it happens.
• Some of the QA test modules don't set the telemetryReporterActivate engine property to false. This should be standard, as all tests should expect telemetry to be switched off. Tests that require telemetry features usually handle the necessary properties themselves.