Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

Closed
miaooss opened this issue May 10, 2022 · 12 comments
Closed

Upgrading from KEDA 2.5.0 to 2.7.0 with external scaler #3032

miaooss opened this issue May 10, 2022 · 12 comments
Labels
bug Something isn't working

Comments

@miaooss
Copy link

miaooss commented May 10, 2022 •
edited by tomkerkhove
Loading

Report

We are using au external scaler
Keda seems to be connecting to the service correctly.

But some expected metrics or metadata are missing

Here is our KEDA configuration:

  scalingStrategy:
    strategy: "accurate"                        # Optional. Default: default. Which Scaling Strategy to use.
  triggers:
    - type: external
      metadata:
        scalerAddress: servicename.<env>:3009
        service: servicename

This is the C# code we are using


    public async override Task<GetMetricSpecResponse> GetMetricSpec(ScaledObjectRef request, ServerCallContext context)
    {
        var resp = new GetMetricSpecResponse();

        if (request.ScalerMetadata.TryGetValue("service", out string serviceValue) && serviceValue == "servicename")
        {
            resp.MetricSpecs.Add(new MetricSpec
            {
                MetricName = "targetAverageValue",
                TargetSize = 1
            });
        }

        return await Task.FromResult(resp);
    }
    
    
    public override async Task<GetMetricsResponse> GetMetrics(GetMetricsRequest request, ServerCallContext context)
    {
        var resp = new GetMetricsResponse();

        if (request.ScalerMetadata.TryGetValue("service", out string serviceValue) && serviceValue == "servicename")
        {
            if (request.MetricName == "queueLength")
            {
                long pendingJobsCount = await GetPendingJobsCount();
                resp.MetricValues.Add(new MetricValue
                {
                    MetricName = "queueLength",
                    MetricValue_ = pendingJobsCount
                });
            }
        }

        return resp;
    }

Expected Behavior

Expect KEDA to scale up the service

Actual Behavior

We do return no actual job.

Steps to Reproduce the Problem

  1. We update KEDA version to use 2.7.0
  2. We can tell that the old version is working fine 2.5.0

Logs from KEDA operator

1.6522127784315143e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "servicename", "scaledJob.Namespace": "dev", "Number of running Jobs": 0}

KEDA Version

2.7.0

Kubernetes Version

1.23

Platform

Microsoft Azure

Scaler Details

External

Anything else?

No response
@miaooss miaooss added the bug Something isn't working label May 10, 2022
@JorTurFer
Copy link
Member

hey @miaooss
sorry for the slow response :(
We had problems related with the name generation in v2.6.0 but they have been solved in v2.7.0.
Is there any error in the external metric server?
In the log that you shared seems all normal, could you share more lines please?

@miaooss
Copy link
Author

miaooss commented May 13, 2022 •
edited
Loading

Hi,

Thank you for the assistance.

Here is the logs for the keda-operator-metrics-apiserver

I0505 18:50:52.872453       1 main.go:147] keda_metrics_adapter "msg"="KEDA Version: 2.7.0"
I0505 18:50:52.872505       1 main.go:148] keda_metrics_adapter "msg"="KEDA Commit: 5f8af5cd90805d8b9c5964e1619a8b2bdcec6cfd"
I0505 18:50:52.872512       1 main.go:149] keda_metrics_adapter "msg"="Go Version: go1.17.3"
I0505 18:50:52.872518       1 main.go:150] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I0505 18:50:55.024359       1 request.go:665] Waited for 1.047381127s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/security.istio.io/v1beta1?timeout=32s
I0505 18:50:56.379197       1 logr.go:261] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen"  "addr"=":8080"
I0505 18:50:56.379673       1 provider.go:65] keda_metrics_adapter/provider "msg"="starting"
I0505 18:50:56.379697       1 main.go:214] keda_metrics_adapter "msg"="starting adapter..."
2022/05/05 18:50:56 Starting metrics server at :9022
I0505 18:50:56.380078       1 internal.go:362] keda_metrics_adapter "msg"="Starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0505 18:50:56.380194       1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0505 18:50:56.380211       1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0505 18:50:56.380221       1 controller.go:186] keda_metrics_adapter/controller/scaledobject "msg"="Starting Controller" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject"
I0505 18:50:56.481278       1 controller.go:220] keda_metrics_adapter/controller/scaledobject "msg"="Starting workers" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "worker count"=1
I0505 18:50:56.885629       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0505 18:50:57.606025       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0505 18:50:57.606114       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0505 18:50:57.606148       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0505 18:50:57.606156       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0505 18:50:57.606164       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0505 18:50:57.606166       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0505 18:50:57.606394       1 secure_serving.go:266] Serving securely on [::]:6443
I0505 18:50:57.606475       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0505 18:50:57.606769       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0505 18:50:57.706800       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0505 18:50:57.706845       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0505 18:50:57.706978       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file

here is more log for the keda-operator

1.6524569446550982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569446551404e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569446554346e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652456944655462e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456958601352e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569586013892e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569595817766e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652456959581809e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569596552165e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569596552806e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569596554308e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569596554503e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569736168468e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569736168823e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569745867527e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569745868163e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456974658725e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569746587565e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569746588607e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569746588833e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569886073647e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569886074188e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456989584188e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569895844314e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524569896585844e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569896586394e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652456989658696e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524569896587121e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457003604612e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570036046746e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570045903165e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570045903482e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570046629522e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570046629887e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570046630619e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457004663086e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457018608771e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457018608892e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570195849993e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570195858319e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570196627512e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570196627936e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570196629586e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570196629784e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570335978518e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570335978868e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457034582769e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570345828042e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570346581707e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570346582224e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570346662562e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570346662822e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457048595017e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570485950663e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457049587505e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570495875382e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570496693807e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570496694207e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570496695542e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570496700332e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245706359464e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570635946712e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064585202e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457064585233e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064656624e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570646566572e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457064659847e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570646598754e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570786188848e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570786189187e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570795922773e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245707959231e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457079659782e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570796598115e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570796600237e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570796600382e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570936210456e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570936210818e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570945882676e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570945882998e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570946588528e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457094658886e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524570946590633e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524570946590824e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457108618759e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457108618796e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571095873528e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571095873878e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457109631762e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571096317909e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571096811817e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571096812131e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245712359978e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571235998347e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457124635557e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246355932e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571246629527e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246629891e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571246710186e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571246710455e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571386051843e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571386052206e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571395921571e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457139592191e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457139661193e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571396612282e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457139661309e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571396613333e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571536284606e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571536285079e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571545890434e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571545890923e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457154661795e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571546618304e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571546619413e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457154661958e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571686101148e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571686101503e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457169594603e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457169594639e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571696704307e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457169670483e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457169670598e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571696706245e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571836137216e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571836137826e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571845879164e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571845879683e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571846665828e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457184666625e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457184666774e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571846667974e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571986211176e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571986211545e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571995916958e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571995917401e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571996617348e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571996617699e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524571996751537e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524571996751778e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457213617805e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572136178405e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572145926697e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572145927079e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457214664278e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572146643088e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572146645582e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572146645832e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457228600856e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572286008997e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572295974908e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572295975227e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457229668759e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572296687949e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572296689763e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572296689982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572436069596e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572436070082e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457244596044e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572445960753e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572446663008e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457244666337e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572446665025e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457244666519e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572586117945e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245725861183e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572595968733e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572595969095e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457259672644e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572596726744e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572596727903e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457259672809e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.65245727361469e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572736147256e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572745920768e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572745921156e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457274664208e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457274664243e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572746644068e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572746644235e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572886217391e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572886217892e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457289598154e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572895981982e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457289655974e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457289656167e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524572896771553e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524572896771848e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573036131463e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573036131837e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457304597814e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573045978465e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573046702592e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573046702964e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573046704085e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573046704376e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573186286345e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457318628688e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457319596347e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573195963778e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573196797385e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573196797764e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573196800365e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573196800718e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457333612797e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573336128378e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457334596943e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457334596976e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573346715412e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573346715832e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457334671983e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573346720085e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457348621807e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573486218617e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573495978022e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573495978377e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573496727686e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573496728072e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573496730149e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457349673032e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573636113615e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.652457363611409e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service4", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573645989597e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573645989912e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service3", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.6524573646701136e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.65245736467015e+09    INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service2", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}
1.652457364670288e+09   INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of running Jobs": 0}
1.6524573646703088e+09  INFO    scaleexecutor   Scaling Jobs    {"scaledJob.Name": "service1", "scaledJob.Namespace": "env1", "Number of pending Jobs ": 0}

@miaooss
Copy link
Author

miaooss commented May 13, 2022

Actually all the logs seems normal

In our code, we have a condition if Keda does not request the correct metric we send nothing.

It seems like Keda does not send the expected metric.

Thank you,
Jeremy.

@JorTurFer
Copy link
Member

I have an idea about why this can be happening, I'm checking it, I'm checking it locally

@JorTurFer JorTurFer mentioned this issue May 13, 2022
Merged
3 tasks
@JorTurFer
Copy link
Member

I have found the problem and unfortunately there is not any workaround.
Basically, it's due to an internal changes introduced in v2.6 to generate unique metric names and how we use the metric names in ScaledJobs. I have opened a PR with a possible fix but next week is KubeCon so we'll be quite offline and we have to discuss the implementation I have done.
Sorry for the inconvenience 😞

@JorTurFer
Copy link
Member

hey @miaooss ,
I have just merged a PR I expect solves your problem, if you can test it and give feedback about if it's solved or not, we can go deeper if it's not solved (just for not waiting till next release)

@miaooss
Copy link
Author

miaooss commented May 26, 2022

@JorTurFer
Thank you, I will give it a try
Can you let me know what version the fix is applied to?

@JorTurFer
Copy link
Member

Can you let me know what version the fix is applied to?

it's in main

@miaooss
Copy link
Author

miaooss commented Jul 12, 2022 •
edited
Loading

It seems like the issue is still present on the 2.7.1.

We are using a ScaledJob with an external trigger

Return metrics for GetMetricSpec

resp.MetricSpecs.Add(new MetricSpec { MetricName = "testmetrics", TargetSize = 1 });

Return metrics for GetMetrics

resp.MetricValues.Add(new MetricValue { MetricName = "testmetrics", MetricValue_ = pendingJobsCount });

Logs of the service
We can see that the GetMetrics endpoint is called

info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.2699ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 7.2223ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.3122ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 0.6693ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.3065ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2627ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2627ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 45.4623ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.6638ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.2603ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.2903ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 6.1950ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/1.1 GET http://10.240.0.28:5000/servicename/health - - - 200 - application/json 0.3217ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. info: ServiceName.Api.Grpc.ExternalScalerService[0] GetMetricSpec info: ServiceName.Api.Grpc.ExternalScalerService[0] add metrics dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetricSpec' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/GetMetricSpec application/grpc - - 200 - application/grpc 1.0164ms info: Microsoft.AspNetCore.Hosting.Diagnostics[1] Request starting HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' dbug: Grpc.AspNetCore.Server.ServerCallHandler[10] Reading message. dbug: Grpc.AspNetCore.Server.ServerCallHandler[15] Sending message. info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1] Executed endpoint 'gRPC - /externalscaler.ExternalScaler/IsActive' info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished HTTP/2 POST http://servicename.dev:3009/externalscaler.ExternalScaler/IsActive application/grpc - - 200 - application/grpc 8.1591ms

@JorTurFer
Copy link
Member

Hey,
Yes I know it, v2.7.1 was released on May 10 and the fix was merged on May 26. That's why I said to give a try using main tag because next version will be in August. If you still can give a try using main tag to ensure that v2.8 fixes it, would be nice. If not, don't worry, next version will be released in a month

@miaooss
Copy link
Author

miaooss commented Jul 12, 2022

Thank you for the clarification.

@JorTurFer
Copy link
Member

No worries, happy to help :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@miaooss @JorTurFer