Application webhook not refreshing application

[BFreitas16]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

A push code to master branch in an Azure DevOps repository is not triggering the automatic Application refresh.

Azure DevOps Repository

For the sake of this, let's take as an example an ElasticSearch application (omitting non relevant applications).

| apps
    \ elasticsearch.yaml
\ elasticsearch
    \ overlays
        | prod
        \ staging
            | kustomization
            \ myelasticsearch.yaml

Description:

• apps - Helm Chart folder containing all ArgoCD Applications and ApplicationSets templates.
• apps/elasticsearch.yaml - ArgoCD Application ElasticSearch with repoURL poiting to this same repo with path as elasticsearch/overlays/staging
• elasticsearch - Folder containing all ElasticSearch Applications
• elasticsearch/overlays - Folder containing the different environments
• elasticsearch/overlays/staging - Folder containing the ElasticSearches from Staging environment
• elasticsearch/overlays/staging/myelasticsearch.yaml - Configuration of K8s ElasticSearch

To Reproduce
Add a webhook as described in documentation.

Create an Ingress to expose the "argocd-server" Service Webhook externally:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
spec:
  rules:
  - host: webhook.argocd.url
    http:
      paths:
      - path: /api/webhook
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 80

Commit a push to the Azure DevOps Repository changing the elasticsearch/overlays/staging/myelasticsearch.yaml configuration.

Expected behavior

Argocd refresh applications and not waiting before it updated by refresh interval.

Screenshots

Version

argocd version
argocd-server: v2.11.5+c4b283c
  BuildDate: 2024-07-15T17:39:54Z
  GitCommit: c4b283ce0c092aeda00c78ae7b3b2d3b28e7feec
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.4+g81c902a
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs

As expected, the Azure DevOps triggers the HTTP request and receives an HTTP 200 response. Analysing the argocd-server logs we can see that it received the Push request with the right repository:

time="2024-09-26T09:57:12Z" level=info msg="Received push event repo: https://my.git.repo.com/_git/az-repository, revision: master, touchedHead: true"

Logs right after the "Received push" log:

time="2024-09-26T09:57:33Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"at_hash\":\"ABC\",\"aud\":\"XYZ\",\"c_hash\":\"123\",\"email\":\"my_email\",\"email_verified\":true,\"exp\":123,\"iat\":123,\"iss\":\"my_oidc_url",\"name\":\"my_email\",\"namespace\":\"root\",\"preferred_username\":\"my_email\",\"sub\":\"123\",\"username\":\"my_email\"}" grpc.request.content="name:\"elasticsearch\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2024-09-26T09:57:33Z" span.kind=server system=grpc
time="2024-09-26T09:57:33Z" level=info msg="received streaming call /application.ApplicationService/WatchResourceTree" grpc.method=WatchResourceTree grpc.request.claims="{\"at_hash\":\"ABC\",\"aud\":\"XYZ\",\"c_hash\":\"123\",\"email\":\"my_email\",\"email_verified\":true,\"exp\":123,\"iat\":123,\"iss\":\"my_oidc_url\",\"name\":\"my_email\",\"namespace\":\"root\",\"preferred_username\":\"my_email\",\"sub\":\"123\",\"username\":\"my_email\"}" grpc.request.content="applicationName:\"elasticsearch\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2024-09-26T09:57:33Z" span.kind=server system=grpc
time="2024-09-26T09:57:39Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=WatchResourceTree grpc.service=application.ApplicationService grpc.start_time="2024-09-26T09:57:33Z" grpc.time_ms=5820.791 span.kind=server system=grpc
time="2024-09-26T09:57:39Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2024-09-26T09:57:33Z" grpc.time_ms=5821.349 span.kind=server system=grpc
time="2024-09-26T09:57:52Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-secret"
time="2024-09-26T09:57:52Z" level=info msg="invalidated cache for resource in namespace: argocd with the name: argocd-notifications-cm"
time="2024-09-26T09:58:32Z" level=info msg="Alloc=29964 TotalAlloc=89473575 Sys=92024 NumGC=18033 Goroutines=132"

No other relevant logs.

[BFreitas16]

I have followed the following bug Webhook not refreshing applications, but that one is related to the ApplicationSet Webhook - fix in a further release.

We currently use both Application and ApplicationSets and both Webhooks do not work as expected (meaning that they do not refresh the respective application).

[dhruvang1]

Could you look at #20039 to see if you are having the same issue?

[BFreitas16]

Could you look at #20039 to see if you are having the same issue?

Well, it is similar. Thank you so much.

The problem is using SSH because the URL that Azure DevOps gives to clone is different from the one given in the Azure DevOps Webhook POST data. Just to record it here for future reference, the URL returned from the Azure DevOps Webhook POST data is:

...
      "remoteUrl": "https://<company>.visualstudio.com/<project>/DefaultCollection/_git/<repository>"
...

The clone using SSH gives the following URL:

<user>@vs-ssh.visualstudio.com:v3/<company>/<project>/<repository>

This will make the regex to fail and it gives the following logs:

time="2024-09-27T10:45:58Z" level=debug msg="<user>@vs-ssh.visualstudio.com:v3/<company>/<project>/<repository> does not match (?i)^(http://|https://|[a-zA-Z0-9_\\.][a-zA-Z0-9_\\.-]{0,30}[a-zA-Z0-9_\\.\\$-]?@|ssh://([a-zA-Z0-9_\\.][a-zA-Z0-9_\\.-]{0,30}[a-zA-Z0-9_\\.\\$-]?@)?)<company>\\.visualstudio\\.com(:[0-9]+|)[:/]<project>/_git/<repository>(\\.git)?$"

Here is the Regex used:

I will close this issue and continue the tracking in the following one, since it is the same: #16854