Webhook not refreshing applications #15427
Comments
|
Hi @AlekseiTikhonov-azur, can you check if the revision |
|
Revision is defined: "source" = {
"directory" = {
"recurse" = true
}
"path" = "projects/shared/dev/aws/eu-central-1/argocd/_init"
"repoURL" = "https://github.com/test-app.git"
"targetRevision" = "webhook-fix"
} |
|
I checked on argocd v2.9.3. The same result. Argocd not refreshing application. Applicationset-controller received github webhook succesfully. |
|
We have been experiencing the same issue with ArgoCD 2.8: ApplicationSet webhook does not update Application resources. The webhook is logged, but It is notable that scaling down applicationset-controller replicas to zero and back to 1 do not have an effect. There clearly is something cached in Redis which is not invalidated by the webhook or not even by the newly created applicationset-controller replica. That said, when the |
|
We were running v2.8.3 - It seems v2.8.8 has a possible fix! We'll try with v2.8.9 soon and see if it fixes the issue! https://github.com/argoproj/argo-cd/releases/tag/v2.8.8 |
|
@pre can you try 2.9.3 version ? I tested it. Webhooks not working. |
|
@AlekseiTikhonov-azur the fixes I linked in my previous message are introduced in v2.9.4, see https://github.com/argoproj/argo-cd/releases/tag/v2.9.4 So v2.9.3 is still broken - note that v2.8.9 is newer than v2.9.3. |
|
It seems the problem is not fully fixed yet in v2.8.9 -- but something has changed! The delay between However, with the next environment it took 9 minutes until So the problem seems to be not fixed yet, but something did change as the deterministic "10 minutes + reconciliation" changed to "something between 3-9 mintues". Here are some redacted from argocd-applicationset-controller: |
|
Here's an even more reduced log of an experiment I just ran Push a commit which changes Every three minutes log gets a message of Until 10 minutes have passed: after 10 minutes, something expires in the Redis cache and the .. nothing happens .. .. 3 minutes later, nothing happens .. .. 3+3=6 minutes later, nothing happens .. .. 3+3+3=9 minutes later, nothing happens .. .. until 3+3+3+3 = 12 minutes later, it finally updates: .. followed with an immediate |
|
I can confirm that ArgoCD v2.10.6 still has the issue: argocd-applicationset-controller webhook does not update the generated applications. However, there is a change!
Also, it's weird that when the update finally happens (after 10 minutes) the same 
|
|
Possible bugfix found in Slack:
|
|
Hi do you know when this bug fix will be merged? |
Somebody should craft a pull request to figure out if the above is a proper fix. @anyone ? |
Possibly relevant pieces of information from |
|
ApplicationSet not refreshed after webhook event if |
|
I don’t think new tickets help, we’d need someone to look at the issue on the code level. |
|
Realised I needed to send webhook requests to the argocd-server and not the argocd applicationset-controller. When I correctly sent webhook events to argocd-server the applications refresh as expected upon receiving the event. The reason why I was initially getting a 404 was because my argocd server was configured with a rootpath of argocd, so the correct path for me was Basically I ran into skill issues :s |
|
You’re now talking about the application webook which does work. The applicationset-webhook needs to be routed to applicationset-controller (instead of argocd-server), but the applicationset webhook does not refresh applications even though the webhook call is received and showing up in logs of applicationset-controller. Ie. There are two separate webhooks. |
|
Similar error in 2.11.2 ,git webhook didn't refresh my application.By the way, i use mutileplate source repo(helm and git ) and custom namespace.I didn't test git web hook for namespace argocd,so confused |
You should see the webhook being received in the applicationset-controller logs. Then you have configured it correctly. However, the target Application resources behind that ApplicationSet resource are still being cached by the git generator, and as a result, the applicationset-controller does not refresh any resources upon receiving the applicationset webhook. |
I just tested the application not applicationset,when i configured git webhook firstly,encountered the error.I got the log in argocd-server,but application didn't refresh
|
|
I have a similar problem, but I am using Azure DevOps. From Azure DevOps, it says "Success" and I can see that ArgoCD responds with HTTP 200. From the logs point of view, the argocd-server received the event, but nothing happens: ArgoCD Version: v2.11.5 |
The argocd-server handles the Application webhook, which doesn’t have an issue at least in ArgoCD 2.10. This issue is about the ApplicationSet webhook which needs to be routed to the applicationset-controller. The ApplicationSet webhook does not currently refresh any git generator based Application resources. |
|
I also encountered a similar problem. After git repository was updated, Argocd was triggered for deployment through webhook, but Argocd occasionally did not respond, and it was necessary to REFRESH manually on the web page to display the configuration difference OutOfSync |
Refreshing does not help in this issue with applicationset-webhook. Even with a hard refresh, the cached resources are still being used by the statefulset/application-controller because the deployment/applicationset-controller did not invalidate the existing cache. As a result, only waiting for 10 minutes +/- 3 minutes will eventually cause the git generator cache be regenerated and only after that the updated Application manifests will mark the existing Application resources OutOfSync. This can be demonstrated easily by updating the targetRevision via ApplicationSet: even after Hard Refresh the previous targetRevision remains visible. Only after the git generator cache has expired, the new targetRevision will be updated. |
|
any solution that can work? |
|
No fix available, but you can manually remove the affected cache keys from Redis to trigger the update. |
|
I have same issue with v2.10.7+b060053 |
|
Well in order to make webhook to work on my github repo i just change the ingress of the webhook to navigate repo-server ingress abd it works just fine for me |
|
@AsafZoki you’re probably talking about the application webhook. That works. The applicationset webhook is the one broken. |
|
@pre could you please correct/confirm my understanding of what is happening? You have some application , |
|
@rumstead We have a repository ”release-control”.
Another repository ”continuous-delivery” contains the actual manifests of the application. The tag v1.2.3 exists in this repository. We have various ApplicationSet resources which generate the Application resources with git generator, using *.yaml as the source from the ”release-control” repository. The targetRevision of Application is defined by ApplicationSet. We can see applicationset-controller receives the webhook when ”release-control” is pushed with a new commit with an updated value for targetRevision. Here we expect applicationset-controller to regenerate the Application resource(s) using this new value. However, unless the Redis cache is emptied, it will take 10min +/- 3 minutes until the new attributes are used. Even restarting applicationset-controller uses the previous values in ”release-control” until the Redis cache has expired. This issue is not only about the applicationset-webhook. The issue is nothing except waiting for 10min +/- 3 minutes (or emptying Redis cache manually) will generate the Application resources with updated attributes from ”release-control”. TargetRevision is just an example. I have tested with any values read from *.yaml by the appset and they are all cached. I’m happy to provide more information, this issue is consistent and we want to help it get fixed. |
|
Got it, I use a similar workflow without the webhook. In a very basic setup, I cannot reproduce this. I see the revision cache returning the new commit and not using any cached files but grabbing the new ones from git and updating my application immediately. Here is a snippet from the repo server logs, I added a log line when there is a cache miss. {"level":"debug","msg":"cache hit for repo: https://github.com/rumstead/argocd-playground.git revision: cc3b9469e89dbd9831612f67b1c55450c117580b pattern: manifests/applicationsets/git-file/config/config.yaml","time":"2024-08-02T16:26:40-04:00"}
{"level":"info","msg":"cache miss for repo: https://github.com/rumstead/argocd-playground.git revision: fe8ac1e353f75464bf161f55e991c8fec1472348 pattern: manifests/applicationsets/git-file/config/config.yaml","time":"2024-08-02T16:28:28-04:00"}Appset Logs {"level":"info","msg":"Received push event repo: https://github.com/rumstead/argocd-playground.git, revision: main, touchedHead: true","time":"2024-08-02T16:28:28-04:00"}
{"level":"info","msg":"refresh ApplicationSet argocd/git-file from webhook","time":"2024-08-02T16:28:28-04:00"}
{"applicationset":{"Namespace":"argocd","Name":"git-file"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-02T16:28:30-04:00"}
{"applicationset":{"Namespace":"argocd","Name":"git-file"},"generator":{"git":{"repoURL":"https://github.com/rumstead/argocd-playground.git","files":[{"path":"manifests/applicationsets/git-file/config/config.yaml"}],"revision":"HEAD","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 1 applications","time":"2024-08-02T16:28:30-04:00"}
Sanity checks I was also slow to switch over to my terminal as I was looking at the logs. The reason we see so many "generated applications" is because the applicationset gets updated 2 times during a webhook event. each update causes the applicationset to be requeued
EDIT: I am also now realizing the repo-server log don't help much because we overload the revision variable. |
|
If you look at the ApplicationSet controller logs and search for “end reconcile” for a single ApplicationSet, how long is between the two log messages? |
|
@rumstead There are some earlier timestemps above: For the context, I'm using the argocd-ha manifests which come with the Redis cache with three instances. Here's a fresh example run with ArgoCD 2.10.11.
The key word to watch from the applicationset-controller logs is In total the update took 9 minutes and 12 seconds, which fits the window of 10 minutes +/- 3 minutes. The first {"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 7 applications","time":"2024-08-16T06:09:34Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:09:34Z"}
{"level":"info","msg":"Received push event repo: https://github.enterprise.com/OurOrg/ourorg-release-control, revision: env/dev, touchedHead: false","time":"2024-08-16T06:09:40Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:09:41Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:09:42Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 9 applications","time":"2024-08-16T06:09:50Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:09:51Z"}
Reconciling for 10 minutes, but the Application is not updated: {"level":"info","msg":"Alloc=26607 TotalAlloc=13812576 Sys=58661 NumGC=845 Goroutines=92","time":"2024-08-16T06:10:05Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/cr-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 13 applications","time":"2024-08-16T06:10:30Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:10:30Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:10:42Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:10:42Z"}
{"applicationset":{"Namespace":"argocd","Name":"pullrequest-frontend"},"generator":{"pullRequest":{"github":{"owner":"OurOrg","repo":"ourorg-frontend","api":"https://github.enterprise.com/","tokenRef":{"secretName":"git-credentials-ourorg-corp","key":"password"},"labels":["deploy-to-dev"]},"requeueAfterSeconds":300,"template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 15 applications","time":"2024-08-16T06:10:51Z"}
{"applicationset":{"Namespace":"argocd","Name":"pullrequest-frontend"},"level":"info","msg":"end reconcile","requeueAfter":300000000000,"time":"2024-08-16T06:10:52Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/cr-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 13 applications","time":"2024-08-16T06:11:14Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:11:14Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:11:22Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:11:23Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 1 applications","time":"2024-08-16T06:11:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:11:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 2 applications","time":"2024-08-16T06:11:56Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:11:57Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-integration/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 6 applications","time":"2024-08-16T06:12:25Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:12:25Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 7 applications","time":"2024-08-16T06:12:34Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:12:34Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:12:42Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:12:43Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 9 applications","time":"2024-08-16T06:12:51Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:12:51Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:13:12Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:13:13Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/cr-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 13 applications","time":"2024-08-16T06:13:30Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:13:31Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:13:42Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:13:43Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:14:12Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:14:13Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:14:23Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:14:24Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 1 applications","time":"2024-08-16T06:14:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:14:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 2 applications","time":"2024-08-16T06:14:57Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:14:57Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-integration/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 6 applications","time":"2024-08-16T06:15:25Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:15:25Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 7 applications","time":"2024-08-16T06:15:34Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:15:34Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:15:43Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:15:43Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 9 applications","time":"2024-08-16T06:15:51Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:15:51Z"}
{"applicationset":{"Namespace":"argocd","Name":"pullrequest-frontend"},"generator":{"pullRequest":{"github":{"owner":"OurOrg","repo":"ourorg-frontend","api":"https://github.enterprise.com/","tokenRef":{"secretName":"git-credentials-ourorg-corp","key":"password"},"labels":["deploy-to-dev"]},"requeueAfterSeconds":300,"template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 15 applications","time":"2024-08-16T06:15:52Z"}
{"applicationset":{"Namespace":"argocd","Name":"pullrequest-frontend"},"level":"info","msg":"end reconcile","requeueAfter":300000000000,"time":"2024-08-16T06:15:52Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/cr-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 13 applications","time":"2024-08-16T06:16:31Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-cr-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:16:31Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:17:13Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:17:14Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 19 applications","time":"2024-08-16T06:17:24Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:17:24Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 1 applications","time":"2024-08-16T06:17:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-bs-cluster"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:17:53Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"app-runtime/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 2 applications","time":"2024-08-16T06:17:57Z"}
{"applicationset":{"Namespace":"argocd","Name":"app-runtime-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:17:57Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-integration/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 6 applications","time":"2024-08-16T06:18:26Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-integration"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:18:26Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/bs-cluster/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 7 applications","time":"2024-08-16T06:18:35Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-bs-cluster-1"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:18:35Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"generator":{"git":{"repoURL":"https://github.enterprise.com/OurOrg/ourorg-release-control","files":[{"path":"cloud-release/ag-service/*.yaml"}],"revision":"env/dev","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 9 applications","time":"2024-08-16T06:18:51Z"}
After 10 minutes the Application got updated, notice the keyword {"app":"argocd/quota-calculator","applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"level":"info","msg":"updated Application","time":"2024-08-16T06:18:52Z"}
{"applicationset":{"Namespace":"argocd","Name":"cloud-release-ag-service"},"level":"info","msg":"end reconcile","requeueAfter":180000000000,"time":"2024-08-16T06:18:52Z"}First mention of the updated Application (quota-calculator) in the application-controller logs is only after applicationset-controller shows {"application":"argocd/quota-calculator","level":"info","msg":"Refreshing app status (spec.source differs), level (3)","time":"2024-08-16T06:18:52Z"}
{"application":"argocd/quota-calculator","level":"info","msg":"Comparing app state (cluster: https://kubernetes.default.svc, namespace: quota-calculator)","time":"2024-08-16T06:18:52Z"}Same with argocd-repo-server: first mention is only after {"level":"info","msg":"manifest cache miss: \u0026ApplicationSource{RepoURL:[...]:"2024-08-16T06:18:52Z"}
|
|
Could someone please rename this issue title to This issue is about the applicationset-controller. The normal Application webhook works! There were some fix attempts in v2.8.8 but they didn't work. |
|
@pre can you share your appset? |
|
@rumstead certainly I can! Here's an example where I changed OurOrg and acme as names, but otherwise these are real manifests:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: cloud-release-ag-service
namespace: argocd
spec:
generators:
- git:
files:
- path: cloud-release/ag-service/*.yaml
repoURL: https://github.enterprise.com/OurOrg/acme-release-control
revision: env/dev
template:
metadata:
name: '{{acmeService.name}}'
spec:
destination:
namespace: '{{acmeService.namespace}}'
server: https://kubernetes.default.svc
project: ag-service
source:
path: design-time/ag-service/{{acmeService.name}}/dev
plugin:
env:
- name: AG_APP_NAMESPACE
value: '{{acmeService.namespace}}'
- name: AG_ENVSUBST_SOURCE
value: design-time/common/ag-envsubst/dev
- name: AG_TARGET_REVISION
value: '{{release.targetRevision}}'
name: kustomize-agenvsubst-vault
repoURL: https://github.enterprise.com/OurOrg/acme-continuous-delivery
targetRevision: '{{release.targetRevision}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
# acme-release-control/cloud-release/ag-service/quota-calculator.yaml
acmeService:
name: quota-calculator
namespace: quota-calculator
release:
targetRevision: v3.7.0 |
|
For the record, the issue is still in ArgoCD v2.12.2 |
|
I am still unable to reproduce. I changed the What does the workqueue look like for applicationset controller? Do you see it going to 0? It is exposed via the |
|
Thank you for testing, @rumstead! Did you try updating an existing Application, which had already been created? Creation of an Application is always instant, it's the update which takes between 6-13 minutes. I also tested by creating a public test in https://github.com/pre/public-argocd-test-release-control/blob/main/release/example-application.yaml Using this ApplicationSet apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: public-test-1
namespace: argocd
spec:
generators:
- git:
files:
- path: release/*.yaml
repoURL: https://github.com/pre/public-argocd-test-release-control
revision: main
template:
metadata:
name: example-{{acmeService.name}}
spec:
destination:
namespace: '{{acmeService.namespace}}'
server: https://kubernetes.default.svc
project: ag-service
source:
path: design-time/ag-service/{{acmeService.name}}/int
plugin:
env:
- name: AG_APP_NAMESPACE
value: '{{acmeService.namespace}}'
- name: AG_ENVSUBST_SOURCE
value: design-time/common/ag-envsubst/dev
- name: AG_TARGET_REVISION
value: '{{release.targetRevision}}'
- name: EXAMPLE_RELEASE_CONTROL_VALUE
value: '{{release.acmeValue}}'
name: kustomize-agenvsubst-vault
repoURL: https://github.enterprise.com/acme/acme-continuous-delivery
targetRevision: '{{release.targetRevision}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
In these two cases the webhook works.
The workqueue metric Timestamps
GrafanaWe have the default ArgoCD Grafana dashboard (it doesn't seem to have ApplicationSet metrics), here is a snapshot from the test bench during the time of my test:
|
|
Scrap that above - my test repository's github.com webhook was not correct. I'll try again. |
|
All right - I fixed my webhook in the github.com test example repository, and it works: The message So .. there is some difference in the webhook payload between github.com and Github Enterprise (currently I ran out of time today - next I will set up a test bench that let's me compare the webhook's payload between github.com and Github Enterprise. |
|
Awesome! I know it doesn’t fix the issue but at least gives us a hint. |
|
I did some further digging, and now I can reproduce the problem also with the Summary: Webhook does not work with git tags or branches which contain the character apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: public-test-5
namespace: argocd
spec:
generators:
- git:
files:
- path: release/*.yaml
repoURL: https://github.com/pre/public-argocd-test-release-control
revision: env/hello # <-- does not work with the slash, but "main" and "dev" do work
[..]I inspected the webhook payload between Github Enterprise and Github.com - the payloads are equivalent, only difference are the My reproduction in github.com where the ApplicationSet webhook does not work:
Also notable: The git repository cache is not updated for
-> ApplicationSet B will fail with After waiting for ~10 minutes, ApplicationSet B will generate the Application and see that If changes are pushed to I guess the tags break because internally eg. Test scriptUsing my example ApplicationSet above, I tracked a value in source.plugin.env which was defined in https://github.com/pre/public-argocd-test-release-control. Polling the value of while true; do
kubectl get app -n argocd example-envhello-acme-calculator -o jsonpath='{.spec.source.plugin.env[3]}'
date
sleep 1
done
|
|
This is extremely helpful, I found the issue. GitHub sends the full branch ref eg, I will submit a PR |
|
/assign rumstead |
|
The alternative to my PR is to use the fully qualified refs git generator. Eg, |
|
Thanks, @rumstead! I tested prepending the git revision with |
…19783) * fix(appset): allow for shorthand git refs in git generators Signed-off-by: rumstead <[email protected]> * Retrigger CI pipeline Signed-off-by: rumstead <[email protected]> * attempt to fix goimports Signed-off-by: rumstead <[email protected]> * attempt to fix goimports Signed-off-by: rumstead <[email protected]> * remove redundant test Signed-off-by: rumstead <[email protected]> --------- Signed-off-by: rumstead <[email protected]>
Checklist:
argocd version.Describe the bug
github webhook seems not working. It does not refreshing argo-cd applications but dropping git repository cache.
To Reproduce
Add a hook just like in the documentation.
helm values
ingressClassName: ingress - internal ingress controller
ingressClassName: public-nginx - public ingress controller
Expected behavior
Argocd refresh applications and not waiting before it updated by refresh interval.
Version
Logs applicationset-controller created by applicationset api
Logs applicationset-controller created by application api
The text was updated successfully, but these errors were encountered: