Eventhub Scaler: add new dapr checkpoint strategy to eventhub scaler #3732
Conversation
|
Oh you are right. I didn’t see that parallel activity. |
|
Yes, currently, I test the e2e test on this base, but I could cherry-pick the e2e over so that we can close this PR |
|
What works for you! |
|
@tomkerkhove I'm ready with the e2e test, but before the new test is green, the related image from the PR kedacore/test-tools#99 should be merged. I've tested it locally with a custom image on my registry. I've added the e2e test to my PR because I have no access rights to the other fork. I hope it is ok. |
christle
force-pushed
the
master
branch
2 times, most recently
from
792c639
to
f4dc0ca
Compare
|
/run-e2e |
|
it seems that the required package "ghcr.io/kedacore/tests-azure-eventhub-dapr" isn't pushed |
|
I'll check, I guess that it's private as default :( |
|
/run-e2e event* |
|
It was private, I have already changed it+ |
|
It seems like scaling in doesn't work, maybe the timeout is too small or the worker doesn't consume the messages correctly |
|
/run-e2e event* |
|
On my private setup, based on the "test-infrastructure" repo, I can run all event* tests without problems. The "scale in" needs around 10+ seconds. Currently, I don't see the difference. I see inside the pipeline run a lot of errors around missing eventhub infrastructure and some BlobNotFound exceptions regarding the storageAccount. The BlobNotFound could be the reason for the missing "scale in" because there is no checkpoint. But I don't get this on my private azure, and I have no further instruments for the underlying pipeline infrastructure or the container logs. Could it be that are some parallel e2e tests that cause the deletion of the eventhubs and Storage Blobs? |
We use the random names to avoid conflicts between tests, so it shouldn't happen. Let me check the logs |
|
ok thank you, I will check another possible reason regarding different dapr versions |
|
I'm checking in depth the e2e test, and you aren't installing dapr, so basically the checkpoint probably isn't created, and the messages aren't consumed. The e2e cluster doesn't have any infra as default, all needed infrastructure should be installed/removed during the tests. I can't see either any dapr manifest in your e2e test. |
|
you have examples about how you can install dapr "globally" here, where we install Workload Identity and AWS Role Assumptions, and you also have other samples about "test scoped" installation in other scalers e2e tests like Datadog or NewRelic, where we install their against in the scope of the test, and we remove them at the end |
|
i deploy dapr as single edge container within the deployment template |
|
I have created a cluster with the kedacore/test-infrastructure repo, and there it runs as expected, but I will do some further checks. I think as a second container within the deployment, it is very easy to set up and to clean up |
true, I didn't see it, sorry |
|
/run-e2e azure_event_hub_dapr_test* |
|
I've found the issue. I have an older dapr image. I put "ImagePullAllways" inside the template and now I could reproduce the pipeline issue. Ok, i will check that. |
|
let me trigger them again and check the container logs |
|
wow, nice catch! 👏 |
|
Could you create another PR to docs repo adding this new checkpoint? |
|
Yes, I will do it |
|
btw, I like the whole test setup very much, it was very easy to set up a test environment with the test-tools and test-infrastructure repo. |
|
I've created a pull request for the docs repository kedacore/keda-docs#996 |
|
Thank you! |
|
/run-e2e azure_event_hub_dapr_test* Update: You can check the progress here |
|
@christle Thanks a ton! Should we abandon the other PR then? |
|
If it is ok for you, then yes. It was easier for me to test & develop this PR. |
Signed-off-by: Christian Leinweber <[email protected]>
Signed-off-by: Christian Leinweber <[email protected]>
Signed-off-by: Christian Leinweber <[email protected]>
|
/run-e2e azure_event_hub_dapr_test* |
Signed-off-by: Christian Leinweber <[email protected]>
Signed-off-by: Tom Kerkhove <[email protected]>
|
/run-e2e azure_event_hub_dapr_test* |
…edacore#3732) Co-authored-by: Tom Kerkhove <[email protected]> Co-authored-by: Zbynek Roubalik <[email protected]>
Hello,
Dapr uses a prefix for the Checkpoint blob "dapr-hubName-consumerGroup-". So here is another Checkpoint Strategy to support that case.
I will open another PR to update the docs.
Relates to #3022