Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix disappearing metrics-probe() counters #243

Merged
merged 11 commits into from
Aug 13, 2024

Conversation

MrAnno
Copy link
Member

@MrAnno MrAnno commented Aug 8, 2024

The TLS cache of metrics-probe() has been extended with a global cache in order not to make metrics orphaned whenever an ivykis worker stops (after 10 seconds of inactivity).

The global cache requires a lock, but it is only updated on thread shutdown. It is purged during reload.

Refactor overview:

  • metrics-cache --> dyn-metrics-store
  • metrics-tls-cache --> dyn-metrics-cache

I've also tried to decouple metrics-template from the cache functionality, but it would have had performance implications, so I only added a comment about the coupled functionality.

This is the first step of a refactor round that tries to decouple a few
dynamic-metric-related units from the cache built around it.

The end goal is to fix a lifetime issue that is related to the TLS cache
version of this store.

Signed-off-by: László Várady <[email protected]>
MrAnno added a commit to MrAnno/axosyslog that referenced this pull request Aug 8, 2024
Signed-off-by: László Várady <[email protected]>
@MrAnno MrAnno force-pushed the fix-metrics-probe-cntr-lifetime branch from 78b9ffe to 9e04f46 Compare August 8, 2024 07:54
It is no longer called a "cache", so "retrieve" may express the
functionality better.

Signed-off-by: László Várady <[email protected]>
This is a preparation step for fixing a lifetime bug related to TLS
caching. This unit will be extended with a global cache in order not to
make metrics orphaned whenever an ivykis worker stops.

Signed-off-by: László Várady <[email protected]>
MrAnno added a commit to MrAnno/axosyslog that referenced this pull request Aug 8, 2024
Signed-off-by: László Várady <[email protected]>
@MrAnno MrAnno force-pushed the fix-metrics-probe-cntr-lifetime branch from 9e04f46 to 4e1745c Compare August 8, 2024 09:15
@MrAnno
Copy link
Member Author

MrAnno commented Aug 8, 2024

@kira-syslogng do stresstest

Okay, I'm stupid :D

To denote that this is about templating dynamic counters.

Signed-off-by: László Várady <[email protected]>
It would have been better to decouple the metrics template unit from the
cache, but this would have had performance implications.

Signed-off-by: László Várady <[email protected]>
Signed-off-by: László Várady <[email protected]>
This is required for merging the TLS-cache into a global cache to avoid
making dynamic metrics orphan on thread shutdown.

Signed-off-by: László Várady <[email protected]>
This unit has been extended with a global cache in order not to
make metrics orphaned whenever an ivykis worker stops (after 10 seconds of
inactivity).

The global cache requires a lock, but it is only updated on thread
shutdown.

The global cache is purged during reload.

Signed-off-by: László Várady <[email protected]>
Signed-off-by: László Várady <[email protected]>
@MrAnno MrAnno force-pushed the fix-metrics-probe-cntr-lifetime branch from 4e1745c to 10376b4 Compare August 8, 2024 09:56
Copy link
Member

@bazsi bazsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Great work.

@MrAnno MrAnno merged commit d356f61 into axoflow:main Aug 13, 2024
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants