Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alleviate excessive periodic logging during conflict check #1400

Merged
merged 2 commits into from
Oct 5, 2023

Conversation

tpantelis
Copy link
Contributor

@tpantelis tpantelis commented Oct 3, 2023

If a service is imported without a local service export, the conflict checking tries to update the status of a non-existent ServiceExport, specifically to remove the Conflict status condition. This results in a log message: ServiceExport (...) not found - unable to update status

Due to the 2 minute resync intervaL for the EndpointSlice syncer, this can exhaust the pod disk storage over time.

(first commit) To alleviate this issue, check the local cache for a ServiceExport and, if non-existent, skip the conflict checking. If it does exist but the Conflict status condition doesn't exist in the local ServiceExport then elide trying to remove the status condition. Also, change the log level to TRACE for the offending message.

(second commit) Optimize conflict checking using the local cache to list the EndpointSlices instead of the API server client.

@submariner-bot
Copy link
Contributor

🤖 Created branch: z_pr1400/tpantelis/excess_logging
🚀 Full E2E won't run until the "ready-to-test" label is applied. I will add it automatically once the PR has 2 approvals, or you can add it manually.

pkg/agent/controller/types.go Outdated Show resolved Hide resolved
If a service is imported without a local service export, the conflict
checking tries to update the status of a non-existent ServiceExport,
specifically to remove the Conflict status condition. This results in
a log message:

 "ServiceExport (...) not found - unable to update status"

Due to the 2 minute resync intervaL for the EndpointSlice syncer, this
can exhaust the pod disk storage over time.

To alleviate this issue, check the local cache for a ServiceExport and,
if non-existent, skip the conflict checking. If it does exist but the
Conflict status condition doesn't exist in the local ServiceExport then
elide trying to remove the status condition.

Finally, change the log level to TRACE for the offending message.

Signed-off-by: Tom Pantelis <[email protected]>
Use the local cache to list the EndpointSlices instead of the API server
client.

Signed-off-by: Tom Pantelis <[email protected]>
@submariner-bot submariner-bot added the ready-to-test When a PR is ready for full E2E testing label Oct 4, 2023
@sridhargaddam sridhargaddam merged commit 4973339 into submariner-io:devel Oct 5, 2023
26 checks passed
@submariner-bot
Copy link
Contributor

🤖 Closed branches: [z_pr1400/tpantelis/excess_logging]

@sridhargaddam
Copy link
Member

Do we want to backport this PR?

@skitt
Copy link
Member

skitt commented Oct 5, 2023

Do we want to backport this PR?

I think so, I’ve taken care of it — if there are objections feel free to raise them on the backport PR.

@tpantelis tpantelis deleted the excess_logging branch December 13, 2023 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport backport-handled ready-to-test When a PR is ready for full E2E testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants