Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build more regular expressions at boot time #604

Merged
merged 1 commit into from
Oct 8, 2024
Merged

Conversation

seanlinsley
Copy link
Member

@seanlinsley seanlinsley commented Oct 4, 2024

For support ticket 5773, I noticed that some places in the code we compile regular expressions at runtime for regular expressions that don't change. This PR fixes that, which should reduce CPU/memory usage.

state/logs.go Outdated Show resolved Hide resolved
@seanlinsley seanlinsley changed the title Reduce memory pressure from logs Build regular expressions at compile time Oct 8, 2024
@seanlinsley seanlinsley marked this pull request as ready for review October 8, 2024 18:26
@seanlinsley seanlinsley requested a review from a team October 8, 2024 18:26
@seanlinsley seanlinsley changed the title Build regular expressions at compile time Build more regular expressions at boot time Oct 8, 2024
@seanlinsley
Copy link
Member Author

TIL: despite Go being a compiled language, these variables are actually defined when the program boots, not when it's compiled https://stackoverflow.com/a/72185182

Copy link
Contributor

@msakrejda msakrejda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the cleanup.

// We ignore failures here since we want the per-backend stitching logic
// that runs later on (and any other parsing errors will just be ignored).
// Note that we need to restore the original trailing newlines since
// AnalyzeStreamInGroups expects them and they are not present in the GCP
// log stream.
logLine, _ := parser.ParseLine(in.Content + "\n")
logLine, _ := server.GetLogParser().ParseLine(in.Content + "\n")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No big deal, but if we get a nil panic here, it now won't be clear if server is nil or GetLogParser() returned nil (or in is nil, but that was the case before).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since in is coming from a channel and we check ok, I think it's impossible for in to be nil. But even if it was, lines of code above this would have already panicked by accessing in.

state/logs.go Outdated Show resolved Hide resolved
@seanlinsley seanlinsley merged commit e672751 into main Oct 8, 2024
3 checks passed
@seanlinsley seanlinsley deleted the logs-memory branch October 8, 2024 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants