Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache conditions when applying variables to config #6229

Merged
merged 2 commits into from
Dec 11, 2024
Merged

Cache conditions when applying variables to config #6229

merged 2 commits into from
Dec 11, 2024

Conversation

swiatekm
Copy link
Contributor

@swiatekm swiatekm commented Dec 5, 2024

What does this PR do?

Our configurations can have conditional sections. The conditions are expressed in an EQL-like syntax, and we parse the expressions lazily on evaluation. However, we don't cache the parsed expression, and parse it every time it's evaluated. This change instead only parses the condition expression once, and it's then cached in the AST node.

Why is it important?

When there are a lot of variables from dynamic providers - notably in Kubernetes, on a Node with a lot of Pods - we end up spending significant resources on re-parsing the condition expressions for every var entry.

See benchstat showing the effect of this change on the benchmark from #6180 (not that this includes #6184, as otherwise we just make a copy before applying and never actually use the cache):

goos: linux
goarch: amd64
pkg: github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator
cpu: 13th Gen Intel(R) Core(TM) i7-13700H
                                      │ bench_noclone.txt │            bench_eql.txt            │
                                      │      sec/op       │   sec/op     vs base                │
Coordinator_generateComponentModel-20         32.36m ± 4%   27.45m ± 2%  -15.17% (p=0.000 n=10)

                                      │ bench_noclone.txt │            bench_eql.txt             │
                                      │       B/op        │     B/op      vs base                │
Coordinator_generateComponentModel-20        25.38Mi ± 0%   20.51Mi ± 0%  -19.17% (p=0.000 n=10)

                                      │ bench_noclone.txt │            bench_eql.txt            │
                                      │     allocs/op     │  allocs/op   vs base                │
Coordinator_generateComponentModel-20         580.4k ± 0%   490.1k ± 0%  -15.57% (p=0.000 n=10)

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

How to test this PR locally

Unit tests are sufficient, the benchmark in #6180 helps as well.

Related issues

@swiatekm swiatekm added enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify labels Dec 5, 2024
@swiatekm swiatekm requested a review from a team as a code owner December 5, 2024 15:19
@swiatekm swiatekm requested review from blakerouse and pchila December 5, 2024 15:19
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@swiatekm swiatekm force-pushed the feat/ast-eql-parsing branch from 9165189 to 83fcb38 Compare December 5, 2024 15:37
cmacknz
cmacknz reviewed Dec 9, 2024
View reviewed changes
internal/pkg/agent/transpiler/ast_test.go
"data": "info",
}})

input := NewKey("condition", NewStrVal("${other.data} == 'info'"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be a test for the case where the condition is edited for the same input.

As an implementation question does the cache need to be invalidated when this happens, or is a new AST node created so that we don't need to care?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not possible to update the configuration without generating a new AST. That means its not possible for the value of a key to change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, if there's already a test that conditions edits (or any configuration edit) results in a new AST node then that covers this.

If there isn't, we should add one. Reusing a cached condition would be quite a difficult bug to track down from an end user report.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ability to modify an AST is very limited to begin with. The only public API to do it is the Insert function, which only works on Dicts and Lists. I can add a test where I replace a key in a dict and verify the condition doesn't get saved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b5ac1c4

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 9, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/ast-eql-parsing upstream/feat/ast-eql-parsing
git merge upstream/main
git push upstream feat/ast-eql-parsing

blakerouse
blakerouse approved these changes Dec 9, 2024
View reviewed changes
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

internal/pkg/agent/transpiler/ast_test.go
"data": "info",
}})

input := NewKey("condition", NewStrVal("${other.data} == 'info'"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not possible to update the configuration without generating a new AST. That means its not possible for the value of a key to change.

@swiatekm
Cache conditions when applying variables to config
ff23be4 
# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go
@swiatekm swiatekm force-pushed the feat/ast-eql-parsing branch from 83fcb38 to ff23be4 Compare December 10, 2024 14:48
@swiatekm swiatekm requested a review from cmacknz December 10, 2024 14:55
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
88.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@swiatekm swiatekm merged commit cab5754 into main Dec 11, 2024
14 checks passed
@swiatekm swiatekm deleted the feat/ast-eql-parsing branch December 11, 2024 10:37
mergify bot pushed a commit that referenced this pull request Dec 11, 2024
@swiatekm @mergify
Cache conditions when applying variables to config (#6229)
25baa7e 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)
mergify bot pushed a commit that referenced this pull request Dec 11, 2024
@swiatekm @mergify
Cache conditions when applying variables to config (#6229)
5ecfc6c 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go
This was referenced Dec 11, 2024
Merged
Merged
mergify bot pushed a commit that referenced this pull request Dec 11, 2024
@swiatekm @mergify
Cache conditions when applying variables to config (#6229)
4a89cf8 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go
@mergify mergify bot mentioned this pull request Dec 11, 2024
Merged
4 tasks
swiatekm added a commit that referenced this pull request Dec 11, 2024
@swiatekm
Cache conditions when applying variables to config (#6229)
8d2d764 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go
swiatekm added a commit that referenced this pull request Dec 11, 2024
@mergify @swiatekm
Cache conditions when applying variables to config (#6229) (#6279)
ed44ce9 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

Co-authored-by: Mikołaj Świątek <[email protected]>
swiatekm added a commit that referenced this pull request Dec 11, 2024
@mergify @swiatekm
Cache conditions when applying variables to config (#6229) (#6280)
8b73c42 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

Co-authored-by: Mikołaj Świątek <[email protected]>
swiatekm added a commit that referenced this pull request Dec 13, 2024
@swiatekm
Cache conditions when applying variables to config (#6229)
eeab30d 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go
swiatekm added a commit that referenced this pull request Dec 16, 2024
@mergify @swiatekm
Cache conditions when applying variables to config (#6229) (#6281)
4af8f21 
* Cache conditions when applying variables to config

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

* Add test for AST insertion

(cherry picked from commit cab5754)

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

# Conflicts:
#	internal/pkg/agent/transpiler/ast_test.go

Co-authored-by: Mikołaj Świątek <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blakerouse blakerouse blakerouse approved these changes

@cmacknz cmacknz cmacknz approved these changes

@pchila pchila Awaiting requested review from pchila pchila is a code owner automatically assigned from elastic/elastic-agent-control-plane

Assignees

@swiatekm swiatekm

Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@swiatekm @elasticmachine @cmacknz @blakerouse