Cacheability improvements for thirdparty audit task #42085
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/es-core-infra |
rjernst
approved these changes
May 10, 2019
|
|
||
| // Mark successful third party audit check | ||
| getSuccessMarker().getParentFile().mkdirs(); | ||
| Files.write(getSuccessMarker().toPath(), new byte[]{}, StandardOpenOption.CREATE); |
There was a problem hiding this comment.
Wouldn't this fail on subsequent runs? I think we want the default options?
There was a problem hiding this comment.
I'm not sure, I think it just creates if it doesn't exist. I stole this from PrecommitTask. I think using the default options is probably the simpler thing to do though.
mark-vieira
added a commit
to mark-vieira/elasticsearch
that referenced
this pull request
May 15, 2019
mark-vieira
added a commit
that referenced
this pull request
May 15, 2019
gurkankaymak
pushed a commit
to gurkankaymak/elasticsearch
that referenced
this pull request
May 27, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While doing some build cache experimentation I noticed that we had already made our
ThirdPartyAuditTaskcacheable. This PR makes some improvements here:First, we no longer consider the expanded jars directory as an output. While it is technically "output" that the task creates, it's not the "result" of the task. This is just intermediary output that is created but the result of the task is effectively the error (or lack of error) thrown to the console if an issue is detected. The issue with tracking that expanded jar directory as an output is when we cache this task we pack all that up, essentially, the entire project runtime classpath. This can be quite large and negatively effects build cache performance. When we fetch this task from the cache we don't really need any of that so instead this has been refactored to use the "success marker" pattern of simply writing an empty file to the build directory to indicate this task was successful.
Second, we now use the runtime classpath normalization strategy for the
jarsToScan. This ensures that things like jar entry order or creation timestamps don't affect the cache hash. When using@InputFileswe do a strict byte-to-byte comparison of the files, so for archives, this can cause cache misses when the upstream archive is rebuilt, but effectively identical.