-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache images slightly differently #165
Conversation
Let me know when this is ready for review, it looks exciting! |
for (const implFamily of fs.readdirSync(path.join(multidimInteropDir, 'impl'))) { | ||
const ig = ignore() | ||
|
||
addGitignoreIfPresent(ig, path.join(multidimInteropDir, ".gitignore")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to extract path.join(multidimInteropDir, ".gitignore")
and others constants as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's the same, but unless you feel strongly I'd rather just leave it.
Nice work @MarcoPolo! |
closes #161
In an effort to speed things up, this changes the caching strategy from relying on Docker's somewhat opaque rules to simply caching the generated image. Here's how it works:
$IMAGE_NAME-$CACHE_KEY-$ARCH.tar.gz
, if we do import that image.make -o image.json
which allows the implementation to do some extra work from cache (e.g. js-libp2p needs to build both the node and browser images, so it can cache the base and do quick work to build the final node and browser images).PUSH_CACHE
env we push the built image to s3.Q: Why not a GH registry?
A: Then we have to deal with GC. This lets us rely on the s3 behavior of a lifetime for objs.
Q: Why not an S3 registry?
A: That requires a bit more work to set up. You need a registry container running in the background. This is simpler. It also supports public read only access to the cache.
Benefits
Why didn't we do this from the start?
One step at a time. The build layer caching strategy was convenient and it was correct. I was hoping it would be better, but here we are.
Note that this solution is different (and imo better) than each implementation managing its own images and pushing to a registry. I say better since only this repo needs to worry about pushing images and GC'ing them.