Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Periodically update image with pre-installed drivers #309

Closed
lwsanty opened this issue Sep 18, 2019 · 6 comments · Fixed by #314
Closed

Periodically update image with pre-installed drivers #309

lwsanty opened this issue Sep 18, 2019 · 6 comments · Fixed by #314

Comments

@lwsanty
Copy link
Member

lwsanty commented Sep 18, 2019

Pre-history

Currently we build docker image with all pre-installed drivers only when travis ci is triggered(on per-commit to master basis)

Thus we skip new driver versions until the next ci trigger for bblfshd

Proposal

Add cron job to bblfshd's travis ci so we will have an ability to update drivers once in a period

Problems

  • is not set up in config, so we need to file an issue to infra and mention desired period
  • what should we do if no changes detected? IMO we can allow travis to run on the repo with no changes in drivers since the prev time, because in case of the same images the won't be pushed.
@dennwc
Copy link
Member

dennwc commented Sep 18, 2019

This makes a lot of sense, but we should also think about a few more things:

  • If we always update builtin drivers, it won't be possible for anymore to lock bblfshd version with a specific versions of builtin drivers. Maybe we can come up with some tag naming scheme to cover both bblfshd version and drivers? (x.y.z-drivers-2019-09?)
  • When updating, we should probably reuse existing bblfshd image and install drivers over it, instead of rebuilding bblfshd again. This will make sure the change is incremental in regards to Docker layers.

@lwsanty
Copy link
Member Author

lwsanty commented Sep 18, 2019

@vmarkovtsev wdyt about @dennwc 's comment, will it be useful for ML team?

@vmarkovtsev
Copy link
Contributor

As long as there is a tag which includes both the bblfshd version and the drivers date, it should be fine.

Speaking about us, we always install the drivers manually and choose only those which we need. We could switch to the full image if it is updated regularly.

lwsanty added a commit that referenced this issue Oct 4, 2019
@lwsanty
docker image cron update
7f471ef 
This PR contains changes to travis and Makefile configs for periodical docker image update using travis ci cron jobs

Behavior
1) if TRAVIS_EVENT_TYPE is cron then
  a) try to get tag using `$( git describe --exact-match "$(git rev-parse HEAD)`
  b) if tag is not empty run make push-drivers
  c) in push-drivers: if TRAVIS_EVENT_TYPE is cron pull base bblfshd image instead of rebuilding it from the scratch
2) else build as always

Changes to docker tags:
- timestamp is added to the version tag, so the final image will look like bblfsh/bblfshd:v2.14.0-drivers-2019-10-04T14_11

Problems:
- afaik deploy will not be triggered on cron job because tags are not accessible with this event type travis-ci/travis-ci#8146 ... cannot check it on practice thus 1.a) workaround has been implemented as suggested travis-ci/travis-ci#8146 (comment)

closes #309

Signed-off-by: lwsanty <[email protected]>
@lwsanty lwsanty mentioned this issue Oct 4, 2019
Merged
lwsanty added a commit that referenced this issue Oct 4, 2019
@lwsanty
docker image cron update
e2988a9 
This PR contains changes to travis and Makefile configs for periodical docker image update using travis ci cron jobs

Behavior
1) if TRAVIS_EVENT_TYPE is cron then
  a) try to get tag using `$( git describe --exact-match "$(git rev-parse HEAD)`
  b) if tag is not empty run make push-drivers
  c) in push-drivers: if TRAVIS_EVENT_TYPE is cron pull base bblfshd image instead of rebuilding it from the scratch
2) else build as always

Changes to docker tags:
- timestamp is added to the version tag, so the final image will look like bblfsh/bblfshd:v2.14.0-drivers-2019-10-04

Problems:
- afaik deploy will not be triggered on cron job because tags are not accessible with this event type travis-ci/travis-ci#8146 ... cannot check it on practice thus 1.a) workaround has been implemented as suggested travis-ci/travis-ci#8146 (comment)

closes #309

Signed-off-by: lwsanty <[email protected]>

change tag datetime to date only

Signed-off-by: lwsanty <[email protected]>
dennwc pushed a commit that referenced this issue Oct 8, 2019
@lwsanty @dennwc
docker image cron update
49ac0e4 
This PR contains changes to travis and Makefile configs for periodical docker image update using travis ci cron jobs

Behavior
1) if TRAVIS_EVENT_TYPE is cron then
  a) try to get tag using `$( git describe --exact-match "$(git rev-parse HEAD)`
  b) if tag is not empty run make push-drivers
  c) in push-drivers: if TRAVIS_EVENT_TYPE is cron pull base bblfshd image instead of rebuilding it from the scratch
2) else build as always

Changes to docker tags:
- timestamp is added to the version tag, so the final image will look like bblfsh/bblfshd:v2.14.0-drivers-2019-10-04

Problems:
- afaik deploy will not be triggered on cron job because tags are not accessible with this event type travis-ci/travis-ci#8146 ... cannot check it on practice thus 1.a) workaround has been implemented as suggested travis-ci/travis-ci#8146 (comment)

closes #309

Signed-off-by: lwsanty <[email protected]>

change tag datetime to date only

Signed-off-by: lwsanty <[email protected]>
@lwsanty
Copy link
Member Author

lwsanty commented Oct 8, 2019 •
edited
Loading

need to wait for https://github.com/src-d/infrastructure/issues/1203 until changes start to work

@dpordomingo
Copy link
Member

I know this is already closed, but found when bumping sourced-ce with the new v2.16.1-drivers release:
Did you consider to release, with the cron-ed release, an extra v2.16.1-drivers-latest one?

@ncordon
Copy link
Member

ncordon commented Oct 31, 2019

I think this is related: #329

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

docker image cron update bblfsh/bblfshd
5 participants
@dennwc @dpordomingo @vmarkovtsev @ncordon @lwsanty