One pipeline, two dockerfiles and hadolint

[ldelcano]

I have created and tested two separated pipelines for each Docker file but I would like to join them in one.
For that I create a single pipeline and two services: scipion-worker and scipion-master.
Then I go to the criteria description and there I have problems:
First I tried the following: I add a single criterion (Sty) with two hadolint builder tools: one to be run on the worker Dockerfile and one to be run on the master Dockerfile but I can see that the second overrides the first one. See config.yaml file:
config:
project_repos:
github.com/EOSC-synergy/sqaaas-tooling:
repo: "https://github.com/EOSC-synergy/sqaaas-tooling"
branch: "main"
credentials: []
environment: {}
sqa_criteria:
QC.Sty:
repos:
this_repo:
container: "qc.sty_boozy-ruby-dalmatian"
commands:
- "hadolint master-image/Dockerfile --ignore SC2046 --ignore DL3002 --ignore DL3059 --ignore DL3047 --ignore SC2086 --ignore DL3027 --ignore DL3015 --ignore DL3008 --ignore DL3007 --ignore DL3003 --ignore SC2016 --ignore DL4006 --ignore DL3020 --ignore SC2028"
- "hadolint master-image/Dockerfile --ignore SC2046 --ignore DL3002 --ignore DL3059 --ignore DL3047 --ignore SC2086 --ignore DL3027 --ignore DL3015 --ignore DL3008 --ignore DL3007 --ignore DL3003 --ignore SC2016 --ignore DL4006 --ignore DL3020 --ignore SC2028"
timeout: 600

Then I add two criterions one to run a hadolint tool on each Dockerfile, but the yaml does not look better:
config:
project_repos:
github.com/EOSC-synergy/sqaaas-tooling:
repo: "https://github.com/EOSC-synergy/sqaaas-tooling"
branch: "main"
credentials: []
environment: {}
sqa_criteria:
QC.Sty:
repos:
this_repo:
container: "qc.sty_homey-teal-peccary"
commands:
- "hadolint master-image/Dockerfile --ignore SC2046 --ignore DL3002 --ignore DL3059 --ignore DL3047 --ignore SC2086 --ignore DL3027 --ignore DL3015 --ignore DL3008 --ignore DL3007 --ignore DL3003 --ignore SC2016 --ignore DL4006 --ignore DL3020 --ignore SC2028"
timeout: 600

how should I do to test both dockerfiles on a single pipeline?
thanks
Laura

[orviz]

Thanks @ldelcano we are looking into this, there is definitely something wrong when using the hadolint tool available in the dropdown list. While we solve this, the way to do what you want is to rely on commands builder so there you define both commands using the same service (a predefined service with the hadolint image, such as the usual 'hadolint-scipion`). See the following screenshot:

[ldelcano]

Thanks Pablo, I will do that.
I have two more questions:

1. what do I do with the pipeline when it's ready? Should you create a fork from our git repo in EOSC-Synergy project and add there the yaml file so it is executed automatically? (but both repos should then be syncronized).
2. can you delete from eosc-synergy jenkins portal all old Scipion related executions? They were tests and are not useful anymore. I would prefere to have the good one to easily check the status.
   thanks

[ldelcano]

Sorry I forgot to add this screenshot with the search for everything Scipion related in the project jenkins:

[orviz]

@ldelcano comments inline

• what do I do with the pipeline when it's ready? Should you create a fork from our git repo in EOSC-Synergy project and add there the yaml file so it is executed automatically? (but both repos should then be syncronized).

have you tried the 'Pull Request' feature? Through this feature the SQAaaS will create a PR to any repo as long as it exists on GitHub, so you can review & approve it if everything is ok. The PR will only contain the files needed for the pipeline execution (so no need to sync the content).

Also, there is no requirement that the target repo shall be on the GitHub's EOSC-Synergy org (only on GitHub), but the added value of using this org is that your pipeline will run automatically everytime something changes in your code (since our jenkins is configured to watch for changes in all the EOSC-Synergy repos)

• can you delete from eosc-synergy jenkins portal all old Scipion related executions? They were tests and are not useful anymore. I would prefere to have the good one to easily check the status.

Sure I can do that, I will go through the list you provided

Thanks!

[ldelcano]

@orviz I am not sure if the argument is working cause I realized that if I work on a pipeline that already exists the Docker build cache is there and image is not built again. Is there a way to force that the image is built regardless of the cache?

[orviz]

@ldelcano sorry could you tell me what argument it this? SQAaaS is currently using version 2.1.1 of JePL that still does not allow to avoid Docker cache mechanism. This feature is planned for the incoming release 2.2.0, but we don't have yet a date for that (@samuelbernardolip ?)

[orviz]

btw if you have sort of a definitive Dockerfile I can manually remove the cache from our Docker server to circumvent this issue for this time. Just tell me the image name you are using

[ldelcano]

@orviz the argument that I pass to the scipion-master service is NOGPU=yes and what it does is to skip some command in the Dockerfile that only should be run if a GPU is present in the host. I think it works but I am not sure anymore since due to the cache the docker image is never built again.
The image is in https://github.com/I2PC/scipion-docker/blob/master/master-image/Dockerfile
Another thing, I tried the Pull request feature, first on the EOSC-Synergy repo, which I think does not make any sense cause jenkins files are already there, and also in our repo, but I guess having them there does not make any test automation unless I connect a jenkins server which I do not have. I guess what I would like is to trigger an execution anytime there is a push on a git branch to make sure nothing was broken.

[orviz]

@ldelcano just changed the default behaviour in the SQAaaS API when pushing images. Could you please try to recreate the pipeline to check that now the latest built image is the one being pushed? Or either, if you prefer to modify the existing pipeline, be sure that you enable the JPL_DOCKERFORCEPUSH variable in config.yml such as:

environment:
    JPL_DOCKERFORCEPUSH: "enabled"

In what regards to the PR, the best way for your case (no access to a Jenkins server) is to fork your repo inside the EOSC-Synergy org. Since as you said you already have the pipeline files there, you don't need to do anything else, our Jenkins will detect your repo (and trigger a build whenever something changes there). I've just sent you an invitation to be a member of EOSC-Synergy org, I think you should be able to create the fork once you accept it. Let me know any issues.