feat: Split pipeline in module stages #14484
Conversation
kuisathaverat
force-pushed
the
feat/split-stages
branch
9 times, most recently
from
825285c to
d74db83
Compare
| @@ -0,0 +1,24 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
@exekias as soon as we have this in, we should probably disable it in travis?
There was a problem hiding this comment.
Why not having it in both places?
There was a problem hiding this comment.
It's a more general comment. If we get the power of the Jenkins Pipeline in our repo, we should disable travis all together so we only have to worry about 1 flaky environment. And I'm sure @kuisathaverat guarantees us 100% availability for Jenkins 🤣
There was a problem hiding this comment.
Yeah, I guess we can have this discussion once we feel comfortable with jenkins. One point to still having both in the past was that jenkins is not run by default for external contributions.
There was a problem hiding this comment.
Good point, forgot about this one. Would still be nice to have Jenkins also run for them by default, or at least the once which have signed the CLA.
There was a problem hiding this comment.
It should be safe to drop versions older than 1.11, cc @odacremolbap
There was a problem hiding this comment.
If k8s causes trouble, I would suggest to drop it from the initial PR and then follow up.
There was a problem hiding this comment.
it is working with k8s versions "v1.16.2","v1.15.3","v1.14.6","v1.13.10","v1.12.10", and "v1.11.10"
kuisathaverat
force-pushed
the
feat/split-stages
branch
2 times, most recently
from
eca07f8 to
344245d
Compare
There was a problem hiding this comment.
This is just great! Approving assuming it will be green ;-)
All the comments below are more a follow up of this PR and should not be implemented in this PR. I post them here so we have an easy place to find them again:
- It is great that linux oss is run always first because if this one fails, it is likely that the other ones will fail too.
- Is there a way to retrigger all jobs? What I mean by that is assuming I changed heartbeat but I really want to run the full testsuite, is this possible?
- I see in the CI that for example Linux oss was run but not Linux-Xpack for filebeat. I assume that is something that shouldn't happen?
- Exclude params: This is not 100% correct yet but good enough. We probably need something like
!glob(**/*beat/). It means that if also any global file not inside a beat directory (for example inside dev-tools) all builds are triggered. - We will need to add Windows support
- We will need crosscompile for all oss and x-pack builds. This could happen in parallel. Is is possible to do this last but in parallel?
- It is great like right now for libbeat for example, we just call
testsuite. If libbeat comes the slowest one to build, we can start splitting it up into unit, system, integrations tests and run it in parallel. Like this we can always tackle the slowest in the overall build and speed up things.
kuisathaverat
force-pushed
the
feat/split-stages
branch
13 times, most recently
from
e3cb78a to
229c95d
Compare
kuisathaverat
force-pushed
the
feat/split-stages
branch
3 times, most recently
from
da9137d to
e1931ba
Compare
kuisathaverat
force-pushed
the
feat/split-stages
branch
3 times, most recently
from
126ed2e to
59f7388
Compare
kuisathaverat
force-pushed
the
feat/split-stages
branch
from
59f7388 to
6753bea
Compare
|
example PR with Metricbeat changes only
|
|
jenkins run tests |
kuisathaverat
force-pushed
the
feat/split-stages
branch
from
b14f798 to
8693ec6
Compare
|
jenkins run the tests |
|
🎉 |
This copies the build process that we have in Travis. Every module has its own stage that runs in parallel with the other modules stages, modules are only built and test if there are changes in the module or in libbeat, this is a start point to build only the things that we need.
* cherry-pick aad07ad * Add stages to Jenkins pipeline (#14484) * ci: avoid to modify go.mod file (#16842) * Ensure install scripts only install if needed (#20349) * ci: fix warnings with wildcards and archive system-tests (#18695) * ci: run test on Windows (#15570) * [CI] fail if not possible to install python3 (#19164) * [CI] lint stage doesn't produce test reports (#21888) * [CI] Add stage name in the step (#21887) * [CI] kind setup fails sometimes (#21857) * [CI] refactor runbld post build action (#21256) * [CI] Archive build reasons (#21347) * [CI] Fix runbld when workspace does not exist (#21350) * [CI] Send slack message with build status (#21428) * [CI] Setup git config globally (#21562) * [CI] fix 'no matches found within 10000' (#21466) * [CI] Change notification channel (#21559) * [CI] Support skip-ci label (#21377) * [BUILD][CI] fetch dependencies with retry (#21614) * Disable x-pack/metricbeat in linux since build is broken #21962 * Disable tests for windows in auditbeat. See #21958 * Disable integration testing for libbeat. See #21959 * Disable integration testing for metricbeat. See #21961 * Disable windows build for auditbeat. See #21958 * Disable build for libbeat. See #21959 * Disable build for metricbeat. See #21961 * Disable windows build for metricbeat. See #22257 * Disable build for x-pack/metricbeat. See #21962 * Disable windows build for x-pack/winlogbeat. See #22263 * Disable windows build for x-pack/filebeat. See #21963 * integration-tests-environment is not available for winlogbeat. see #22265 * integration-tests-environment is not available for packetbeat. see #22266 * Build not available for x-pack/metricbeats in 6.8. See #22269 * Cherry-pick 96c60d0
What does this PR do?
It copies the build process that we have in Travis. Every module has its own stage that runs in parallel with the other modules stages, modules are only built and test if there are changes in the module or in libbeat, this is a start point to build only the things that we need.
Why is it important?
It is the start point to improve the user CI experience and optimize the build times.