Skip to content

Commit

Permalink
fix(apache#592): Introduce "dependencies" build order strategy
Browse files Browse the repository at this point in the history
- Introduce new build order strategy "dependencies"
- Strategy looks at the list of dependencies required by an Integration and queues builds that may reuse base images produced by other scheduled builds in order to leverage the incremental build option
- The strategy allows non-matching builds to run in parallel to each other
  • Loading branch information
christophd committed Jul 5, 2023
1 parent 2e05e48 commit b851342
Show file tree
Hide file tree
Showing 26 changed files with 662 additions and 53 deletions.
2 changes: 2 additions & 0 deletions config/crd/bases/camel.apache.org_builds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ spec:
orderStrategy:
description: the build order strategy to adopt
enum:
- dependencies
- fifo
- sequential
type: string
Expand Down Expand Up @@ -215,6 +216,7 @@ spec:
orderStrategy:
description: the build order strategy to adopt
enum:
- dependencies
- fifo
- sequential
type: string
Expand Down
4 changes: 2 additions & 2 deletions config/crd/bases/camel.apache.org_integrationkits.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the build
Expand Down
10 changes: 6 additions & 4 deletions config/crd/bases/camel.apache.org_integrationplatforms.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ spec:
orderStrategy:
description: the build order strategy to adopt
enum:
- dependencies
- fifo
- sequential
type: string
Expand Down Expand Up @@ -469,8 +470,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the build
Expand Down Expand Up @@ -1766,6 +1767,7 @@ spec:
orderStrategy:
description: the build order strategy to adopt
enum:
- dependencies
- fifo
- sequential
type: string
Expand Down Expand Up @@ -2164,8 +2166,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the build
Expand Down
4 changes: 2 additions & 2 deletions config/crd/bases/camel.apache.org_integrations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6195,8 +6195,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the build
Expand Down
4 changes: 2 additions & 2 deletions config/crd/bases/camel.apache.org_kameletbindings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6470,8 +6470,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the
Expand Down
4 changes: 2 additions & 2 deletions config/crd/bases/camel.apache.org_pipes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6467,8 +6467,8 @@ spec:
of memory required by the pod builder.
type: string
orderStrategy:
description: The build order strategy to use, either `fifo`
or `sequential` (default sequential)
description: The build order strategy to use, either `dependencies`,
`fifo` or `sequential` (default sequential)
type: string
properties:
description: A list of properties to be provided to the
Expand Down
1 change: 1 addition & 0 deletions docs/modules/ROOT/pages/architecture/cr/build.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ You can choose from different build order strategies. The strategy defines in wh
At the moment the available strategies are:

- buildOrderStrategy: sequential (runs builds strictly sequential so that only one single build per operator namespace is running at a time.)
- buildOrderStrategy: dependencies (strategy looks at the list of dependencies required by an Integration and queues builds that may reuse base images produced by other scheduled builds in order to leverage the incremental build option. The strategy allows non-matching builds to run in parallel to each other.)
- buildOrderStrategy: fifo (performs the builds with first in first out strategy based on the creation timestamp. The strategy allows builds to run in parallel to each other but oldest builds will be run first.)

[[build-queue]]
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/ROOT/partials/apis/camel-k-crds.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5412,7 +5412,7 @@ string
|
The build order strategy to use, either `fifo` or `sequential` (default sequential)
The build order strategy to use, either `dependencies`, `fifo` or `sequential` (default sequential)
|`requestCPU` +
string
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/traits/pages/builder.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The following configuration options are available:

| builder.order-strategy
| string
| The build order strategy to use, either `fifo` or `sequential` (default sequential)
| The build order strategy to use, either `dependencies`, `fifo` or `sequential` (default sequential)

| builder.request-cpu
| string
Expand Down
74 changes: 74 additions & 0 deletions e2e/builder/build_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,80 @@ func TestKitMaxBuildLimitFIFOStrategy(t *testing.T) {
})
}

func TestKitMaxBuildLimitDependencyMatchingStrategy(t *testing.T) {
WithNewTestNamespace(t, func(ns string) {
createOperator(ns, "8m0s", "--global", "--force")

pl := Platform(ns)()
// set maximum number of running builds and order strategy
pl.Spec.Build.MaxRunningBuilds = 2
pl.Spec.Build.BuildConfiguration.OrderStrategy = v1.BuildOrderStrategyDependencies
if err := TestClient().Update(TestContext, pl); err != nil {
t.Error(err)
t.FailNow()
}

buildA := "integration-a"
buildB := "integration-b"
buildC := "integration-c"

doKitBuildInNamespace(buildA, ns, TestTimeoutShort, kitOptions{
operatorID: fmt.Sprintf("camel-k-%s", ns),
dependencies: []string{
"camel:timer", "camel:log",
},
traits: []string{
"builder.properties=build-property=A",
},
}, v1.BuildPhaseRunning, v1.IntegrationKitPhaseBuildRunning)

doKitBuildInNamespace(buildB, ns, TestTimeoutShort, kitOptions{
operatorID: fmt.Sprintf("camel-k-%s", ns),
dependencies: []string{
"camel:cron", "camel:log", "camel:joor",
},
traits: []string{
"builder.properties=build-property=B",
},
}, v1.BuildPhaseRunning, v1.IntegrationKitPhaseBuildRunning)

doKitBuildInNamespace(buildC, ns, TestTimeoutShort, kitOptions{
operatorID: fmt.Sprintf("camel-k-%s", ns),
dependencies: []string{
"camel:timer", "camel:log", "camel:joor", "camel:http",
},
traits: []string{
"builder.properties=build-property=C",
},
}, v1.BuildPhaseScheduling, v1.IntegrationKitPhaseNone)

var notExceedsMaxBuildLimit = func(runningBuilds int) bool {
return runningBuilds <= 2
}

limit := 0
for limit < 5 && BuildPhase(ns, buildA)() == v1.BuildPhaseRunning {
// verify that number of running builds does not exceed max build limit
Consistently(BuildsRunning(BuildPhase(ns, buildA), BuildPhase(ns, buildB), BuildPhase(ns, buildC)), TestTimeoutShort, 10*time.Second).Should(Satisfy(notExceedsMaxBuildLimit))
limit++
}

// make sure we have verified max build limit at least once
if limit == 0 {
t.Error(errors.New(fmt.Sprintf("Unexpected build phase '%s' for %s - not able to verify max builds limit", BuildPhase(ns, buildA)(), buildA)))
t.FailNow()
}

// verify that all builds are successful
Eventually(BuildPhase(ns, buildA), TestTimeoutLong).Should(Equal(v1.BuildPhaseSucceeded))
Eventually(KitPhase(ns, buildA), TestTimeoutLong).Should(Equal(v1.IntegrationKitPhaseReady))
Eventually(BuildPhase(ns, buildB), TestTimeoutLong).Should(Equal(v1.BuildPhaseSucceeded))
Eventually(KitPhase(ns, buildB), TestTimeoutLong).Should(Equal(v1.IntegrationKitPhaseReady))
Eventually(BuildPhase(ns, buildC), TestTimeoutLong).Should(Equal(v1.BuildPhaseSucceeded))
Eventually(KitPhase(ns, buildC), TestTimeoutLong).Should(Equal(v1.IntegrationKitPhaseReady))
})
}

func TestKitTimerToLogFullBuild(t *testing.T) {
doKitFullBuild(t, "timer-to-log", "8m0s", TestTimeoutLong, kitOptions{
dependencies: []string{
Expand Down
26 changes: 26 additions & 0 deletions e2e/common/traits/builder_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,32 @@ func TestBuilderTrait(t *testing.T) {
Expect(Kamel("reset", "-n", ns).Execute()).To(Succeed())
})

t.Run("Run build order strategy dependencies", func(t *testing.T) {
name := "java-fifo-strategy"
Expect(KamelRunWithID(operatorID, ns, "files/Java.java",
"--name", name,
"-t", "builder.order-strategy=dependencies").Execute()).To(Succeed())

Eventually(IntegrationPodPhase(ns, name), TestTimeoutLong).Should(Equal(corev1.PodRunning))
Eventually(IntegrationConditionStatus(ns, name, v1.IntegrationConditionReady), TestTimeoutShort).Should(Equal(corev1.ConditionTrue))
Eventually(IntegrationLogs(ns, name), TestTimeoutShort).Should(ContainSubstring("Magicstring!"))

integrationKitName := IntegrationKit(ns, name)()
builderKitName := fmt.Sprintf("camel-k-%s-builder", integrationKitName)
Eventually(BuildConfig(ns, integrationKitName)().Strategy, TestTimeoutShort).Should(Equal(v1.BuildStrategyRoutine))
Eventually(BuildConfig(ns, integrationKitName)().OrderStrategy, TestTimeoutShort).Should(Equal(v1.BuildOrderStrategyDependencies))
// Default resource CPU Check
Eventually(BuildConfig(ns, integrationKitName)().RequestCPU, TestTimeoutShort).Should(Equal(""))
Eventually(BuildConfig(ns, integrationKitName)().LimitCPU, TestTimeoutShort).Should(Equal(""))
Eventually(BuildConfig(ns, integrationKitName)().RequestMemory, TestTimeoutShort).Should(Equal(""))
Eventually(BuildConfig(ns, integrationKitName)().LimitMemory, TestTimeoutShort).Should(Equal(""))

Eventually(BuilderPod(ns, builderKitName), TestTimeoutShort).Should(BeNil())

// We need to remove the kit as well
Expect(Kamel("reset", "-n", ns).Execute()).To(Succeed())
})

t.Run("Run build order strategy fifo", func(t *testing.T) {
name := "java-fifo-strategy"
Expect(KamelRunWithID(operatorID, ns, "files/Java.java",
Expand Down
Loading

0 comments on commit b851342

Please sign in to comment.