Create "7.16" branch to support backporting process

[jlind23]

The goal of this issue is to support the backporting process of packages. There will be two branches: main and "7.16". Developers will keep pushing changes to the main branch and automation will backport those PRs which affect 7.16 stack.
If a developer pushes a package with Kibana version constraint = ^8.0.0, it won't be backported.

We will try to postpone the decision about creating the "7.16" branch and related CI pipeline as much as possible as it causes additional operational load on the team. We'll focus case by case on packages potentially introducing breaking changes and suggest introducing "compatibility" ingest pipelines.

---

Previous description:

v8.0 is ongoing and integrations will have to be compatible.
As 7.16 will be a LTS version they will both live alonside.

8.X will contain some breaking changes for integrations as mentioned here:
elastic/beats#28215

In order to be able to fix bugs on 7.X release one possible way is to:

• Create a 7.16 branch in elastic/integrations repository
• Create specific build pipelines for 7.16 branch
• Update master branch to 8.X

[mtojek]

I can take care of this one, it's more like replicating current behavior from master, enabling the automation, documenting the workflow. I suppose I will need some help from the @elastic/observablt-robots team to configure Jenkins pipeline, dependabot, any permissions, etc.

As it's the LTS: do you think we should introduce/support backporting process?

cc @jsoriano - sorry for mentioning, but this is also kind of "discuss" issue.

[v1v]

Question:

• Do you want to support 7.x and 7.<minor> (where <minor> represents the minor version for instance 7.16)? Then this could help to be flexible if 7.17 or another version becomes available. Then, the backport might behave similar to what the Beats, APM-Server and other ElasticStack repositories work.
• Backport automation could be done similar to the fleet-server -> https://github.com/elastic/fleet-server/blob/master/.mergify.yml (they use the v7.minor.patch labels rather than backport-xxx labels.

Sure, please feel free to open an issue in our private github repo then we can plan this :)

[mtojek]

My assumption is as follows (and we can discuss it during team meeting):

1. Branch from master and call it 7.16.
2. Backport every PR pushed to master if the kibana.constraint in manifest is valid for 7.16 or this is a non-package related file.
3. Automation handles PR merging (auto-approve).
4. Developers contribute to 7.16 if only they have backfixes.

We don't want to have any branches like 7.14, 7.15, 7.x. Just one for 7.16, which is LTS.

[MichaelKatsoulis]

@mtojek I am also relying on this implementation. I have a breaking change for Kubernetes Integration (#1792) intended for 8.0. If I merge it now to master it seems that it will block the development of 7.16. So I will wait for your implementation!

[mtojek]

We had an offline conversation with @jsoriano about this and noticed that there is a difference in Kibana logic based on the branch:

7.x

• Kibana calls Package Registry's Search API and passes kibana.constraint:
  /search?package=fleet_server&internal=true&experimental=true&kibana.version=7.16.0

8.0 / main branch

• Kibana searches for all packages (no constraint is defined):
  /search?package=fleet_server&internal=true&experimental=true

[jlind23]

How will it be displayed in the UI? Does it mean that Kibana will show multiple time the same package if it has different versions?

[mtojek]

How will it be displayed in the UI? Does it mean that Kibana will show multiple time the same package if it has different versions?

It will show only the newest one, but there is a risk that it will show also incompatible packages. Ideally, we should release all packages with kibana.constraint ^8.0.0 and Kibana should obey it.

[ruflin]

• Make default_field: false the default for v8.0 beats#28215 should not have any effect on integrations
• For breaking changes in a package, what is our upgrade story? In Update kubernetes namespace field. breaking change #1792 I see only adjustments to field mappings but no update to the ingest pipeline. Maybe we can use this change to discuss our compatibility story.
• For compatiblity, lets make sure we never use >= as it means it is compatible with all future majors which is likely not correct. @mtojek Maybe we could even add validation to not allow it?

[mtojek]

For breaking changes in a package, what is our upgrade story? In Update kubernetes namespace field. breaking change #1792 I see only adjustments to field mappings but no update to the ingest pipeline. Maybe we can use this change to discuss our compatibility story.

(we had an offline discussion about it with @jsoriano)

If we agree on adjustments in ingest pipelines (rewrite fields, remove deprecated fields, convert types), then we may not need to create the branch immediately. So far we survived without any troubles without backporting in place. @ruflin WDYT?

For compatiblity, lets make sure we never use >= as it means it is compatible with all future majors which is likely not correct. @mtojek Maybe we could even add validation to not allow it?

I don't see any objections to this, it seems to be a good idea to prevent it in general. But it also means that we have to adjust ALL packages before major releases (additional operational duty).

[ruflin]

• If we don't need the branch immediately I'm more then happy :-)
• Update all packages: Yes, that is unfortunate. But also major release are "rare". I hope by the time we get to the next major release we have a better solution. BTW on the compatibility we can always use something like ^7.16.0 AND ~8.0.0. But yes, that means for 8.1 we need to update packages again.

[mtojek]

Summary:

We decided to postpone the decision about the pipeline until we really need this one. Let's try to develop and release packages without a sophisticated backporting process and just involve compatibility pipelines.