Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only pin version of upstream packages when needed #1530

Closed
gdemonet opened this issue Aug 19, 2019 · 1 comment
Closed

Only pin version of upstream packages when needed #1530

gdemonet opened this issue Aug 19, 2019 · 1 comment
Labels
kind:dependencies Pull requests that update a dependency file kind:enhancement New feature or request topic:build Anything related to building steps

Comments

@gdemonet
Copy link
Contributor

Component: build

Why this is needed:

With #1477, all packages installed explicitly by Salt (e.g. through some pkg.installed <name>) were pinned to the effectively installed version. For most of them, this meant the latest version available upstream (since we download, and not build, packages for the most part).

Pinning so many packages will eventually become cumbersome to maintain. A lot of manual efforts are required for no gains whatsoever.

What should be done:

We want to remove the necessity of providing version information for desired packages, in buildchain.versions. One should be able to add a Package(name=name, version=None, release=None), or find an equivalent way to manage such occurences.

For such packages, we should let the target system's package manager download the latest version it has available, and then store that version information into the generated <ISO>/salt/metalk8s/versions.json file.

Even if upstream packages are not pinned for a given MetalK8s version, we want to pin the version of those packages once shipped in offline repositories (see #666). This is necessary to handle downgrades, most notably: pkg.installed kubelet will not downgrade kubelet even if the only version available in the selected repositories is lower than the one currently installed.

Implementation proposal (strongly recommended):

  • Handle Package(name, None, None) construct or equivalent to mark an unpinned package version
  • Download packages from upstream with or without version information indistinctly
  • Retrieve this version information from downloaded package and include that information in the generated Salt tree

Test plan:

  • Check that unpinned packages are still referenced in the generated salt/metalk8s/versions.json file
  • Check that a new version released for some upstream package, unpinned on our side, gets updated only for a clean build
@gdemonet gdemonet added kind:enhancement New feature or request moonshot topic:build Anything related to building steps kind:dependencies Pull requests that update a dependency file labels Aug 19, 2019
NicolasT added a commit that referenced this issue Sep 18, 2019
@NicolasT
buildchain: bump package versions to CentOS 7.7
616583b 
See: #1530
@NicolasT NicolasT mentioned this issue Sep 18, 2019
Merged
NicolasT added a commit that referenced this issue Sep 18, 2019
@NicolasT
buildchain: bump package versions to CentOS 7.7
74c0a52 
See: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
packaging: refactor download list generation
0e4f266 
Issue: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
packaging: refactor build list generation
e17c529 
Issue: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
packaging: reorder definitions and add folding
f4f3aea 
Issue: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
versions: introduce a PackageVersion class
e4db7dc 
This allows us to control package listings more finely, and also
provides helpers for version and fully versioned package name
serialization to strings.

Issue: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
versions: introduce common packages and extras/overrides
34b4f9e 
The support of an additional OS requires a 'compatibility' layer for
package names and versions.

We define a common core of packages whose names do not diverge across OS
families and whose versions are either latest or fixed across OS families.

We introduce overrides in package versions, such that a package for a specific
version may override the package referenced in the common core.

We unpin most packages in the RPM listing, where we only really need to pin
K8s core components as well as some of the 'unstable' Python APIs we rely
on.

Issue: #1530
jbertran added a commit that referenced this issue Sep 18, 2019
@jbertran
versions: switch to Mapping type for built packages return value
946a56b 
The `Dict` type is too strict and causes mypy to expect Package type
where we should expect Package _or_ any subclass of Package type.

Issue: #1530
@jbertran jbertran mentioned this issue Sep 18, 2019
Merged
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
packaging: refactor download list generation
daaf76b 
Cherry-picked from cff6b58

Issue: #1530
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
packaging: refactor build list generation
ef44815 
Cherry-picked from 5db249c

Issue: #1530
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
packaging: reorder definitions and add folding
c6cb214 
Cherry-picked from faa8f8d

Issue: #1530
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
versions: introduce a PackageVersion class
e8f9286 
This allows us to control package listings more finely, and also
provides helpers for version and fully versioned package name
serialization to strings.

Cherry-picked from 81e5493

Issue: #1530
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
versions: introduce common packages and extras/overrides
63660d8 
The support of an additional OS requires a 'compatibility' layer for
package names and versions.

We define a common core of packages whose names do not diverge across OS
families and whose versions are either latest or fixed across OS families.

We introduce overrides in package versions, such that a package for a specific
version may override the package referenced in the common core.

We unpin most packages in the RPM listing, where we only really need to pin
K8s core components as well as some of the 'unstable' Python APIs we rely
on.

Cherry-picked from eb2afe7

Issue: #1530
jbertran added a commit that referenced this issue Sep 19, 2019
@jbertran
versions: switch to Mapping type for built packages return value
4435ab4 
The `Dict` type is too strict and causes mypy to expect Package type
where we should expect Package _or_ any subclass of Package type.

Cherry-picked from 4723b9a

Issue: #1530
@gdemonet gdemonet mentioned this issue Oct 11, 2019
Closed
@gdemonet
Copy link
Contributor Author

Done in #1717

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:dependencies Pull requests that update a dependency file kind:enhancement New feature or request topic:build Anything related to building steps
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gdemonet