Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update changelog - 29-December-2021 - 16:41 CET #8587

Merged
merged 1 commit into from
Jan 2, 2022

Conversation

jgsogo
Copy link
Contributor

@jgsogo jgsogo commented Dec 30, 2021

29-December-2021 - 16:41 CET

  • [config] Add Linux GCC 11, Clang 12 & 13 configurations with new epoch feature.
  • [config] Drop Linux GCC 6 and clang 10 configurations.
  • [fix] Fixed some unexpected errors.
  • [feature] Prepare pipeline to work with ansible provisioned machines.

In this deployment, we are introducing a new feature that will help us to evolve ConanCenter. We've called it epochs and I think it deserves some explanation here.

The problem

Whenever we want to add a new compiler version we need to follow the dependency-graph order, so all the binary packages from the requirements are already available when trying to build some consumer library.

In the past we used to generate these new configurations behind the scenes. After any successful build of a PR, we triggered a pipeline in the backstage to generate those configurations, but we didn't report its state. Contributors shouldn't be bothered by these new compilers, otherwise they would need to create a full chain of PRs just to make their PR happy.

Before

To help with that process, we forced the generation of binaries using some batch processes behind the scenes (and manual annotations) and fixed recipes if needed. When this has happened, you have seen many PRs from our side fixing those configurations (remember the journey to generate M1 binaries)

image

This approach was good enough, but it had one major drawback... as these new configurations were not enforced in PRs, any contributor could unexpectedly break some recipe that was already fixed, and break all the consumers (missing binaries). So we needed to run and check all the graph in order again and again.

Epochs

With epochs, new compilers are listed in a new epoch. Every time we are about to build a PR we check the epoch of all the requirements and select the newest one. So, if all the requirements are already in a new epoch, the PR will use that new epoch and will try to build the new configurations.

With this process we achieve two objectives:

  • PRs will be tested for new compilers if the requirements already have their binaries available. Contributors will get feedback about them and their PR will fail if those new configurations fail.
  • Once a reference has been moved to a new epoch, it never goes back. Once we have fixed a recipe for some new configuration, it will always be validated in the following PRs, so there won't be steps back.

This is completely transparent to users. You will get the same notifications in your PR. Eventually of the references will be moved to new epochs just by creating regular PRs. Of course there is no magic here, those PRs need to happen, and we will need to force some compilations behind the scenes to move existing libraries to new epochs, but at least

Once a reference has been moved to a new epoch, it never goes back. Once we have fixed a recipe for some new configuration, it will always be validated in the following PRs, so there won't be steps back.

This is very important to us, as it removes the need for those crunches when we wanted to add new compilers. Now we can add new versions at any time and generation of binaries is decoupled in time.


Random FAQ:

  • Can I know which epoch is being used in my PR?

    Not directly, but in the report you can check the compilers and versions that are being built. Newer epochs always contain more compilers/versions.

  • Can I know which requirement is blocking other consumers to move to a new epoch?

    Not yet, but we are thinking about how to provide this information (maybe something like [Conan v1] Recipe Export Check #2232), because we are sure that you will help us to move those reference forward.

  • Can I know the compilers and version that are used for each epoch?

    We really need to improve the output from this automation: [doc] Update supported platforms and configurations (2021-12-29) #8579


As always, feedback is more than welcome. And feel free to ask anything about this new feature... or to suggest other ideas around it. We are building together a game-changer for C++ developers. You, contributors, rock! 😊

As a final note, special thanks to the people that are working on these features (and not only these), developing, testing and coming up with the ideas: @danimtb , @SSE4 and @uilianries .

And Happy New Year!! 🎉 🎉 (this should be the last release of 2021 🤞 )

@conan-center-bot
Copy link
Collaborator

Updating docs!

Copy link
Member

@uilianries uilianries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice explanation @jgsogo !!

@prince-chrismc
Copy link
Contributor

prince-chrismc commented Jan 2, 2022

Every time we are about to build a PR we check the epoch of all the requirements and select the newest one. So, if all the requirements are already in a new epoch, the PR will use that new epoch and will try to build the new configurations.

If it fails will it default to the old epoch that was successful last time?

Or it is status quo, where new PRs need to pass new requirements? If yes, let's say that new PR just disabled the new compiler(s)... will it still get the the new epoch?

@conan-center-bot conan-center-bot merged commit ab61aa7 into conan-io:master Jan 2, 2022
@danimtb
Copy link
Member

danimtb commented Jan 10, 2022

If it fails will it default to the old epoch that was successful last time?

No, it will try to build always the latest epoch possible

Or it is status quo, where new PRs need to pass new requirements? If yes, let's say that new PR just disabled the new compiler(s)... will it still get the the new epoch?

Yes, even with new configurations disabled in the recipe, it will get the new epoch when built

@jgsogo jgsogo deleted the doc/changelog branch January 10, 2022 12:14
@prince-chrismc
Copy link
Contributor

Thanks for explaining!

@jgsogo jgsogo mentioned this pull request Jan 19, 2022
4 tasks
miklelappo pushed a commit to miklelappo/conan-center-index that referenced this pull request Feb 9, 2022
@SSE4 SSE4 mentioned this pull request Mar 9, 2022
4 tasks
@SSE4
Copy link
Contributor

SSE4 commented Mar 9, 2022

I think we need to extract it to doc, as newcomers ask how to add e.g. GCC11 binaries, but it's explained nowhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants