Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Improve PyPi publishing tool #6542

Merged
merged 8 commits into from
Jul 1, 2024
Merged

[Feature] Improve PyPi publishing tool #6542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c678546
Update PUBLISH.md
montezdesousa Jun 25, 2024
355475c
Refactor publish.py
montezdesousa Jun 27, 2024
277b491
fix ruff check
montezdesousa Jun 28, 2024
f3e121b
final publish.py
montezdesousa Jun 28, 2024
2264659
Merge 'develo' into 'feature/update-publish-md'
montezdesousa Jun 28, 2024
eb1d7a1
feat: add option to change semantic version with poetry
montezdesousa Jul 1, 2024
da1f2f3
Merge branch 'develop' into feature/update-publish-md
montezdesousa Jul 1, 2024
39894e9
replace with publish --build cmd
montezdesousa Jul 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 30 additions & 60 deletions build/pypi/openbb_platform/PUBLISH.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,90 +1,60 @@
# Publishing to PyPI

## Pre-release procedure
## Pre-release

> [!WARNING]
> The `release` branch is solely for the purpose of publishing the package(s)! Any last minute changes should be made in appropriate PRs and merged to the `develop` branch. Once the `release` branch is created, the `develop` branch should be frozen for further commits.
> The `release` branch is solely for the purpose of publishing the package(s)! Any last minute changes should be made in appropriate PRs and merged to the `develop` branch. Once the `release` branch is created, the `develop` branch should be frozen for further commits until this process finishes.

> A `release` branch for a particular package should only cater to the changes for that package directory. For e.g. the `release/openbb-core-2.0.0` branch should only contain changes for the `openbb-core` package i.e. in the `openbb_platform/core` directory.

### Flow to display only files changes during release cycle in the `release/...` -> `main` PR

1. Merge main into develop before release branch exists, solve any conflict
2. In the branch `release/…` (before or after publishing)
3. `git merge main -X ours` (ignore the changes, we brought them in 1.)
4. `git commit -m "Merge branch 'main' into release/4.2.2"` (empty commit)
0. Merge `main` into `develop` and solve any conflict. You can do this by checking out `develop`, creating a branch `feature/merge-main-into-develop` and running `git merge main`. Merge this into `develop` if there are changes to commit.
1. Open a PR with the changes to be published in the format `release/<version>` (for e.g. `release/4.0.0` ). For a particular package use the format `release/<package>-<version>` (for e.g. `release/openbb-core-1.0.1`).
2. Bump `openbb` package version in `openbb_platform/pyproject.toml`.

### Pre-release checklist
> [!WARNING]
> The version must be incremented before running the tests in 4. and 5., since there are tests that rely of this to check for deprecated endpoints.

1. Open a PR with the changes to be published in the format `release/<version>` (for e.g. `release/4.0.0` ). For a particular package use the format `release/<package>-<version>` (for e.g. `release/openbb-core-1.0.1`).
2. Ensure all the CI workflows pass.
3. Ensure all unit tests pass: `pytest openbb_platform -m "not integration"`
4. Ensure all integration tests pass: `pytest openbb_platform -m integration`
3. Ensure all the CI workflows pass.
4. Ensure all unit tests pass: `pytest openbb_platform -m "not integration"`
5. Ensure all integration tests pass: `pytest openbb_platform -m integration`

## Release procedure
## Release

> Ensure you have the appropriate credentials and permissions to publish to PyPI.

1. Run the following commands for publishing the packages to PyPI:

Consider using the `--dry-run` flag to check if everything is correct before publishing.
Also, it might be a good idea to run the script in batches to ensure that the packages are published correctly and the dependencies pick the correct versions.

> For a single package release, the following steps are optional since the package can be bumped manually.

1. For the core package run: `python build/pypi/openbb_platform/publish.py --core`
2. For the extension packages run: `python build/pypi/openbb_platform/publish.py --extensions`
3. For the `openbb` package - **which requires manual publishing** - do the following

3.1. Bump the `openbb` package version and the extension versions on `openbb_platform/pyproject.toml` to the latest version.

> [!TIP]
> Consider using the poetry plugin `up` for updating the extensions to the latest version:
> 1. `poetry self add poetry-plugin-up`
> 2. `poetry up --latest`

> [!WARNING]
> Create a new environment before proceeding.
> Make sure that only required extensions are installed
> [!TIP]
> Consider using the `--dry-run` flag to check if everything is correct before publishing.

3.2. Run `pip install -e .` from `openbb_platform`
> For a single package release (e.g. patching `openbb-core`), the following steps are not required since the package can be published manually.

3.3. Re-build the static assets that are bundled with the package: `python -c "import openbb; openbb.build()"`
- Run `python -c "import openbb"` after building the static to check that no additional static is being built.
- Run any command to smoke test if the static assets are being built correctly.
1. To publish `openbb-core` run: `python build/pypi/openbb_platform/publish.py --core`
2. To publish **ALL** extensions run: `python build/pypi/openbb_platform/publish.py --extensions`
3. To publish `openbb` (the main package) run: `python build/pypi/openbb_platform/publish.py --openbb`
> [!TIP]
> Note that, in order for packages to pick up the latest versions of dependencies, it is advised to clear the local cache of the dependencies: we can do that with `pip cache purge` and `poetry cache clear pypi --all`. Sometimes there might be some delay in the PyPI API, so it might be necessary to wait a few minutes for pip to pick the latest versions.

3.4. Run unit tests to validate the existence of deprecated endpoints (or watch this through GitHub Actions)
2. Publish the CLI

3.5. Run `poetry publish --build` from `openbb_platform`
2.1. Bump `openbb` dependency on `cli/pyproject.toml` to the latest version

3.6. Run `poetry lock` from `openbb_platform`
2.2. Run `poetry publish --build` from `cli`

> [!TIP]
> Note that, in order for packages to pick up the latest versions of dependencies, it is advised to clear the local cache of the dependencies:
>
> We can do that with `pip cache purge` and `poetry cache clear pypi --all`
>
> Also, sometimes there might be some delay in the PyPI API, so it might be necessary to wait a few minutes before publishing the next package.
3. Regenerate assets for external use by running `python assets/scripts/generate_extension_data.py`.
4. Merge `release/<package>-<version>` to the `main` branch.
5. Run `Deploy to GitHub Pages` action in [openbb-docs](https://github.com/OpenBB-finance/openbb-docs/actions) to update the documentation website. Go to [docs.openbb.co](https://docs.openbb.co) to see the changes.

2. Merge the `release/<package>-<version>` branch to the `main` branch.
3. Run the `Deploy to GitHub Pages` action in [openbb-docs](https://github.com/OpenBB-finance/openbb-docs/actions). Go to the [docs](https://docs.openbb.co) website to see the changes.

## Post-release procedure
## Post-release

1. Install the packages on Google Colaboratory via PyPi and test to check if everything is working as expected.
2. Install the packages in a new environment locally via PyPi and test to check if everything is working as expected.
3. Regenerate assets for external use by running `python assets/scripts/generate_extension_data.py`
4. Open a new PR with the `release/<package>-<version>` branch pointing to the `develop` branch.
5. Merge the `release/<package>-<version>` branch to the `develop` branch.
6. If any bugs are encountered, create a new branch - `hotfix` for `main` and `bugfix` for `develop` and merge them accordingly.
3. Open a new PR with the `release/<package>-<version>` branch pointing to the `develop` branch.
4. Merge the `release/<package>-<version>` branch to the `develop` branch.
5. If any bugs are encountered, create a new branch - `hotfix` for `main` and `bugfix` for `develop` and merge them accordingly.

### Generate the changelog

1. Run the changelog automation by using the "release_drafter" GA and passing the number of the previous Release.
1. Run the changelog automation by using the "release_drafter" GitHub action and input the number of the last release PR (not this one!).
2. Edit and make the changelog live on the repository.
3. Paste it in the platform-release-changelog Slack channel.

### Publish the CLI

1. Bump `openbb` dependency on `cli/pyproject.toml` to the latest version.
2. Run `poetry publish --build` from `cli`
Loading
Loading