Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEV - Clean and update dependencies #792

Closed
wants to merge 28 commits into from
Closed
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8d1136e
:pushpin: Clean dev and runtime deps - server
trallard Mar 20, 2024
b85208e
:arrow_up: Bump pre-commit deps
trallard Mar 20, 2024
48ec413
:heavy_minus_sign: Remove runtime deps
trallard Mar 20, 2024
757677f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 20, 2024
d365061
Bump GH actions workflows
trallard Mar 20, 2024
cff7ed5
Merge branch 'trallard/clean-dependencies' of https://github.com/cond…
trallard Mar 20, 2024
e52de2f
Add pre-commit ci
trallard Mar 20, 2024
383236f
Update lint hatch script
trallard Mar 20, 2024
6a34329
Add updated intro docs
trallard Mar 20, 2024
f1dfe9a
Update hatch syntax
trallard Mar 20, 2024
d4c3b73
:construction_worker: Ensure we always install packages
trallard Mar 20, 2024
3f1b16e
:fire: Remove redundant env.yaml file
trallard Mar 20, 2024
b7ea23e
:fire: Remove yaml files
trallard Mar 20, 2024
e9b03a1
Update conda-store/pyproject.toml
trallard Mar 21, 2024
c8cdce9
Update pyproject.toml - isort and maintainers
trallard Mar 21, 2024
cf5df1e
:memo: Update contributing docs
trallard Mar 21, 2024
bcdb39a
:construction_worker: Pin BAIPP to v1
trallard Mar 21, 2024
390fc5f
More updates to GH actions
trallard Mar 21, 2024
69a49ca
:heavy_plus_sign: Add hatch to action
trallard Mar 21, 2024
4aa7d7b
Fix typo
trallard Mar 21, 2024
1431419
Fix artefacts name
trallard Mar 21, 2024
8fcf259
:construction_worker: Improve deps in CI
trallard Mar 22, 2024
967f11a
Merge branch 'trallard/clean-dependencies' of https://github.com/cond…
trallard Mar 22, 2024
696d61b
Pin build deps
trallard Mar 22, 2024
997faa2
Pin build deps
trallard Mar 22, 2024
afc2061
:arrow_up: Use baipp from main
trallard Mar 25, 2024
16bae5c
Fix twine version pin
trallard Mar 25, 2024
16911ad
Remove flower from dev
trallard Mar 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/build_docker_image.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout

env:
FORCE_COLOR: "1" # Make tools pretty.

jobs:
build_and_push_docker_image:
name: "Build Docker Images 🛠"
Expand All @@ -23,7 +26,7 @@ jobs:
uses: actions/checkout@v4

- name: "Retrieve secret from Vault 🗝"
uses: hashicorp/vault-action@v2
uses: hashicorp/vault-action@v3
with:
method: jwt
url: "https://quansight-vault-public-vault-b2379fa7.d415e30e.z1.hashicorp.cloud:8200"
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,10 @@ jobs:
- run: echo "Running on ${{ matrix.directory }}"

- name: "Check package build - ${{ matrix.directory }} 📦"
uses: hynek/build-and-inspect-python-package@v1
uses: hynek/build-and-inspect-python-package@main
with:
path: ${{ matrix.directory }}
upload-name-suffix: "-${{ matrix.directory }}"

# Adding a separate upload for publishing
- name: "Upload build artefacts 📤"
Expand Down
76 changes: 34 additions & 42 deletions .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: Tests

env:
DEFAULT_PYTHON_VERSION: "3.10"
FORCE_COLOR: "1" # Make tools pretty.

on:
pull_request:
Expand All @@ -20,13 +21,6 @@ jobs:
strategy:
matrix:
os: ["ubuntu", "macos", "windows"]
include:
- os: ubuntu
environment-file: conda-store-server/environment-dev.yaml
- os: macos
environment-file: conda-store-server/environment-macos-dev.yaml
- os: windows
environment-file: conda-store-server/environment-windows-dev.yaml
runs-on: ${{ matrix.os }}-latest
defaults:
run:
Expand All @@ -37,9 +31,9 @@ jobs:
uses: actions/checkout@v4

- name: "Set up env ${{ matrix.os }} 🐍"
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
environment-file: ${{ matrix.environment-file }}
environment-file: conda-store-server/environment-dev.yaml
miniforge-version: latest

# This fixes a "DLL not found" issue importing ctypes from the hatch env
Expand All @@ -54,15 +48,16 @@ jobs:

- name: "Linting Checks 🧹"
run: |
hatch env run -e dev lint
hatch env run -e lint lint
trallard marked this conversation as resolved.
Show resolved Hide resolved

- name: "Build package 📦"
- name: "Build and install conda-store-server 📦"
run: |
hatch build
python -Im pip install dist/*.whl

- name: "Unit tests ✅"
run: |
pytest -m "not extended_prefix and not user_journey" tests
python -Im pytest -m "not extended_prefix and not user_journey" tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change (and other like it) necessary now?

Copy link
Collaborator Author

@trallard trallard Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now everything is run with bare pytest.... or similar and we have multiple installations of Python as is. Adding the (I)m ensures we are using the actual executable we intend to.
I might end up removing this once all the dependencies and workflows are aligned but without isolating in the meantime it is hard to tell what is being called


# https://github.com/actions/runner-images/issues/1052
- name: "Windows extended prefix unit tests ✅"
Expand All @@ -76,7 +71,6 @@ jobs:
pytest -m "extended_prefix" tests
if: matrix.os == 'windows'


integration-test-conda-store-server:
name: "integration-test conda-store-server"
runs-on: ubuntu-latest
Expand All @@ -89,15 +83,15 @@ jobs:
uses: actions/checkout@v4

- name: "Set up env 🐍"
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
environment-file: conda-store-server/environment-dev.yaml
miniforge-version: latest

- name: "Install build dependencies 📦"
run: |
pip install hatch
sudo apt install wait-for-it -y
sudo apt update
sudo apt install -y wait-for-it

- name: "Deploy docker-compose"
run: |
Expand All @@ -110,24 +104,23 @@ jobs:

- name: "Run Playwright tests 🎭"
run: |
playwright install
pytest --video on ../tests/test_playwright.py
playwright install --with-deps chromium
python -m pytest --video on ../tests/test_playwright.py

- name: "Upload test results 📤"
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: playwright-tests
path: conda-store-server/test-results
path: conda-store-server/test-results/

- name: "Run integration tests ✅"
run: |
export PYTHONPATH=$PYTHONPATH:$PWD
pytest ../tests/test_api.py ../tests/test_metrics.py
python -m pytest ../tests/test_api.py ../tests/test_metrics.py

- name: "Run user journey tests ✅"
run: |
pytest -m "user_journey"
python -m pytest -m "user_journey"

- name: "Get Docker logs 🔍"
if: ${{ failure() }}
Expand All @@ -142,28 +135,26 @@ jobs:
working-directory: conda-store
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
steps:
- name: "Checkout Repository 🛎"
uses: actions/checkout@v4

- name: "Set up Python 🐍"
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}


- name: "Install Dependencies 📦"
run: |
pip install hatch
sudo apt install wait-for-it -y
python -Im pip install hatch
sudo apt update
sudo apt install -y wait-for-it

- name: "Linting Checks 🧹"
run: |
hatch env run -e dev lint

- name: "Build package 📦"
run: |
hatch build
hatch env run -e lint lint
trallard marked this conversation as resolved.
Show resolved Hide resolved

- name: "Deploy docker-compose"
run: |
Expand All @@ -174,16 +165,17 @@ jobs:
wait-for-it localhost:9000 # minio
wait-for-it localhost:8080 # conda-store-server

- name: "Install conda-store for tests 📦"
- name: "Build and install conda-store 📦"
run: |
pip install .
hatch build
python -Im pip install dist/*.whl

- name: "Run basic tests - not authenticated"
- name: "Run basic tests - not authenticated 🔓"
run: |
sleep 20
./tests/unauthenticated-tests.sh

- name: "Run basic tests - authenticated"
- name: "Run basic tests - authenticated 🔐"
run: |
./tests/authenticated-tests.sh

Expand All @@ -201,28 +193,28 @@ jobs:
docker-compose logs

build-docker-image:
name: "Build docker images"
name: "Build Docker images 🐋"
runs-on: ubuntu-latest
strategy:
matrix:
docker-image:
- conda-store
- conda-store-server
steps:
- name: "Checkout Repository"
- name: "Checkout Repository 🛎"
uses: actions/checkout@v4

- name: Set up Docker Buildx
- name: "Set up Docker Buildx"
uses: docker/setup-buildx-action@v3

- name: Lint Dockerfiles
- name: "Lint Dockerfiles 🔍"
uses: jbergstroem/hadolint-gh-action@v1
with:
dockerfile: ${{ matrix.docker-image }}/Dockerfile
output_format: tty
error_level: 0

- name: Docker Meta
- name: "Docker Meta"
id: meta
uses: docker/metadata-action@v5
with:
Expand All @@ -231,7 +223,7 @@ jobs:
tags: |
type=sha

- name: Build docker
- name: "Build Docker image 🐋"
uses: docker/build-push-action@v5
with:
context: "${{ matrix.docker-image }}"
Expand Down
26 changes: 10 additions & 16 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,33 +8,27 @@
# - Register git hooks: pre-commit install --install-hooks
# - Run on all files: pre-commit run --all-files

ci:
autoupdate_schedule: monthly
autofix_commit_msg: |
[pre-commit.ci] Apply automatic pre-commit fixes

repos:
- repo: https://github.com/psf/black
rev: 23.9.1
rev: 24.3.0
hooks:
- id: black
exclude: "examples|tests/assets"

- repo: https://github.com/charliermarsh/ruff-pre-commit
# Ruff version.
rev: "v0.0.289"
rev: "v0.3.3"
hooks:
- id: ruff
exclude: "examples|tests/assets"
args: ["--fix"]

- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
name: isort
additional_dependencies: [toml]
files: \.py$
args: ["--profile", "black"]
exclude: "conda-store-server/conda_store_server/action/__init__.py"
args: [--fix, --exit-non-zero-on-fix]

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
rev: v4.5.0
hooks:
- id: trailing-whitespace
files: ".*\\.py"
Expand All @@ -44,7 +38,7 @@ repos:
- id: trailing-whitespace

- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.0.3
rev: v4.0.0-alpha.8
hooks:
- id: prettier
exclude: ^(examples/|templates/|)
53 changes: 23 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,53 +10,46 @@ Flexible. Reproducible. Governable.

---

<div align="center">

<a href="https://pypi.org/project/conda-store-server/">
<img src="https://img.shields.io/pypi/v/conda-store-server?label=pypi|conda-store-server" alt="PyPi"/>
</a>

<a href="https://pypi.org/project/conda-store/">
<img src="https://img.shields.io/pypi/v/conda-store-server?label=pypi|conda-store" alt="PyPi"/>
</a>

<a href="https://anaconda.org/conda-forge/conda-store-server">
<img src="https://img.shields.io/conda/vn/conda-forge/conda-store-server?color=green&label=conda-forge%7Cconda-store-server" alt="Conda"/>
</a>

<a href="https://anaconda.org/conda-forge/conda-store">
<img src="https://img.shields.io/conda/vn/conda-forge/conda-store?color=green&label=conda-forge%7Cconda-store" alt="Conda"/>
</a>

</div>
| Information | Links|
| :---------- | :-----|
| Project | [![License](https://img.shields.io/badge/License-BSD%203--Clause-gray.svg?colorA=2b2d42&colorB=206532&style=flat.svg)](https://opensource.org/licenses/BSD-3-Clause) [![Read the docs](https://img.shields.io/badge/%F0%9F%93%96%20Read-the%20docs-gray.svg?colorA=2b2d42&colorB=206532&style=flat.svg)](https://conda.store)|
|Community | [![Open an issue](https://img.shields.io/badge/%F0%9F%93%9D%20Open-an%20issue-gray.svg?colorA=2b2d42&colorB=206532&style=flat.svg)](https://github.com/conda-incubator/conda-store/issues/new/choose) [![Community guidelines](https://img.shields.io/badge/🤝%20Community-guidelines-gray.svg?colorA=2b2d42&colorB=206532&style=flat.svg)](https://conda.store/community/introduction)|
|Releases | [![PyPI release conda-store](https://img.shields.io/pypi/v/conda-store)](https://badge.fury.io/py/conda-store?label=pypi%20conda-store&style=flat.svg) [![PyPI release conda-store-server](https://img.shields.io/pypi/v/conda-store-server?label=pypi%20conda-store-server)](https://badge.fury.io/py/conda-store-server) [![conda-forge release conda-store](https://img.shields.io/conda/vn/conda-forge/conda-store?label=conda-forge%20conda-store)]((https://anaconda.org/conda-forge/conda-store)) [![conda-forge release conda-store-server](https://img.shields.io/conda/vn/conda-forge/conda-store-server?label=conda-forge%20conda-store-server)]((https://anaconda.org/conda-forge/conda-store-server)) |

---

conda-store provides the familiarity and flexibility of conda environments, without compromising reliability for collaborative settings.
conda-store provides the familiarity and flexibility of conda environments, without compromising reliability for
collaborative settings.

conda-store is built to work for all team members from individual data scientists to administrators, while making sure your team follows best practices throughout the environment life cycle: from initial environment creation, to using environments in a production machine.
conda-store is built to work for all team members from individual data scientists to administrators,
while making sure your team follows best practices throughout the environment life cycle:
from initial environment creation to using environments in a production machine.

## Key features

- **Flexiblity**:
- **Flexibility**:
- Users can create and update environments with the Graphical UI or a YAML editor.
- The environments are automatically version-controlled and all versions are readily available.
- **Reproduciblity**:
- User can share environments quickly through the auto-generated artifacts including a lockfile, docker image, YAML file, and tarball.
- **Reproducibility**:
- Users can share environments quickly through the auto-generated artifacts including a lockfile, docker image, YAML file, and tarball.
- conda-store pins exact versions of all packages and their dependencies in all the auto-generated artifacts.
- **Goverance**:
- Users have access to admin-approved packages and channels for their work, and can request new ones when needed.
- **Governance**:
- Users have access to admin-approved packages and channels for their work and can request new ones when needed.
- Admins can insert or require certain packages and versions for organization-level compatibility.
- Admins can manage users' access-levels using "Namespaces", and allow users to share environments across (and only with) their team.
- Admins can manage users' access levels using "Namespaces", and allow users to share environments across (and only with) their team.

## Get started

Learn more, including how to install, use, and contribute to conda-store in our documentation at [**conda.store**](https://conda.store/).

## Related repositories
## Contributing

We welcome all types of contributions. Please read our [Contributing Guide](https://conda.store/community/contribute/) to get started.

## Related repositories and projects

- We are working on a new UI for conda-store at: [`conda-incubator/conda-store-ui`](https://github.com/conda-incubator/conda-store-ui), and
- a JupyterLab extension at: [`conda-incubator/jupyterlab-conda-store`](https://github.com/conda-incubator/jupyterlab-conda-store).
- We are working on a new UI for conda-store at [`conda-incubator/conda-store-ui`](https://github.com/conda-incubator/conda-store-ui) and
- a JupyterLab extension at [`conda-incubator/jupyterlab-conda-store`](https://github.com/conda-incubator/jupyterlab-conda-store).

## Code of Conduct

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
Create Date: 2024-01-29 03:56:36.889909

"""

import sqlalchemy as sa
from alembic import op

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
Create Date: 2023-12-13 21:01:45.546591

"""

import sqlalchemy as sa
from alembic import op

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
Create Date: 2022-08-24 12:01:48.461989

"""

import sqlalchemy as sa
from alembic import op

Expand Down
Loading
Loading