Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to a pip-tools workflow #67

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,49 @@ jobs:
- checkout

- restore_cache:
key: cache-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
- run: pipenv install --deploy --dev
key: cache-{{ .Branch }}-{{ checksum "requirements.txt" }}
- run:
name: Install dependencies
command: |
python3 -m venv venv
RealOrangeOne marked this conversation as resolved.
Show resolved Hide resolved
. venv/bin/activate
pip install -r requirements.txt
- save_cache:
key: cache-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
key: cache-{{ .Branch }}-{{ checksum "requirements.txt" }}
paths:
- "/home/circleci/.local/share/virtualenvs"
- run: pipenv run build
- run: pipenv run lint-yaml
- run:
name: Build
command: |
. venv/bin/activate
./scripts/build
- run:
name: Lint Yaml
command: |
. venv/bin/activate
./scripts/lint-yaml
- run:
name: Ensure requirements.txt and requirements.in are in sync
command: |
. venv/bin/activate
pip-compile --quiet
git diff --exit-code

deploy:
docker:
- image: circleci/python
- image: circleci/python:3
steps:
- add_ssh_keys:
fingerprints:
- "9e:a9:7b:98:8d:0a:69:22:29:eb:34:14:90:73:a2:5a"
- checkout
- restore_cache:
key: cache-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
key: cache-{{ .Branch }}-{{ checksum "requirements.txt" }}
- deploy:
name: Deploy to Github Pages
command: ./.circleci/deploy.sh
command: |
. venv/bin/activate
./.circleci/deploy.sh


workflows:
Expand Down
2 changes: 1 addition & 1 deletion .circleci/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ git config --global push.default simple
git config --global user.email $(git --no-pager show -s --format='%ae' HEAD)
git config --global user.name $CIRCLE_USERNAME

pipenv run mkdocs gh-deploy --verbose -m "Deploy build $CIRCLE_BUILD_NUM - $CIRCLE_SHA1 [ci skip]"
mkdocs gh-deploy --verbose -m "Deploy build $CIRCLE_BUILD_NUM - $CIRCLE_SHA1 [ci skip]"
23 changes: 0 additions & 23 deletions Pipfile

This file was deleted.

195 changes: 0 additions & 195 deletions Pipfile.lock

This file was deleted.

20 changes: 15 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,27 @@ Student Robotics Competition Program

- Python (`>=3.5`)
- `pip`
- `pipenv` (`pip install pipenv`)

## Usage

### Installing dependencies

```
pipenv install --dev
pip install -r requirements.txt
```

### Running development server

```
pipenv run serve
mkdocs serve # or ./scripts/serve
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the commands are this short, I think it's ok to omit the scripts. The scripts are also not windows compatible (sans WSL), which it would be nice to maintain 1st-party support for.

Also, currently the scripts imply that the virtualenv has been activated, which isn't explicitly documented here, so we should probably document that!

Copy link
Member Author

@PeterJCLaw PeterJCLaw Sep 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there was tab completion on mkdocs I'd agree about omitting the scripts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually wondering about including windows implementations for the scripts, so that if we happen to change the build setup again we could do so without needing to change what contributors need to run.

I've mixed feelings about documenting that the virtualenv needs to be active. We weren't doing do before and you don't seem to be advocating to do so for running the mkdocs commands directly. I'm therefore not sure why the scripts should be treated any differently.

In any of these cases (including the previous setup) the reader still needed to manage their environment themselves to ensure that the command being run was available. Nothing has changed in that regard.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't seem to be advocating to do so for running the mkdocs commands directly
Sorry, I was definitely suggesting we do this. They were separate suggestion points.

Including windows scripts could also work, but it's a lot of extra scripting. Having the scripts auto-source the virtualenv would definitely make them more useful in my eyes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the scripts auto-source the virtualenv would definitely make them more useful in my eyes.

How do you propose that they do that? They won't know where the virtualenv is, nor even if one is needed. Setup of the virtualenv is up to the developer at the moment (this has not changed).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire thread feels like we are reimplementing pipenv using an less standard tool. I'd personally rather we used a PEP 518 compliant tool, rather than yet another non-standard one.

This is motivated by the issues around pipenv always upgrading every dependency whenever you touch any dependency.

Please could you give some specific examples of where this has caused an issue with this repo? I dislike pipenv, but it is probably the best tool for this project.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

studentrobotics.slack.com/archives/CEG05AAG6/p1567264146004600

Okay, I agree that this is an issue.

However, I still don't think that we should be working around the limitations of pip-tools by hacking virtualenv activation into scripts. If that is a required feature of the toolset, I suggest we use a tool that does it. Alternatively, we could use pip-tools and provide instructions on how to use a virtualenv / pip-tools. A link to the appropriate docs is sufficient imo.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the scripts depend on pipenv, which handles the virtualenv stuff for the user. I think it feels fine that the new scripts also handle it.

Well, sort-of. In reality pipenv isn't going to be installed for most people by default, either in their host system or within a virtualenv they create. As a result, they end up needing to either install it in this virtualenv (which means they need to activate this virtualenv first anyway) or learn the mastery of having it in another virtualenv without that virtualenv being fully active when they run things (yet having pipenv on PATH). The former of these has pipenv just be a complication over the regular python tools (i.e: pip) and the latter requires substantial Python & system admin-fu which it seems unreasonable to expect from casual contributors.

If we want to take the opinion that this repo wants to control the virtualenv which it runs in, then we should provide scripts and documentation around creating that, rather than assuming that the user knows how to do so. Until now, it hasn't done that. (If we did do this I would argue we should still allow the user to have their environment somewhere else if they wanted, probably by having our scripts detect an existing env is active, so that people can use things like virtualenvwrapper if they want)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This thread has gone to quite a tangent now, and isn't especially constructive to the PR.

I'd not noticed the documented setup instructions don't mention a virtualenv. I can live with the scripts staying there, and as the platform-agnostic commands are documented first, that should be fine.

I'd be interested in simply documenting that virtualenvs exist, and are considered best practice, so volunteers don't pollute their global environment.

```

This will launch a server on http://localhost:8000. Content in the server will live-reload as changes are made. If large refactors of the site structure are made, it's advisable to stop the server, make the changes, then restart it.

### Production build

```
pipenv run build
mkdocs build # or ./scripts/build
```

This will build the site once, and place it in `site/` in the root of the project. This may be useful to see which files are rendered, and how exactly to access them.
Expand All @@ -41,9 +40,20 @@ This will build the site once, and place it in `site/` in the root of the projec
There are some rudimentary tests in the project. These are all run by the CI, and must pass before deployment.

```
pipenv run lint-yaml # Checks the yaml content is well formatted
./scripts/lint-yaml # Checks the yaml content is well formatted
```

## Dependency management

Add any new dependencies to `requirements.in`, then run `pip-compile`.

To update existing dependencies, use `pip-compile --upgrade` for all or
`pip-compile --upgrade-package <name>` for a specific one.

If upgrading a dependency because this project's requirements have actually
changed (i.e: we now depend on a feature in a particular version), then instead
update the entry in `requirements.in` and run `pip-compile` as for a new
dependency.

## License

Expand Down
11 changes: 11 additions & 0 deletions requirements.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
mkdocs
mkdocs-material
pymdown-extensions
mdx-include
mkdocs-markdownextradata-plugin
git+https://github.com/pugong/mkdocs-mermaid-plugin@abf14392b0ed0c7022210b32c4e7c9c3e4c5c68a

# Development tooling
yamllint

pip-tools
31 changes: 31 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile
#
beautifulsoup4==4.8.0
click==7.0 # via mkdocs, pip-tools
cyclic==1.0.0 # via mdx-include
jinja2==2.10.1 # via mkdocs
livereload==2.6.0 # via mkdocs
markdown==3.1 # via mdx-include, mkdocs, pymdown-extensions
markupsafe==1.1.1 # via jinja2
mdx-include==1.3.3
mkdocs-markdownextradata-plugin==0.0.5
mkdocs-material==4.1.1
git+https://github.com/pugong/mkdocs-mermaid-plugin@abf14392b0ed0c7022210b32c4e7c9c3e4c5c68a
mkdocs==1.0.4
pathspec==0.5.9 # via yamllint
pip-tools==4.1.0
pygments==2.3.1 # via mkdocs-material
pymdown-extensions==6.0
pyyaml==5.1 # via mkdocs, yamllint
rcslice==1.1.0 # via mdx-include
six==1.12.0 # via livereload, pip-tools
soupsieve==1.9.1 # via beautifulsoup4
tornado==6.0.2 # via livereload, mkdocs
yamllint==1.15.0

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.2.0 # via markdown
3 changes: 3 additions & 0 deletions scripts/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh
RealOrangeOne marked this conversation as resolved.
Show resolved Hide resolved

exec mkdocs build
PeterJCLaw marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 3 additions & 0 deletions scripts/lint-yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh

exec yamllint -sc yamllint.yml yamllint.yml docs/ mkdocs.yml
3 changes: 3 additions & 0 deletions scripts/serve
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/sh

exec mkdocs serve