ansible: add Pipfile and make python3 compatible #1399

refack · 2018-07-12T19:21:29Z

Add Pipfile to describe dependencies
add needed git ignore rules
make code PEP8 compliant and add .flake8
make code python3 (3.7 specifically) compatible

refack · 2018-07-12T19:22:03Z

refack · 2018-07-12T19:26:39Z

P.S. workflow is:

Setup

> cd ansible
> pipenv install

excecute

> pipenv shell
(venv)> ansible-playbook ...
or
> pipenv run ansible-playbook ...

maclover7 · 2018-07-24T15:41:13Z

@refack Would you be able to update ansible/README.md and the other docs with the new pipenv commands?

rvagg · 2018-11-28T11:06:09Z

I don't understand any of this, I just know ansible is broken for me on my mac

does it have to be so heavy-weight? switching to items() in filter.py fixes my immediate issue. Why do we need these dependencies and what is PEP8 and why do we need it? Is there a minimal version of this?

cclauss · 2019-01-30T07:02:42Z

Rebase?

refack · 2019-03-14T23:29:38Z

Code has been reworked and rebased.
Some stuff that "work for me":

ansible -m ping test*

ansible-playbook -vvv playbooks/upgrade-packages.yml -l test-rackspace-ubuntu1604-x64-2

ansible-playbook -vvvv playbooks/jenkins/worker/create.yml -l test-rackspace-ubuntu1604-x64-2 --check

ansible/plugins/inventory/nodejs_yaml.py

rvagg · 2019-03-15T01:03:07Z

setup/www/host_vars/.gitignore

+infra-*
+!node-www.tmpl


what's this for?

We're tracking this file https://github.com/nodejs/build/blob/master/setup/www/host_vars/node-www.tmpl
(keeping it ignored, make my IDE angry)

rvagg · 2019-03-15T01:04:53Z

So, with this we'd need to execute ansible from inside pipenv and not bother with having a global ansible package?

Can you explain the logic of this for me and what the dependencies are for?

rvagg · 2019-03-15T01:08:19Z

And: does this crate necessary dependencies? Will running the playbooks directly with ansible-playbook even on a machine with Python 3 still work or must we use the steps above? If so, maybe we could script it? ./run-playbook.sh .....

refack · 2019-03-15T02:19:50Z

So, with this we'd need to execute ansible from inside pipenv and not bother with having a global ansible package?

Can you explain the logic of this for me and what the dependencies are for?

It's an alternative approach.
tl;dr python out-of-the-box works with global dependencies.
Then came virtualenv that wraps python with some monkey-patches so it first looks for dependencies in a local dependency store.
Then came pipenv and the Pipefile that allows for a workflow similar to npm (i.e. dependencies stored locally by default, and pip install installs locally and updates the Pipfile).

And: does this crate necessary dependencies? Will running the playbooks directly with ansible-playbook even on a machine with Python 3 still work or must we use the steps above? If so, maybe we could script it? ./run-playbook.sh .....

It's still possible to use global dependencies just as before. Or we can use npm run style recipes that allow doing things like:

pipenv run playbook [what [ever [args]]]

example:
https://github.com/refack/GYP/blob/86ab71caad4640dcc3ec572c2118a1ce3f24150f/Pipfile#L12-L14
used as:
https://github.com/refack/GYP/blob/master/.travis.yml#L35

rvagg · 2019-03-15T02:41:14Z

OK, as long as it's documented I could live with it. I'll try it out soon on some of my known-breaking Python 3 Ansible tasks.

rvagg · 2019-03-17T08:28:09Z

this worked on a freebsd and ubuntu 14.04 reprovision but the problem I have with ccache jinja2 scripting persists:

fatal: [test-requireio_kahwee-debian9-arm64_pi3-1]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ ccache_html_content.stdout | regex_findall('ccache-[0-9]+.[0-9]+(?:.[0-9]+)*.tar.gz') | map('regex_replace', 'ccache-') | map('regex_replace', '.tar.gz') | list | latest_version }}): '<' not supported between instances of 'map' and 'map'"}

cclauss · 2019-03-17T09:10:28Z

You could try putting | list | in between the two calls to map(). Python 3 often creates generators instead of lists so perhaps forcing the results of the first map() call into a list will placate Jinja2.

Update: Did not work at #1674 :-(

refack · 2019-03-17T13:35:33Z

'<' not supported between instances of 'map' and 'map'"

forcing the results of the first map() call into a list will placate Jinja2.

👍 make sense.

anyway.
I'll try to move that super-filter out of the template and into code

thefourtheye · 2019-03-19T04:12:29Z

ansible/plugins/inventory/nodejs_yaml.py

-               'macstadium', 'marist', 'mininodes', 'msft', 'osuosl',
-               'rackspace', 'requireio', 'scaleway', 'softlayer', 'voxer',
-               'packetnet', 'nearform')
+    # taken from nodejs/node.git: ./configure


I don't think they are in sync anymore. https://github.com/nodejs/node/blob/ed2c6965d2f901f3c786f9d24bcd57b2cd523611/configure.py#L48-L49

valid_arch = ('arm', 'arm64', 'ia32', 'ppc', 'ppc64', 'x32','x64', 'x86', 'x86_64', 's390', 's390x')

thefourtheye · 2019-03-19T04:13:37Z

ansible/plugins/inventory/nodejs_yaml.py

-    parser.add_argument('--host', action='store')
-    args = parser.parse_args()
-
+    # parser = argparse.ArgumentParser()


Why are we checking in commented code?

thefourtheye · 2019-03-19T04:14:24Z

jenkins/scripts/coverage/generate-index-html.py

@@ -6,9 +6,9 @@
 with open('out/index.csv') as index:
  index_csv = filter(lambda line: line, index.read().split('\n'))

+# noqa


Is this comment incomplete?

LGTM: This flake8 linter directive will silence the linter as discussed at
http://flake8.pycqa.org/en/3.7.7/user/violations.html#in-line-ignoring-errors

cclauss · 2019-07-29T05:59:31Z

@refack Can you please go through the comments (esp. #1399 (comment)) and make any required updates? It would be a big help to have pipenv in place.

pipenv has the effect of running Python in its own virtualenv on the user's machine which means that the Python version is more predictable and that no outside dependencies are installed.

cclauss

Let's move forward with this to ease the transition to Python 3.

Trott · 2019-08-19T17:57:25Z

@refack Can you please go through the comments (esp. #1399 (comment)) and make any required updates? It would be a big help to have pipenv in place.

@cclauss Unfortunately, @refack has not been active lately. My hope is he's just taking a break from Node.js stuff and hasn't left entirely, but either way, it would probably be best for someone to take this over and finish it. Are you up for doing that?

* add Pipfile * de-lint with `flake8`

Trott · 2019-09-27T20:40:17Z

Rebased to eliminate conflicts and force-pushed. @cclauss @refack

cclauss · 2019-09-30T04:09:58Z

My sense is that this can be closed unless @refack has objections. I like the idea of automating the creation of virtualenvs for our builds to take place in and pipenv would deliver that. However @sam-github and I agreed that there might be compatibility issues on non-mainstream Win/Mac/Linux so we should delay that experimentation until after we have Python 3 working.

Trott · 2019-09-30T17:51:57Z

My sense is that this can be closed unless @refack has objections. I like the idea of automating the creation of virtualenvs for our builds to take place in and pipenv would deliver that. However @sam-github and I agreed that there might be compatibility issues on non-mainstream Win/Mac/Linux so we should delay that experimentation until after we have Python 3 working.

In that case, I"m going to close this, but @refack or anyone else should feel absolutely free to re-open it if they think that's the wrong move.

refack added enhancement ansible labels Jul 12, 2018

refack mentioned this pull request Jul 12, 2018

Add virtualenv to ansible directory #1389

Closed

refack mentioned this pull request Jul 24, 2018

Old style exceptions --> new style for Python 3 #1411

Closed

This comment was marked as off-topic.

Sign in to view

rvagg mentioned this pull request Nov 28, 2018

ansible: minor python2/3 compat fix #1607

Closed

cclauss mentioned this pull request Feb 2, 2019

build: find Python 3 or Python 2 in configure nodejs/node#25878

Closed

4 tasks

refack force-pushed the pipenv+python3 branch from 2b5e063 to f7bc1a7 Compare March 14, 2019 23:23

refack mentioned this pull request Mar 14, 2019

Add Python 3 to more build machines #1674

Closed

rvagg reviewed Mar 15, 2019

View reviewed changes

ansible/plugins/inventory/nodejs_yaml.py Outdated Show resolved Hide resolved

rvagg reviewed Mar 15, 2019

View reviewed changes

thefourtheye reviewed Mar 19, 2019

View reviewed changes

cclauss mentioned this pull request Mar 19, 2019

Put a list between the two maps for Python 3 #1734

Closed

cclauss approved these changes Jul 29, 2019

View reviewed changes

rvagg mentioned this pull request Aug 20, 2019

update arm jump host proxycommand #1890

Merged

cclauss mentioned this pull request Sep 27, 2019

ansible: improve Python 3 compatibility #1929

Merged

python: make code compatible with Python3.7

e7ee43c

* add Pipfile * de-lint with `flake8`

Trott force-pushed the pipenv+python3 branch from b17f2fa to e7ee43c Compare September 27, 2019 20:39

Trott closed this Sep 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ansible: add Pipfile and make python3 compatible #1399

ansible: add Pipfile and make python3 compatible #1399

refack commented Jul 12, 2018 •

edited

Loading

refack commented Jul 12, 2018

refack commented Jul 12, 2018

maclover7 commented Jul 24, 2018

This comment was marked as off-topic.

rvagg commented Nov 28, 2018

cclauss commented Jan 30, 2019

refack commented Mar 14, 2019

rvagg Mar 15, 2019

refack Mar 15, 2019

rvagg commented Mar 15, 2019

rvagg commented Mar 15, 2019

refack commented Mar 15, 2019

rvagg commented Mar 15, 2019

rvagg commented Mar 17, 2019

cclauss commented Mar 17, 2019 •

edited

Loading

refack commented Mar 17, 2019

thefourtheye Mar 19, 2019

thefourtheye Mar 19, 2019

thefourtheye Mar 19, 2019

cclauss Mar 19, 2019

cclauss commented Jul 29, 2019 •

edited

Loading

cclauss left a comment

Trott commented Aug 19, 2019

Trott commented Sep 27, 2019

cclauss commented Sep 30, 2019

Trott commented Sep 30, 2019

		infra-*
		!node-www.tmpl

ansible: add Pipfile and make python3 compatible #1399

ansible: add Pipfile and make python3 compatible #1399

Conversation

refack commented Jul 12, 2018 • edited Loading

refack commented Jul 12, 2018

refack commented Jul 12, 2018

maclover7 commented Jul 24, 2018

This comment was marked as off-topic.

rvagg commented Nov 28, 2018

cclauss commented Jan 30, 2019

refack commented Mar 14, 2019

rvagg Mar 15, 2019

Choose a reason for hiding this comment

refack Mar 15, 2019

Choose a reason for hiding this comment

rvagg commented Mar 15, 2019

rvagg commented Mar 15, 2019

refack commented Mar 15, 2019

rvagg commented Mar 15, 2019

rvagg commented Mar 17, 2019

cclauss commented Mar 17, 2019 • edited Loading

refack commented Mar 17, 2019

thefourtheye Mar 19, 2019

Choose a reason for hiding this comment

thefourtheye Mar 19, 2019

Choose a reason for hiding this comment

thefourtheye Mar 19, 2019

Choose a reason for hiding this comment

cclauss Mar 19, 2019

Choose a reason for hiding this comment

cclauss commented Jul 29, 2019 • edited Loading

cclauss left a comment

Choose a reason for hiding this comment

Trott commented Aug 19, 2019

Trott commented Sep 27, 2019

cclauss commented Sep 30, 2019

Trott commented Sep 30, 2019

refack commented Jul 12, 2018 •

edited

Loading

cclauss commented Mar 17, 2019 •

edited

Loading

cclauss commented Jul 29, 2019 •

edited

Loading