Update travis scripts to install with pip

[sbidoul]

This is a proof of concept to illustrate how installing with pip can help with dependency management.

There is no need for oca_dependencies.txt anymore as all dependencies are expressed in the __openerp__.py, possibly refined by setup.py, itself refined by requirements.txt if present.

This is IMHO a important enabler to progressively move towards smaller repos.

An example build with this PR is here: https://travis-ci.org/acsone/account-financial-reporting/builds/144847025 (including cross-repo dependencies).

If there is interest, I can finish this PR (ie update mqt test suite mainly).

[moylop260]

• Is possible consider auto-create the setup folder when this one is not exists?

  Many people are using MQT to build theirs projects but we will require to create the setup folder and this matter maybe is not welcome.

• After merging what file we will need delete from our repositories? (oca_dependencies.txt, requirements.txt...)

• There is no need for oca_dependencies.txt anymore as all dependencies are expressed in the openerp.py

  What about if I want use a module from my custom repo instead of official repo?
  What about use a dummy PR where I want to use a developer branch of a dependency to test it with other project e.g. reporting-engine-dev-x testing from account-financial-reporting?

[sbidoul]

@moylop260 thanks for looking into this.

The setup directory is easily created with setuptools-odoo-make-default -d . (part of setuptools-odoo package). It could be autogenerated in travis_install_nightly too. For OCA, it's built nightly by the git bot if it does not exist yet.

If you want to work with a specifc branch of a dependency instead of getting it from wheelhouse.odoo-community.org, you just add it to the requirements.txt file, for example
git+https://github.com/acsone/reporting-engine.git@somebranch#subdirectory=setup/report_xslx&egg=odoo-addon-report_xslx

You can even use different branches for different addons in the same repository.

You can also declare other places to look for dependencies by adding --find-links https://mywheelhouse/... in the same requirements.txt.

So it's very flexible.

[moylop260]

In principle, look good to me
Could you list the change that we will need execute in all oca projects after merging, please?

[sbidoul]

@moylop260 there is nothing to change in OCA projects. We can remove oca_dependencies.txt when we are satisfied everything works fine.

One limitation is that it currently does not work with OpenErp 7.0 because

• OpenErp 7.0 does not pip install at the moment (update 7.0 install requirements so python setup.py install works in 2016 odoo/odoo#12816)
• setuptools-odoo does not support 7.0 at the moment (acsone/setuptools-odoo@e39c831)
• 7.0 OCA addons versions are not normalized

Alternatively, I could preserve the current behaviour with oca_dependencies for 7.0, or use the new behaviour only if oca_dependencies.txt is not present.

[moylop260]

I got it
Thank you

I like use pip if oca_dependencies is not present

[sbidoul]

I like use pip if oca_dependencies is not present

Ok, I will do that, it allows for a progressive deployment

[dreispt]

It's a great idea to keep using the oca_dependencies.txt as now, and using pip installation when that file is not available.
I'll be using the dependencies logic for a the sattelite repo PoC (#338), so I need that code to be kept even if we use pip installation for dependencies.

[sbidoul]

use pip if oca_dependencies is not present

That mean pip will be used for all repos that have no oca_dependencies.txt today. There are many of them. I'm personally fine with that. I'll also need to disable the pip approach for 7.0 as we don't build wheels for 7.0 yet.

[dreispt]

That mean pip will be used for all repos that have no oca_dependencies.txt today.

Yes, but I would expect for no Odoo module to be actually pip installed in those repos: if you have no oca_dependencies.txt, you're not using any modules outside the repo.

If you want to play safe, maybe you can use pip install only if a requirements.txt file is present, even if empty...

[sbidoul]

This one should be good now. The new behavior is triggered if oca_dependencies.txt is absent.

[dreispt]

AFAICS when we have an "oca_dependencies.txt" in the repo, behaviour is unchanged.
If it is missing, we then setuptools package the repo addons, and then pip install Odoo and these addons.

Questions:
a) I assume that this will seamlessly handle Python dependencies declared in the manifests, right?
b) What happens if our pip-installed addons have dependencies from other repos that are not packaged for pip installation?

[sbidoul]

a) I assume that this will seamlessly handle Python dependencies declared in the manifests, right?

Yes, setuptools-odoo does that.

b) What happens if our pip-installed addons have dependencies from other repos that are not packaged for pip installation?

OCA addons can only depend on OCA addons. All OCA repos are packaged for pip installation automatically by a script running nightly.

An improvement I'm considering is to have travis push to wheelhouse.odoo-community.org on success, so we don't have to wait the next day in case we have new cross-repo dependencies.

[sbidoul]

One thing that remains TODO here is to disable the pip install behavour for Odoo versions <= 7.

[lmignon]

@sbidoul May be this change is enough... acsone@01d3c99
not yet tested

[pedrobaeza]

You should also do the same for lower versions (6.1, or even 5.0, 4.2 as we have in OCA/l10n-spain).

[lmignon]

@pedrobaeza I'm not aware of projects using travis on branches < 7.0 In OCA/l10n-spain travis is not configured on 6.1, or even 5.0, 4.2 )

[pedrobaeza]

OK, you're right. Transifex is not mandatory for that old ones.

[sbidoul]

I pushed the check for versions <= 7.0. I hate bash.

[blaggacao]

So resuming the tooling, we will see in this space in the future:

• pip (this PR and odoo-setuptools).

  • theoretical advantage: formalized release cycle through packaging, pip ubiquity
  • theoretical disadvantage: special tooling for packaging required, private code requires additional dedicated infrastructure

• git + mv (oca dependencies).

  • theoretical advantage: home made and well known
  • theoretical disadvantage: entry barrier (although it's actually dead simple, but the fact of sounding unknown and specialized scares away). Somewhat a replica of git submodule in bash.

• git submodules

  • theoretical advantage: git, production / development parity
  • theoretical disadvantage: somewhat complicated

• git subtree

  • theoretical advantage: git, many say easier than submodule
  • theoretical disadvantage: huge repositories due to multiple histories in single repo, well odoo shallow is already 110MB, so it's relative.

• other wrappers?

We use git submodule as it is as agnostic as it can be. For us: agnostic == strategic; True
Never the less @sbidoul I like the suggestion!

[sbidoul]

I updated this PR for odoo 10.0 and pypi support.

An example build of account-financial-reporting without oca_dependencies.txt is here: https://travis-ci.org/OCA/account-financial-reporting/jobs/229481888#L403

I think it is ready to merge now.

cc/ @dreispt

[dreispt]

These dependency dirs will still be there if we have an oca_dependencies.txt, so IMO we should keep this.

[sbidoul]

@dreispt indeed, that was a remnant of the initial prototype. Fixed and rebased.

[lmignon]

This code work for months for the Alfodoo project... https://github.com/acsone/alfodoo/blob/10.0/.travis.yml#L32

[bealdav]

nice feature, thanks a lot for this work 👍

[sbidoul]

superseded by #646