Provide binary releases

[breznak]

of nupic.cpp

• start using releases/tags
  • bump minor version on each PR merged we use manual tag process for releases
  • manually bump major version when needed (= new feature release)
• publish binary releases (on GH)
  • Linux/Travis https://github.com/htm-community/htm.core/releases/tag/v2.0.0-4
  • OSX/CircleCI
  • Windows/Appveyor
  • changed to OSX,WIn upload artifacts from build, and Linux CI fetches artifacts and deploys a GH Release

TODO:

• fix Windows binary releases
• check that the releases contain all needed files
  • to run c++ examples, tests
  • to link htm.core as a library with a c++ program
  • OSX
  • linux
  • windows
• optional: publish bin releases from ARM builds

[breznak]

Q: do we want a binary release for each PR merged, or manually on new tagged versions?

[breznak]

https://docs.travis-ci.com/user/deployment/releases/

[ctrl-z-9000-times]

do we want a binary release for each PR merged, or manually on new tagged versions?

I think we only need binary releases for tagged versions.

[breznak]

@rhyolight can you help me with the release/tag process, please?

• we want to build releases (on Travis, for Linux) and publish on GH

• I make a PR

• make it tagged Releases3 #369

  • but deploy is not run as "only PR" https://travis-ci.org/htm-community/nupic.cpp/builds/516106482#L3009

• when merged, the Release does not appear either

🍩

[breznak]

I've been working on this a lot today, some progress:

Linux:

• Travis reacts to "tag" events,
  • had to enable "Push" (along with existing "PR/Pull") events in Travis settings (not part of config file)
  • after that, to prevent "double builds" (travis/pr, travis/push) added excludes to limit builds only to: master & tag "branches".
• updated credentials for GH/Travis
• Linux now creates automatically a binary release for correct tag on master
• still missing: unfortunately, the release does not contain the binary .tar.gz we want 😞 -> FIXED https://github.com/htm-community/htm.core/releases/tag/v2.0.0-4

Work-flow should be as follows:

• PR, developement is unaffected. We make PRs, merge when "Approved" and all checks pass.
• for "Release":
• make tag (in format vX.Y.Z) on master, and push it. It will trigger a new build with release uploaded to GH.
  • git checout master; git tag v2.0.0-5; git push --tags origin

Here is the first passing build status with GH Release deployment:
https://travis-ci.org/htm-community/htm.core/builds/570960809

This is the release:
https://github.com/htm-community/htm.core/releases/tag/v2.0.0-3

• still empty (like my soul now 😢 )

WIP, help wanted.

AppVeyor CI Releases are WIP in
#624

[breznak]

There is an upcoming problem and inconveniences:

• we have the CI on 3 different providers for all 3 platforms.
• so when each CI pushes a GH Release, we'll have "3 releases" for a single release (git tag)

Instead of looking like this:

Release 2.0.0

Files:

• htmcore-2.0.0-win.zip
• htmcore-2.0.0-osx.tar.gz
• htmcore-2.0.0-linux.tar.gz

we will have:

Release 2.0.0-Linux

Files:

• htmcore-2.0.0-linux.tar.gz

Release 2.0.0 OSX

Files:

• htmcore-2.0.0-osx.tar.gz

etc

This is an inconvenience, but workable.

[breznak]

ok, and the "problem" would not be a problem. I was worried about pypi #19 , and the same scenario: we want a pip package with a single name, say htmcore, but we have 3 providers of CI.

But just one CI should be responsible for publishing the PYPI package, and inside its setup.py it would decide which bindings to download based on platform.

[breznak]

First Linux binary release available: https://github.com/htm-community/htm.core/releases/tag/v2.0.0-4

Please test if the binary works
CC @dkeeney @ctrl-z-9000-times

[dkeeney]

Great progress. I think we can live with this for a while but I agree that PyPi is a problem.

What I envisioned was the PR build that succeeds would actually checkin its two packages so that after all three CI's are finished, the git for the PR would contain all 6 packages along with the source. Then during the merge the packages move into the master and a hook just needs to distribute the packages to the places they need to go at GitHub and PyPi. I don't know...is that possible?

[dkeeney]

Cool. I downloaded the package and opened the tar.gz. Then ran the unit_tests and it worked.... of course one test failed.
[ FAILED ] HelloSPTPTest.performance

Let me try to figure out how to use the wheel file and I will try that too.

[dkeeney]

That works too.

pip install htm.core-2.0.0-cp37-cp37m-linux_x86_64.whl

It includes the commands to load the dependencies too. It tells me that everything is already satisfied but I guess that is to be expected.

[breznak]

Great to hear, testing on..Windows..er, Linux VM?

[ FAILED ] HelloSPTPTest.performance

even on a local machine? That's unexpected. Does it fail on time, or exact output?
Thanks for testing!

[breznak]

What I envisioned was the PR build that succeeds would actually checkin its two packages so that after all three CI's are finished, the git for the PR would contain all 6 packages along with the source.

yeah, that would be the best solution. But difficult to execute.

1. AWS
   We could store all the artifacts (=binary for each platform) on a 3rd party storage (that we can access from each CI). Then one CI is "master" and downloads the other 2 artifacts, now it has all 3 required binaries, and it can make a GH Release.

• CONs:
  • need a free storage (AWS S3?)
  • need to access it (credentials) from the other 2 CIs
  • the master CI needs to sync (wait if the other 2 builds are delayed, deploy only when all 3 are avail)
• PROs:
• just 1 CI to resolve GH Releases (currently Travis is ready)

2. Single CI provider

• as discussed in Reconsider CI providers - switch to GitHub Actions #612 , we could consolidate our CI under a single provider (all 3 now support all 3 platforms) (my preference would be CircleCI from their responsive support) (GH Actions CI should be upcoming CI provider, "inhouse")
• CONs
  • probably slower build times (now the 3 platform-builds run effectivele in parallel)

3. wget from other CI

• similar to 1)
• but don't need to use 3rd party file-storage, if we can get URLs to the artefacts from the other CIs, we can just wget them into the master CI.
• PROs
  • gets away with the storage
• CONs
  • still need to sync if the files exist
  • need to get URLs for said tag

[breznak]

I'm trying one more option, if I could "create" a release with an existing name, and just push/update the new file, we could put all 3 artifacts into a single release.

[dkeeney]

Great to hear, testing on..Windows..er, Linux VM?

Ubuntu 19 using VBox.

even on a local machine? That's unexpected. Does it fail on time, or exact output?

I often get this on the local machine...fails on time. Not consistent.

[dkeeney]

yeah, that would be the best solution. But difficult to execute.

No, what I had in mind was to actually put the binaries into the Git repository and push it into GitHub. I know the CI can checkout, so can they also make changes to the PR's repository? Could get messy I guess.

[breznak]

No, what I had in mind was to actually put the binaries into the Git repository

that's not a good solution, and not feasible long term. Git is not suited for large, binary files.
Here you have 3x~70MB, not too much to download by todays standards, but not nice to have it in git. The problem starts that with each release, you would get the new 210MB files. After a few month the repo clone would be impossible.

Further, the CI don't have (and should not) git push access.

[breznak]

since https://travis-ci.org/htm-community/htm.core/builds/571444454
we have first, full Release: https://github.com/htm-community/htm.core/releases/tag/v2.0.0-7

@dkeeney this release can be used as testing for PyPI #19 package uploads.

TODO:

• OSX CI does not run new build on tag to master
• use API to fetch artifacts from the CI, currently we use hard-coded URL only
  • use version in the filenames added versioning from git tags #632

[breznak]

this is a PR with tag:
https://travis-ci.org/htm-community/htm.core/builds/571494896

 v2.0.0-8 Readme: update support for Py2

which is not tested in CI anymore

Commit a627fa7 Compare a627fa7c6618 Tag v2.0.0-8

This is merged the PR above to master:
https://travis-ci.org/htm-community/htm.core/builds/571496907

 master Merge pull request #633 from htm-community/test_trigger

Readme: update support for Py2

Commit adf8039 Compare 947421a..adf8039 Branch master 

So I don't think this woud trigger the deploy on master in Travis, @dkeeney .
Is there a problem if the VERSION is set correcly only during the tagged, GH release builds?

[breznak]

So I don't think this woud trigger the deploy on master in Travis

ok, crisis saved. It works ok with the workflow you need:

• tag a branch
• make a PR
• pass tests
• deploy GH release
  • hack: the release is created whenever the PR passes (likely even doesn't have to be approved, or merged). This could be a security problem, anyone could craft a tagged PR, open PR and have new release (-> on pypi) even before we can see that.
    https://travis-ci.org/htm-community/htm.core/builds/571494896#L4920

[breznak]

ok, crisis saved. It works ok with the workflow you need:

so, no. Crisis is even worse 😛

• your approach does not work, the "Merged PR" does not ingerit the tag and does not trigger a build:
  https://travis-ci.org/htm-community/htm.core/builds/571496907#L2628

• what works is tagging a PR, and opening that. Unfortunately, we must close that as it imposes security risk of un-signed-off releases.

• what we will use is tag on master, issued only by repo-write-access people.

• if the VERSION with git describe is complicated during this, why don't we go the KISS method: manually edit VERSION?

[breznak]

@dkeeney we still have a problem with the releases:

• OSX CI does not seem to run on master tag
• Travis gets timed-out waiting for Win CI to finish
  • How do we go about that?

[dkeeney]

Hmmm, I have no idea. You know more about this than I :-(
Keep in mind that what we are trying to do is probably also being done by a lot of other people so this problem has most likely already been solved in some way or another.

Maybe we need to re-think all of this.
If there was a way to use the last PR build then we would know that all artifacts are already present at merge time. If we could do that then the merge or master tag does not require a build. The problems we need to address to do this:

• When the PR build happens, the Version must be the one that is expected in the release. This is what GitVersion tries to do.
• Get travis to start and run your script when a merge happens and a tag has been set in the PR. Maybe use a GitHub hook rather than the CI.
• Locate the artifacts in the last PR (don't think that will be too hard)
• Add the deployment actions to your script if it is to run as a GitHub hook.

don't know....just throwing out ideas.

I feel a little handicapped not being able to login to AppVeyor and see what artifacts are there but between us we can figure this out.

[breznak]

I feel a little handicapped not being able to login to AppVeyor and see what artifacts are there but

what limits you in AppV if you login with GH account? I'd like to make you able to see as much as you need

[dkeeney]

what limits you in AppV

I login with GH account. That part works. The drop down list in upper right shows my name (for my own projects) and numenta-ci. htm-community is not listed. If I select numenta-ci I can see their projects but there is no way to select htm-community. It is like it does not know that I am a member of htm-community. Very strange.

If I go over to CircleCI or Travis and login I then can see htm-community and all of its projects.
Of course GitHub knows I am a member of htm-community.

[breznak]

@dkeeney can you try again, please? I've changed AppV authorization, we're now using (recommended) "AppV Github App" which I installed under htm-community. Things should work now.

[dkeeney]

No, did not help.
I even cleared all cookies on my browser and tried again.
IF I go to the Security settings on the AppVeyor, under "my accounts" I see

• dkeeney
• numenta-ci

If I go to my profile page on github, I see that I am members of both numenta-ci and htm-community.

I suspect AppVeyor has me messed up in its database.

[breznak]

It should be working now, I've added htm-community/htm.core , and you're member of that team. Please let me know

[dkeeney]

YES! That worked. Thanks.

I will take a look at this and see if this helps me any.

[breznak]

Binary releases are now available!
https://github.com/htm-community/htm.core/releases

@dkeeney would you please check that

• the release contains all needed files? What should be there?
  • distr/dist contains only requirements.txt
  • on my localhost there's also an .egg
  • .whl is in py/, shouldn't be in dist too?
• Windows binary is missing
  • something (the Win Release action?) is failing, might be platform support problem as in Publish bindings on pypi #19 (comment)

[breznak]

Windows binary is missing

this is because Windows build gets terminated early by failing PYPI upload on Linux, so non-issue here. See #19 (comment)

[dkeeney]

the release contains all needed files? What should be there?

I think I told you wrong. The requrements.txt file needs to be in the same folder as setup.py ... which is distr/ not distr/dist AND setup.py needs to be looking there to find it.

I am looking at https://packaging.python.org/tutorials/packaging-projects/, In the section on "Generating distribution archives". This is all very confusing as to where things go.

If I understand correctly, when pip installs using a .whl file, it runs the setup.py file contained in the wheel in order to process the requirements.txt. When setup.py runs in this environment it does not have the entire repository...only what is in the wheel and that is whatever we setup in build/Release/distr . So setup.py and requirements.txt need to be in distr/ and setup.py should be looking there to find requrements.txt.

So the concept is to put together a folder that is needed for the wheel which in our case is build/Release/distr (and dist/ in the tutorial). This should have the following:

• setup.py (pip looks for this in top of wheel)
• requirements.txt (used by setup.py to install requirements)
• dummy.c (used by setup.py, setup() function to assemble the extensions)
• src/ (folder referenced in setup.py, setup() as the location of python modules.)
  Note that src/htm/bindings is where it should find the binary extensions.

You will notice that in our repository top folder, there is a setup.py which redirects to the one in repro/bindings/py/packaging. This in turn tells its setup() function to use the folder build/Release/distr to create the egg and wheel.

[breznak]

So setup.py and requirements.txt need to be in distr/ and setup.py should be looking there to find requrements.txt.

requirements cannot be in distr/ literally, as the folder is being created. It can be relative to setup.py (./requirements.txt, which is the case now)
And I'll place it to distr/, from distr/dist/.

You will notice that in our repository top folder, there is a setup.py which redirects to the one in repro/bindings/py/packaging

ok, the /setup.py is the one called..but the path is relative to the packaging/setup.py,...

[dkeeney]

Oh, I just discovered that I am telling you wrong again....
I unpacked the wheel.... it is just a Zip file with the extension changed.

setup.py is NOT included in the wheel, but requirements.txt is included.

[breznak]

Oh, I just discovered that I am telling you wrong again....

since you are studying the setup now, would you mind making the changes if you run into them? What we need is a whl that one can pip install, once that is working, we will use that for PYPI #19

[dkeeney]

would you mind making the changes if you run into them?

Sure, That is one way I can help. I cannot be much help with the rest of this deployment thing.

[breznak]

Our 90th release v2.0.16 finally contains binaries for all 3 platforms!

• thanks David for working on validating the whl pip files!
• next thing we should check is running c++ tests,examples and linking the htm library to a program on all platforms.
  • OSX is suspiciously thin, 10MB, for others the size is ~90MB (?)