Automate external content update and website redeploy with Travis CI

[ghys]

This is an solution to keep the docs updated automatically on this repo, and the website, using Travis CI builds.

Once this PR is merged and Travis is configured for this repo, the builds will:

1. run update-external-resources.sh
2. run mvn clean package which will execute the Groovy script
3. add a commit with the results
4. push the results back to the gh-pages branch (no PR) (the [skip ci] in the commit message will prevent triggering another build)
5. call a hook at Netlify to trigger the redeploy of the website, which will pick up the changes

If any of these steps fail, 5. won't be executed.

The build will only consider changes to the gh-pages branch; it can also run manually or more interestingly as a cron job (configured in the settings):

Inspired by https://gist.github.com/willprice/e07efd73fb7f13f917ea and https://gist.github.com/Maumagnaguagno/84a9807ed71d233e5d3f

Example build log: https://travis-ci.org/ghys/openhab-docs/builds/388941296
Example resulting commit by the above build: ghys@245359a

@kaikreuzer if you agree with the principle of this PR, the 2 "secure" lines in .travis.yml have to be replaced (background info) - you need Ruby and the travis command line tool (gem install travis).

• echo "GH_TOKEN=92..." | travis encrypt --add
• echo "NETLIFY_BUILD_HOOK=5b..." | travis encrypt --add

For the value of GH_TOKEN, create one at https://github.com/settings/tokens with the openhab-bot account (with the public_repo scope). For the value of NETLIFY_BUILD_HOOK go to https://app.netlify.com/sites/openhab-website/settings/deploys#build-hooks (it's already there)

[Confectrician]

I love the approach. 👍
But I will leave this pr to @kaikreuzer .

[ThomDietrich]

Looks good to me. Back when I worked on these scripts I deliberately did not implement it as an automated travis script because you can't run it with every new gh-pages commit and running it manually is easily done by hand. I like the suggestion to define a cron job though!

[ghys]

Now that there's an additional step with the website I thought it was worth giving it a try and luckily it worked well, hence the PR :) This first iteration doesn't really change the process, only automates it (even if the builds are not triggered for each commit, it's still nice to trigger them manually from the Travis website if you don't have the build environment at hand).
If there's a "cleaned up" master branch in the future as a result of #665, the build script could be changed to merge the changes from master to gh-pages before updating the external content, and committing back to gh-pages. That way the master branch won't have those automated commits from openhab-bot.

[Confectrician]

it's still nice to trigger them manually from the Travis website if you don't have the build environment at hand

That was exactly my thought too.
I think i could start that job too from my travis dashboard.
At least i can already see the repo there, since i have got write access.
A really nice way of let someone different worry about the build environment and no need to install several dependencies again just for that single purpose.

[ThomDietrich]

Definitely! My comment was an anecdote, not a critique 😃
Way to go!

[kaikreuzer]

Thanks!
Two remarks/questions though:

1. As mentioned in Update ZWave documentation #695 (comment), I would not want to run this update before we have a 2.3 version of the docs out there.
2. Is there any strong reason for using Travis instead of Jenkins? All our official builds are currently only on Jenkins and this also has access to the credentials for openhab-bot. Furthermore, we could trigger this build also upon new builds of the add-on projects, so that updates on READMEs are directly considered. I'd clearly prefer it to be located there.

[Confectrician]

Last versioned docs were mady by @ThomDietrich.
I don't know if there is a bash script that does the work or if it is simple copy/paste work.

If you let me know i could prepare another versioned subfolder @ThomDietrich or you can do that yourself if you have enough time.

[kaikreuzer]

@Confectrician See #695 (comment): I wouldn't want the 2.3 stuff on the old website, but on the new one. That's why I hope that @ghys will eventually find some solution to do so. Ideally, we would not even have to create a folder with the copied content, but rather take the content from a 2.3-tag on openhab-docs.

[ghys]

Is there any strong reason for using Travis instead of Jenkins?

I don't have access to a Jenkins instance to play around, and Travis was quick & easy to set up and allowed testing on my fork, that's all.

I hope that @ghys will eventually find some solution to do so. Ideally, we would not even have to create a folder with the copied content, but rather take the content from a 2.3-tag on openhab-docs.

Putting the previous versions in the same VuePress app is definitely a bad option - it clutters the router, increases the build time, and other technical constraints make it not viable in the long term as new versions are added especially with faster release cycles. So there likely will be a "frozen" website (read: VuePress app) by version. I'll strip the home page, blog, about pages etc. only keeping /docs/ and /addons/ and will make all these pages (at least, the top level navigation links) redirect the user back to the canonical URLs on the main site, so it would be practically unnoticeable by users.

Now as to how to organize them, let's look at how a few other projects deal with versioning.
There seem to be mainly 2 practices:

1. putting them in "folders" either on the same domain (like it's done now on this repo), or another one, examples:

• the LInux kernel: https://www.kernel.org/doc/html/
• ReadTheDocs.io explain it here: https://docs.readthedocs.io/en/latest/versions.html - dropdown in the footer to switch between latest and stable
• Python: https://docs.python.org/3/ (dropdown in the header)
• NodeJS: https://nodejs.org/en/docs/
• AngularJS (1.x) is hosting the latest docs on https://docs.angularjs.org/guide while all the others versions (dropdown in the header) are hosted on a different subdomain, e.g. https://code.angularjs.org/1.2.32/docs/guide

2. putting them each in their own subdomains:

• https://angular.io/docs present the stable docs, while switching to other versions (dropdown at the bottom of the sidebar) take the user to https://next.angular.io/docs or https://v5.angular.io/docs etc.
• https://quasar-framework.org/guide/ also have separate subdomains by version (dropdown in the sidebar)
• React (https://reactjs.org/versions) does it with Netlify deploys but it's rather ugly...

We can host the v2.3 docs on https://www.openhab.org/v2.3 for now, but I worry even if it's pre-built, uploading them to the site at each CI build (there are about 3000 files and 65 MB) won't be reasonable after a few versions. Another option is to continue uploading VuePress sites here, and host them at https://docs.openhab.org/v2.x like it's done now (and like AngularJS) or another completely different subdomain like https://archive.openhab.org/v2.x.

Option 2. is also interesting, because they could be separate CI builds for former versions (no need to add pre-built websites to repos), and updating them even if the content is "frozen" - for instance if there's a style or navigation menu change on the main site - would be easier. I kind of like the https://next.angular.io way of having the "next" (a.k.a. snapshot) docs while the primary website hosts the stable version - as I said, the whole website wouldn't be duplicated like they do, only the docs.

[ghys]

We can host the v2.3 docs on https://www.openhab.org/v2.3 for now

I have just done that - it is visible especially at https://www.openhab.org/docs/installation/, where recent commits (replace ESH Designer mention by VS Code) are integrated vs. https://www.openhab.org/v2.3/docs/installation/ where they're not.

It seems to only add 20-30s of build time for the website, this is acceptable but we should keep this discussion in mind - deploy previous versions separately from the main website - for future versions.
Nothing is carved in stone and adjustments are always possible if we change our mind.

Note that I can easily re-generate the 2.2.0 docs as a VuePress site and put them under https://www.openhab.org/v2.2 as well if it's desirable. (2.1.0 is more complicated)

[ghys]

Note that I can easily re-generate the 2.2.0 docs as a VuePress site and put them under https://www.openhab.org/v2.2 as well if it's desirable. (2.1.0 is more complicated)

Done that as well. Since I won't do 2.1 and it's the only version left, I've removed it from the dropdown. Having the stable and the former stable version is enough IMHO.

[kaikreuzer]

I don't have access to a Jenkins instance to play around

That should be easy to change - do you already have at Cloudbees? If so, give me your e-mail that you are using on it and I'll add you as an admin on our instance.

Thanks for the static version deployments, that's great!
One small issue/question: When I select version 2.3, the version 2.2 disappears from the version selection box. Likewise, being on 2.2, I do not see 2.3 anymore. Is that a bug or due to some technical constraints? If so, I don't think it is severe and we can keep things as they are as the most important part is being able to get back to latest.

[Confectrician]

May i ask for Jenkins access too?
If its the preferred build provider for openHAB it would be nice to dig into it.
(If a reduced access is available that would be fine for me too.)

[ghys]

That should be easy to change - do you already have at Cloudbees? If so, give me your e-mail that you are using on it and I'll add you as an admin on our instance.

I signed up with GitHub:

Thanks, I'll look into it and won't break anything ;)

When I select version 2.3, the version 2.2 disappears from the version selection box. Likewise, being on 2.2, I do not see 2.3 anymore.

Yes, that's actually by design. The "frozen" site for a paticular release would be built around the time of that release, and won't have a knowledge of future versions - that's why instead of having incomplete dropdowns with only the past versions it's basically only a "back to latest" link where you have the complete list and can switch to another one. Granted, it might be confusing at first, didn't spend too much time on it but it can always be improved while there aren't so many versions (for example by fetching a JSON file with the version numbers at a well known location or some other trick).

[kaikreuzer]

Thanks, I'll look into it and won't break anything ;)

I've added you to Cloudbees - for some reason, I cannot switch you from user to admin; maybe you first have to join the team before I can change the role, so I'll try again later.

Yes, that's actually by design.

Fine for me - not worth to spend much time on it.

[kaikreuzer]

@ghys Managed to make you admin - so go ahead!

[ghys]

First successful Jenkins build 🎉
https://openhab.ci.cloudbees.com/view/Sandbox/job/sandbox-update-docs/6/consoleFull

I'm only working a temporary branch but the changes have been successfully pushed back:
7e4e72b

Had to configure a downstream project to call Netlify but this worked too (it's a hook for another website, didn't want to rebuild the prod now):
https://openhab.ci.cloudbees.com/view/Sandbox/job/sandbox-redeploy-website/4/console

What triggers the builds (cron jobs only, every commit...) is left for discussion.

The merge from (future) master to gh-pages would likely be another build (using the "merge before build" feature).

Feedback appreciated!

[kaikreuzer]

And the website can only grab it from a single repo/branch or would it be theoretically feasible to have the manual stuff in master and the generated/synched stuff in some other branch?

[ghys]

Yes, I suppose that could be done.

Still like the master -> master+generated merge option better though 😉, because wouldn't it be easier to tag the entire doc base for a specific release at a certain commit/point in time (and keep the existing ones) if everything is gathered in a single branch?

[Confectrician]

Still like the master -> master+generated merge option better though 😉, because wouldn't it be easier to tag the entire doc base for a specific release at a certain commit/point in time (and keep the existing ones) if everything is gathered in a single branch?

I like that approach too.

Imho, we can leave it in place as it is and serve the old docs.openhab.org from it.

We can add of course a new branch "served content" and feeze the "gh-pages" branch for now,
to keep docs.openhab.org useable as long as possible.
We will get more and more broken links and images while adapting content to the new page, so this would be worth to think of.

[kaikreuzer]

because wouldn't it be easier to tag the entire doc base for a specific release at a certain commit/point in time

Touché. Good point, you win ;-)

[ghys]

(full disclosure: I deleted my comment above about fast-forwards because I was wrong ;))

Okay, I think I have something working, which would solve this and #665.
Here's the principle:

You can see this live at: https://github.com/openhab/openhab-docs/network

The new default branch visible to users will be jenkins-tmp-master (we can rename those, of course) , in blue in the picture above, with only the strictly necessary content - much less intimidating!
Changes to the documentation like PRs would be made in that branch only - I've made a few fixes without PRs to test. Sorry @Confectrician I know you prepared a master branch in your fork too, but I needed it in the main repo ;)

Regularly, the merge-docs job will merge the latest from the master branch to the jenkins-tmp-final branch (in black in the picture). Sadly, it turns out merge commits are unavoidable; imho this is not that big a deal since that branch is a "technical" one, the "master" branch would remain clean of those technical commits (including external content update commits).

In parallel, the gather-external-docs will execute the shell script + Maven goal to update the "generated" content from the other repos, committing directly to the jenkins-tmp-final branch, then trigger the website build - if there is nothing to commit (i.e. no external documentation was updated since the last build) then the build is marked "instable" (yellow) so that redeploy-website won't be triggered - no need to redeploy the website and invalidate browser caches etc. if nothing has changed.

What do you think?

[Confectrician]

Sorry @Confectrician I know you prepared [...]

No problem. I would have pushed it directly, if you left a comment that you already need it for testing.
Anyways, its there now.

I would prefer "master" as this is a known name for users with some git knowledge.
I can delete my prepared master then?

What do you think?

Looks good to me in general.

Question, for my unferstanding:
Will gather-external-docs only be executed from the merge-docs job?
I have seen that it could be triggered manually, which would be good to check for updates without the need of a change on master.

And will redeploy-website be executed when something on master has changed and succeeded?
This info is missing in the explanation, or i misread something^^

[Confectrician]

We should then clean up the final branch before starting deployment.
there are still jekyll related files in it.

The tmp-master branch looks nice and clean.
That's what i imagined as i opened #665.
Will be a huge effort. 👍

[ghys]

I can delete my prepared master then?

I guess ;) - I took it as an example to build the jenkins-tmp-master branch anyways. It would only have to be renamed (and "jenkins-tmp-final" to "final" or similar).

Will gather-external-docs only be executed from the merge-docs job?
I have seen that it could be triggered manually, which would be good to check for updates without the need of a change on master.

That's discussed a little bit above, maybe once a day and also as part of the distro build.

And will redeploy-website be executed when something on master has changed and succeeded?
This info is missing in the explanation, or i misread something^^

Good point, currently that job is not redeploying the website. Ideally those deploys wouldn't be done too often (because of the cache invalidations I mentioned), so that's still to be refined.

[Confectrician]

Friendly ping over here.
Are we waiting for feedback from other @ghys?
There are quire some readme changes that have been made, so it would be nice to get this process started.

From my view it technically works fine and we can improve it through the corresponding scripts later, if needed.

So it would be nice to start building the stuff continous and not have t cherry pick a whole bunch of commits.

[ghys]

Okay, so here's what I think needs to be done in order:

1. Rename the jenkins-tmp-master and jenkins-tmp-final branches to resp. master and final;
2. Set the master branch as the default (not critical, and it seems only the repo owner i.e. @kaikreuzer can do this);
3. Alter the Jenkins jobs to reflect the branch renames; ideally move them out of the "sandbox" tab;
4. Change the website prepare-docs script to check out the final branch instead of the default;
5. Set up a token at Netlify for the production website and set it up in Jenkins

And also:

• Cherry-pick the commits made since the 2 branches were created and commit them into master
• Change the base branch of other currently open PRs.

And obviously verify everything works. :)

[kaikreuzer]

Just ping me when I shall do (2). I have a very slow and instable Internet access atm, so please do not expect me to help on the other steps, sorry...

[ghys]

Just ping me when I shall do (2).

Enjoy your holidays, it's not urgent ;)

1. Done
2. Todo (https://github.com/openhab/openhab-docs/settings/branches) when everyone's okay with it;
3. Done - I have created a view for those 3 builds (https://openhab.ci.cloudbees.com/view/Documentation/) and removed the "sandbox-" prefixes
4. Created prepare-docs: Check out "final" branch by default website#80 - merged since it looks good
5. Done

To clarify:

• as of now, the website uses the "final" branch for the documentation! (result of 4.)
• triggering https://openhab.ci.cloudbees.com/view/Documentation/job/redeploy-website/ will redeploy the production website (result of 5.). It will be also done after https://openhab.ci.cloudbees.com/view/Documentation/job/gather-external-docs/ but NOT https://openhab.ci.cloudbees.com/view/Documentation/job/merge-docs/, still left for discussion.

Those last 2 Jenkins jobs are still triggered manually for now.

Cherry-pick the commits made since the 2 branches were created and commit them into master

A PR/rebase on GitHub shall be enough: created #733 for that, please review!

[kaikreuzer]

(2) done!

[Confectrician]

Perfect and thanks for the fast reaction.
I will have a look at the pr now.

[Confectrician]

A PR/rebase on GitHub shall be enough: created #733 for that, please review!

And done.

[ghys]

Okay, I'll run the Jenkins jobs to try out the whole process now.

[Confectrician]

Triggering a new build of redeploy-website
Finished: SUCCESS

Looks god so far.
Netlify is building too atm.

[Confectrician]

As i have recognized now the concepts folder is generated automatically too.
Has this been forgotten or do the files get overwritten now?

[ghys]

Looking good indeed!

The newly added SolarEdge binding (added in 43d9795#diff-bdeb28d1d2a493d34aec71b739b9b2ab) is now at: https://www.openhab.org/addons/bindings/solaredge/

As i have recognized now the concepts folder is generated automatically too.
Has this been forgotten or do the files get overwritten now?

No, it's my mistake...
I suppose it's safe to remove it - the next "merge" job will remove it from "final" too so it would have to be recreated by the "gather-external-docs" job before deploying.

[Confectrician]

https://www.openhab.org/docs/concepts/units-of-measurement.html#list-of-units
Has been overwritten by the script and choosen the sh version.
So it should be save to remove.

I will prepare a PR for that.

[ghys]

The maven goal might not be happy if the folder is removed completely, if that's the case we can revert the merged commit on the final branch to restore it.

[Confectrician]

Thanks to all involved. 🙂
That's a huge progress for the docs management.
I think we can close this one unmerged then^^.

Next step will then be some smaller things.
I have created https://github.com/openhab/openhab-docs/milestone/2 some time ago to collect the needed things for completing the website migration.

Feel free to add issues and PRs there.
I am continuing the Readme refactoring now as a first step.

[Confectrician]

Solved through a jenkins implementation.
Thanks again to all involved!