Build front-end dependencies locally

[virtualtam]

This is a follow-up to #754.

@nodiscc wrote:

Is it possible/recommended to fetch 3rd-party eot,css,js... files during the build process? (as opposed to storing them in the git repository)

@ArthurHoaro wrote:

Yes. We could use bower (dependency manager, kind of like composer, for front end libs) and include these in our release archives.

I have a mixed opinion regarding Bower:

• it works fine and fills the role of a dependency manager, yet...
• Node stuff is a PITA to install in a safe way on a server, as one is expected to blindly use the NodeSource repositories:

curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
sudo apt-get install -y nodejs

Nice, ain't it? curl | sudo... When the shell script merely adds the appropriate new APT entry and PGP key.

I wrote an Ansible task a while back to automate this for Debian-based systems:

---
- name: Add APT key for NodeSource repositories
  apt_key: url=https://deb.nodesource.com/gpgkey/nodesource.gpg.key

- name: Add NodeSource 6.x repository
  apt_repository:
    repo: 'deb https://deb.nodesource.com/node_6.x {{ ansible_distribution_release }} main' 
    update_cache: yes

- name: Install Node.js
  apt:
    name: nodejs
    state: latest

- name: Set NODE_PATH under /etc/environment
  lineinfile:
    dest: /etc/environment
    regexp: '^NODE_PATH'
    line: 'NODE_PATH="/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript:/usr/local/lib/npm/lib/node_modules"'

- name: Install Bower and Gulp globally
  npm: name={{ item }} global=yes state=present
  with_items:
    - bower
    - gulp

This would be easy to add to Docker images and Shaarli Wiki, but I fear users will have a hard time to get it working, whether they install Node on a global- or user-scope.

[ArthurHoaro]

Oh yes, NodeJS. If any, we should probably go for a PHP based solution, to avoid installing additional software to the system. I've never used any, but a quick search gave me

• BowerPHP which is (surprise!) a PHP implementation of Bower
• composer-asset-plugin: Composer plugin which handles front end dependencies.

While it may seems like using a sledgehammer to crack a nut, both solutions can be integrated to grab front end dependencies when running composer install/update, which has to be run anyway.

[virtualtam]

I'd rather keep PHP & front-end dependency management a separate step, and am not too fond of declaring front-end libs within composer.json as:

• they're usually heavy to resolve
• it would add unnecessary operations to unit tests

The following sequence would be convenient:

• define front-end dependencies within bower.json (the standard & Cap'n Obvious-approved way)
• add BowerPHP to the Composer dependencies and define a script entry, e.g. composer bowerphp <args>
• let Composer resolve dependencies as usual
• let BowerPHP resolve front-end dependencies
• ?
• profit!

[ArthurHoaro]

It sounds good, it takes one additional step to install the dev version, but without adding a manual installation. I'll wait for @nodiscc opinion, and look into it.

PS: your meme level is highly improving 👍

[nodiscc]

Why not use simple wget commands in the Makefile to fetch front-end dependencies?

Otherwise BowerPHP looks like a good compromise. The process stays automatic and the dev environment stays clear of Nodejs.

(btw on debian the proper installation method would be apt install npm; npm install -g bower, so no curl | sudo bash ... )

[virtualtam]

Why not use simple wget commands in the Makefile to fetch front-end dependencies?

Mainly 'cause dependencies come with dependencies that come with dep... Hence a dependency manager ;-)

btw on debian the proper installation method

which would leave you with quite an old NPM distribution on Debian hosts, and an okay one on recent Ubuntu versions ^^

[nodiscc]

quite an old NPM distribution
npm package in Debian stable/testing/unstable: 1.4.21
upstream version 4.1.2

eww ok, I had no idea. @ArthurHoaro did you use bower for #754? (hence my question about using wget, I thought you had manually downloaded the libs).

Ok for BowerPHP.

[ArthurHoaro]

No I didn't, that's why all dependencies appear in the diff. I'll look into it then.

[ArthurHoaro]

Bower clones git repos. PureCSS repo don't include a compiled version. To build it, you need npm + grunt.

I think we should drop this. There is not much gain, anyway.

[virtualtam]

npm/bower/grunt/gulp builds can be automated, but it's a bit of a pain (and downloads the whole Internet. twice. or more?) and requires a recent NodeJS distribution; I'm OK for versioning 3rd-party frontend deps as long as we have an entrypoint to easily upgrade and rebuild them (e.g. gruntfile or gulpfile)

[ArthurHoaro]

While I'd prefer to have front end dependencies handled by a dependency manager, my issue with this approach is that installing Shaarli from git source would then require the whole npm stack. For instance, we won't be able to use the current demo as it is hosted on a shared PHP server.

[ArthurHoaro]

I've been thinking about this issue and I have a few propositions. While it seems to be a good practice to do so, PureCSS is the only front dependency we have which doesn't provide a compiled/minified version within its repo. So suggestions, we keep our idea of using BowerPHP and either:

• host or find a repo containing PureCSS compiled versions (it's really stable, there have been something like 3 releases in 2 years).
• download the compiled version from their CDN using a composer script or the makefile

Or we go for a proper grunt build as @virtualtam suggested. This solution is cleaner, but keep in mind that it increases complexity for eventual contributors.

[virtualtam]

We could start by introducing lint and test tools in the development process, document the installation and usage process and caveats, and eventually use it to manage frontend assets (as we did with Composer: first as a testing tool, then for production dependencies).

This might change how people manage their Shaarli installation:

• release archives => shared hosting and newcomers
• Docker images => dedicated servers, cloud™
• Git sources => developers, testers, advanced users

[nicolasdanelon]

Bower is not a good idea, npm install (or yarn add with the awesome lock file) is just enough

[ArthurHoaro]

@virtualtam OK, you've convinced me. I've no experience in front end tests, but it seems interesting. As I said, it'll be a bit more complicated for people wanting to contribute to the code, but yes as you said, they're developers and likely able to install packages. I'd like @nodiscc validation.

@nicolasdanelon I thought that npm was for JS dependencies and bower for any assets, but looking at npm repos with our dependencies you might be right. In any case, we need grunt for asset build.
I've never heard of yarn, any reason we should use it?

[nicolasdanelon]

Please read the following links about yarn (:

• http://blog.npmjs.org/post/151660845210/hello-yarn
• https://yarnpkg.com/en/
• https://yarnpkg.com/en/docs/yarn-lock

This is way we dont need to add gulp (or grunt).

• https://medium.com/swlh/you-might-not-need-gulp-js-89a0220487dd#.mp3o9ppks

TL;DR
Grunt and gulp provide pipelines and a clean file but we can do a lot of things like 'compile' SASS to CSS, minify CSS and JS with simple js files. Like the medium post does

Yarn (with the lock file) allow us to lock dependencies in a specific version. beyond 0.2.3 =P

@ArthurHoaro
I can help with this task if you want =) ping me in gitter and we can debate about this, if you want

[virtualtam]

@ArthurHoaro I've got a (very) limited experience with frontend testing, here are some useful tools I was recommended / know of / wrote Gulp tasks and Jenkins integration for:

• ESLint to lint JS / ES 6 code
• HTMLHint to check static HTML and LESS/SASS/whatever HTML templates
• CSScomb to automagically format CSS files (the config builder is impressive in its simplicity)
• Stylelint to lint CSS

AFAIK projects use either Grunt or Gulp to script tasks, both tools provide similar functionality and have their pros and cons:

• https://ponyfoo.com/articles/gulp-grunt-whatever
• https://medium.com/@preslavrachev/gulp-vs-grunt-why-one-why-the-other-f5d3b398edc4#.cu94pz23j

Yarn is a package management alternative to NPM. Though it has a myriad stargazers on Github, it is still quite new to the dev scene and has limitations compared to vanilla NPM.

@nicolasdanelon Does yarn provide easy integration with CI tools? I remember writing tasks to get Checkstyle or XUnit/JUnit formatted reports being quite tedious...

• https://www.npmjs.com/package/gulp-htmlhint-checkstyle-file-reporter/ (special mention for this one)
• https://www.npmjs.com/package/gulp-stylelint-checkstyle

[nicolasdanelon]

yarn is for us. yarn use the package.json file as npm. yarn.lock file will help to lock dependencies (:
running commands like npm start or npm build (this can be tailor made in the packages.json)

we avoid to use wrapped packages. take a look to the source code of every gulp or grunt plugin and you'll find out that all the plug-ins are wrappers of one, two or six thousand of npm packages.

[nicolasdanelon]

BTW gulp-stylelint-checkstyle is deprecated :/

[nodiscc]

I'm ok with either solution, but I'd prefer something that doesn't require adding third-party repositories to my APT sources.list, or installing a whole new stack of tools to build CSS/JS that we will essentially never edit/customize.

Downloading prebuilt assets from upstream git repos/CDNs seems to be the simplest approach, so in the first time I think we should implement that method (keep it simple), and as @virtualtam suggested, implement a more complex approach, using a frontend dev toolchain over time (tests first, then fetching deps - if it actually has benefits for maintenance/dev?)

To summarize, back to this:

Why not use simple wget commands in the Makefile to fetch front-end dependencies?

Mainly 'cause dependencies come with dependencies that come with dep... Hence a dependency manager ;-)

So why not download prebuilt versions (from git repos/CDNs) of all frontend libs/files/dependencies introduced in https://github.com/shaarli/Shaarli/pull/754/files from upstream from the Makefile?

Does this method have drawbacks ?

[nicolasdanelon]

@nodiscc if you want to install nodejs you can simply use https://github.com/creationix/nvm and add the version you want to your local enviroment. this will add node and npm without touch you APT sources.

[tcitworld]

If you're going this way you might as well do it fully from the start with the following stack :

Dependencies management : yarn. Faster than npm, locking dependencies, etc. Hard to find something on bower that's not on the npm directory these days anyway. Tasks are done through npm scripts.
Bundling : webpack 2. More simple than gulp and grunt plugins start to be outdated
JS Compiler : babel. ES6 Anyone ?
JS Linting : eslint. My favourite config is Airbnb.
HTML Liniting : HTMLHint. Didn't use it. Nice
CSS Linting : stylelint. Nothing else.
CSS Compiling : Postcss with at least autoprefixer. To get rid of vendor prefixes in your code

Add js & css linting to the travis jobs and you're good to go.

[virtualtam]

TODO:

• Manage frontend dependencies with npm/yarn and webpack  #1072 - Manage dependencies with Yarn and Webpack
• Docker: build frontend dependencies with node and yarn #1113 - Docker images
• Use Travis stages to run JS tests separately  #1116 - Introduce Travis build stages
• Lint:
  • Manage frontend dependencies with npm/yarn and webpack  #1072 - ESlint
  • Reformat SCSS to SASS format and run SASSLint in CI #1135 - Stylelint

[ArthurHoaro]

I've started to work on SASSLint, but it takes forever.

[virtualtam]

Closing as resolved, thanks everyone for the feedback on JS tooling, and @ArthurHoaro for working on this! :)

[ArthurHoaro]

You're welcome, and yes, thanks everyone, I learnt a bunch of things in the process. :)