Dockerfile (with some convenience Makerules) for development, testing and serving #461

vidbina · 2017-08-08T06:14:40Z

@johndmulhausen started this with #423 which led to a subsequent PR into his set of changes johndmulhausen#1. Since his master branch is lagging, the diff contains a lot of noise (commits that are unrelated to the Dockerization things) as pointed out by @benbalter. This PR minimizes the diff to something more suited for scrutiny.

Changes

Cleaned up the Dockerfile to the bare minimum for using it as a test/dev environment and even using is as a server to view any jekyll-site in a github-pages-like environment
Set up a bundle override for the github-pages gem to point to the local directory
Added some make rules to simplify usage

Usage

Use make image to build the github-pages image
Use make shell to produce a bash shell
- run the scripts inside that shell by simply running
  - ./script/bootstrap, usually run this first to ensure a fresh
    bundle install (note that with the make shell rule mounting the pages-gem directory into the container's /gh-pages path (the bundler override for the github-pages gem), one may test changes made to the directory on the host, rather than being limited to having to rebuild the image on every file change 😉)
  - ./script/cibuild
  - ./script/test-site
  - ./script/fmt
Run the server by (alternative approaches in order of preference):
- prepending an assignment to SITE to the make server rule, e.g: SITE=../path-to-jekyll-site make server
- exporting SITE first and then calling make server, e.g.: export SITE=../path-to-jekyll-site; make server. After the export, one may trigger the server by simply running make server, but the SITE assignment becomes less explicit. When you change to another terminal that perhaps already has the variable SITE defined, perhaps even for other purposes, one will basically be altering the terminal's state.

Demo

See https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#apt-get

deeped hierarchy isn't really necessary 😉

https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#cmd

- Use `make image` to build the github-pages image - Use `make shell` to produce a bash shell - run the scripts inside that shell by simply running - `./script/bootstrap`, usually run this first to ensure a fresh `bundle install` - `./script/cibuild` - `./script/test-site` - `./script/fmt` - Run the server by: - exporting `SITE` first and then calling `make server`, e.g.: `export SITE=../path-to-jekyll-site; make server` - prepending an assignment to `SITE` to the `make server` rule, e.g: `SITE=../path-to-jekyll-site make server`

johndmulhausen · 2017-08-29T22:37:12Z

This is great; I'll close my pr :)

benbalter · 2017-08-30T16:19:30Z

@vidbina this is great. I'd love to get this merged. Would you be able to add documentation? If so, I believe this should be big improvement.

benbalter · 2017-08-30T16:21:07Z

Dockerfile

+WORKDIR /gh-pages
+
+RUN bundle config local.github-pages /gh-pages
+RUN bundle install


We may need to modify the Gemfile to add the Gem to the jekyll_plugins group so that it can hook earlier into the bootstrap process and set defaults, e.g., gem "github-pages", group: :jekyll_plugins

I'm not sure I totally understand what you mean with this, please elaborate and I'll try to get on it asap 😉.

My GitHub Pages projects do contain the gem "github-pages", group: :jekyll_plugins line, but then I believe I should then probably fire up "bundle exec jekyll server ..." instead of just "jekyll serve ...".

Is there any logic performed on the jekyll_plugins group? Is it not only there for organization?

Is there any logic performed on the jekyll_plugins group? Is it not only there for organization?

The Jekyll Plugins group allows the GitHub Pages Gem to get bootstrapped earlier in Jekyll's process and set configuration defaults and constraints to match GitHub Pages. I'm struggling to load a gemspec into a gem group, so we may need to create a new / modify the Gemfile for docker.

Have you guys figures this out in the end? It seems to me that jekyll in the container ignores github-pages entirely and just uses its own default configuration, making this whole make server and contrib/func.sh thing rather useless. :-/

vidbina · 2017-09-02T22:13:55Z

@johndmulhausen and @benbalter can you check if the documentation I added makes any sense at all?!? Thanks in advance 🙇

wolfgang42 · 2017-10-02T18:59:31Z

This image looks very exciting for allowing local testing of GitHub Pages sites. I've been testing it on the GitHub Pages examples and I've noticed some examples where it produces errors. All of these were produced by cloning the site and then running SITE=... make server.

https://github.com/jekyll/jekyll

  Liquid Exception: invalid byte sequence in US-ASCII in _layouts/redirect.html
jekyll 3.5.2 | Error:  invalid byte sequence in US-ASCII

https://github.com/github/government.github.com

jekyll 3.5.2 | Error:  Could not find a JavaScript runtime. See
https://github.com/rails/execjs for a list of available runtimes.

https://github.com/facebook/react/

  Dependency Error: Yikes! It looks like you don't have redcarpet or one of its 
dependencies installed. In order to use Jekyll as currently configured, you'll
need to install this gem. The full error message from Ruby is: 'cannot load such
file -- redcarpet' If you run into trouble, you can find helpful resources at 
https://jekyllrb.com/help/!
  Conversion error: Jekyll::Converters::Markdown encountered an error while
converting '_posts/2013-06-02-jsfiddle-integration.md':
                redcarpet
         ERROR: YOUR SITE COULD NOT BE BUILT:
                ------------------------------------
                redcarpet

I'm not sure if this is what's causing the above errors, but it looks like the way Jekyll has been set up doesn't match how GitHub Pages does rendering: Pages runs Jekyll in 'safe mode' with a specific whitelist of plugins, so sites which work in this container might not work on Pages if they're using custom plugins. I've found the settings for all of this in configuration.rb but I'm not sure how to hook that into Jekyll.

vidbina · 2017-10-12T22:57:49Z

Hey @wolfgang42, thanks for this. I'll take some time soon enough to look into this issue. I've been flooded with some other chores meanwhile so I'll get back at you as soon as I know more 😉. Thanks for listing the specific examples and the errors they produce.

parkr

Wow! I really, really like this! I added a few requests below, but I'd definitely be happy merging this. The demo really shows it off beautifully. 🎉 ❤️ Very well done.

parkr · 2017-10-20T17:53:43Z

Dockerfile

+    git \
+    make
+
+COPY . /gh-pages


Due to how Jekyll.sanitized_path works, let's put this in a subdirectory 2 levels deep, e.g. /gh/pages or /src/gh-pages. Otherwise, if you have a file called gh-pages.html, you'll get an error. It's confusing so let's tackle that before merging.

Something like cbe4cdf?

parkr · 2017-10-20T17:55:19Z

Makefile

+# exposing it using `expose SITE="../path-to-jekyll-site"` prior to calling or
+# by prepending it to the make rule e.g.: `SITE=../path-to-site make server`
+server:
+	ls "${SITE}" >/dev/null && \


Can an error message be printed here if the SITE variable doesn't exist?

Will an echo suffice? 03709aa

parkr · 2017-10-20T17:55:38Z

Makefile

+		-p 4000:4000 \
+		-v ${PWD}:/gh-pages \
+		-v `realpath ${SITE}`:/site \
+		-w /site \


Can we use /src/site (a subdirectory 2 levels deep) for reasons stated above?

Done 😉 9d8fe18

parkr · 2017-10-20T17:56:36Z

README.md

+**Note:** the `github-pages` function may be enabled by sourcing func.sh. This can be done by appending
+
+  ```bash
+  source $PATH_TO_THIS_DIRECTORY/func.sh


This is great! Can it go in a contrib/ subdirectory instead of the root of the project? contrib/docker maybe?

Done 😉 f4a8d87

In response to github#461 (comment)

In reponse to github#461 (comment)

In response to github#461 (comment)

vidbina · 2017-11-26T22:45:22Z

@parkr it took forever for me to get around to this, apologies. It's only now when I'm chilling on an island 🌴 that I can sit for a few minutes and catch up on items on my to-do list. I still intent to address the issues pointed out by @benbalter and @wolfgang42 but it could take a little while. Just bear with me and rest assured that I've heard and acknowledged the concerns loud and clear 👂 😉

vidbina · 2017-11-27T18:12:39Z

@wolfgang42 just called github-pages inside the jekyll directory resulting to the following:

            Source: /src/site
       Destination: /src/site/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
             Error: could not read file /src/site/lib/site_template/_posts/0000-00-00-welcome-to-jekyll.markdown.erb: Invalid date '<%= Time.now.strftime('%Y-%m-%d %H:%M:%S %z') %>': Document 'lib/site_template/_posts/0000-00-00-welcome-to-jekyll.markdown.erb' does not have a valid date in the YAML front matter.
             ERROR: YOUR SITE COULD NOT BE BUILT:
                    ------------------------------------
                    Invalid date '<%= Time.now.strftime('%Y-%m-%d %H:%M:%S %z') %>': Document 'lib/site_template/_posts/0000-00-00-welcome-to-jekyll.markdown.erb' does not have a valid date in the YAML front matter.

Not sure why I doesn't evaluate the the erb template. Will look into it.

Running github-pages inside the government.github.com directory resulted to:

Configuration file: /src/site/_config.yml
            Source: /src/site
       Destination: /src/site/_site
 Incremental build: disabled. Enable with --incremental
      Generating... 
  Liquid Exception: invalid byte sequence in US-ASCII in _layouts/redirect.html
jekyll 3.6.2 | Error:  invalid byte sequence in US-ASCII

for which I initially figured that it could be related to the old LC_ALL var issue, but I have set LANG, LANGUAGE and LC_ALL to no avail.

For facebook/react I haven't found a jekyll site. Perhaps the link you referenced doesn't point to the static page for React?

Will have to do a bit of digging tomorrow to figure out what's up but perhaps @benbalter or @parkr have some insights that could nudge me in the right direction.

wolfgang42 · 2017-11-27T18:34:06Z

@vidbina It looks like the React Jekyll site was deleted last month: facebook/react#10941, facebook/react@887ccf2

jekyll/jekyll#2327

vidbina · 2017-11-28T18:34:18Z

Hey @wolfgang42. The Jekyll site now builds as long as you use the docs subdir as the site directory

github-pages /path/to/jekyll-repository/docs or SITE=/path/to/jekyll-repository/docs make server will do just fine.

The invalid byte sequence issue was resolved by configuring the locale of the container through the setting of the LANG, LC_ALL and LANGUAGE variables, modification of the /etc/locale.gen file and running of the locale-gen command. I probably overdid it but was too lazy to figure out which ones of these operations constitute the bare minimum to get the setup working.

As for the execjs issue, that was resolved by apt-get install nodejs as recommended in the troubleshooting page and jekyll#2327.

Let me know if that solves the issues and bump @parkr and @benbalter to take it from there once you're convinced that it works well enough 😉.

vidbina

Hey @benbalter, I've more or less addressed every issue you highlighted. Let me know if this suffices, otherwise let me know what still needs to be improved. ✌️

johndmulhausen and others added 8 commits March 9, 2017 13:59

Create Dockerfile

21b1871

Merge branch 'master' into dockerize

c192457

update ruby in container

4ac341c

follow recommendations for apt usage in image

76e40e0

See https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#apt-get

use /src as workdir

ab2a31f

deeped hierarchy isn't really necessary 😉

mount gh-pages + setup local override

4f97a96

serve site by default

1ffa175

https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#cmd

vidbina mentioned this pull request Aug 8, 2017

Dockerize johndmulhausen/pages-gem#1

Open

vidbina changed the title ~~Dockerization~~ Docker environment (with some convenience Makerules) for development, testing and serving Aug 8, 2017

vidbina changed the title ~~Docker environment (with some convenience Makerules) for development, testing and serving~~ Dockerfile (with some convenience Makerules) for development, testing and serving Aug 8, 2017

benbalter requested a review from parkr August 8, 2017 13:32

Merge branch 'master' into dockerize

46e410d

benbalter reviewed Aug 30, 2017

View reviewed changes

vidbina and others added 9 commits September 2, 2017 22:44

default to /site as workdir after bundle install

5733af2

sourceable func for bash and zsh

2bb738f

1st attempt at documenting the docker approach

209aa53

Merge branch 'master' into dockerize

a7a7e73

Merge documentation and shell func

1381be0

tried to be clearer... hopefully it worked

0a6fc82

which function?!?

6cfbab0

how-to enable func.sh + moved some bits to asides

a31946e

forgot to close

3b30541

parkr approved these changes Oct 20, 2017

View reviewed changes

Merge remote-tracking branch 'upstream/master'

21101e2

vidbina added 4 commits November 26, 2017 22:55

docker image: moved /gh-pages to /src/gh/pages-gem

cbe4cdf

In response to github#461 (comment)

docker image: echo err on undef or invalid $SITE

03709aa

In reponse to github#461 (comment)

docker image: moved /site to /src/site

9d8fe18

In response to github#461 (comment)

docker image: moved func.sh to contrib/func.sh

f4a8d87

In response to github#461 (comment)

vidbina and others added 3 commits November 27, 2017 09:21

Merge branch 'master' into dockerize

bc487a3

Merge branch 'upstream/master' into dockerize

8b6a3d2

docker helper: better shebang

9d3ffff

vidbina force-pushed the dockerize branch from 2e5a0c6 to 2cb9894 Compare November 27, 2017 17:34

docker helper: set uid and gid

be1ac57

vidbina force-pushed the dockerize branch from 2cb9894 to be1ac57 Compare November 27, 2017 17:42

vidbina added 3 commits November 28, 2017 19:19

docker image: set locale

7753628

docker image: installed nodejs

72f81a3

jekyll/jekyll#2327

docker image: minimized locale ops to single RUN

c09e851

Merge branch 'master' into dockerize

b3b4b84

vidbina commented Dec 11, 2017

View reviewed changes

Merge branch 'master' into dockerize

e8eb664

benbalter merged commit e7eb0a9 into github:master Mar 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dockerfile (with some convenience Makerules) for development, testing and serving #461

Dockerfile (with some convenience Makerules) for development, testing and serving #461

vidbina commented Aug 8, 2017 •

edited

Loading

johndmulhausen commented Aug 29, 2017

benbalter commented Aug 30, 2017

benbalter Aug 30, 2017

vidbina Sep 2, 2017

benbalter Sep 5, 2017

liskin Apr 23, 2020

vidbina commented Sep 2, 2017

wolfgang42 commented Oct 2, 2017

vidbina commented Oct 12, 2017

parkr left a comment

parkr Oct 20, 2017

vidbina Nov 26, 2017 •

edited

Loading

parkr Oct 20, 2017

vidbina Nov 26, 2017

parkr Oct 20, 2017

vidbina Nov 26, 2017

parkr Oct 20, 2017

vidbina Nov 26, 2017

vidbina commented Nov 26, 2017 •

edited

Loading

vidbina commented Nov 27, 2017

wolfgang42 commented Nov 27, 2017

vidbina commented Nov 28, 2017 •

edited

Loading

vidbina left a comment •

edited

Loading

Dockerfile (with some convenience Makerules) for development, testing and serving #461

Dockerfile (with some convenience Makerules) for development, testing and serving #461

Conversation

vidbina commented Aug 8, 2017 • edited Loading

Changes

Usage

Demo

johndmulhausen commented Aug 29, 2017

benbalter commented Aug 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vidbina commented Sep 2, 2017

wolfgang42 commented Oct 2, 2017

vidbina commented Oct 12, 2017

parkr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vidbina Nov 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vidbina commented Nov 26, 2017 • edited Loading

vidbina commented Nov 27, 2017

wolfgang42 commented Nov 27, 2017

vidbina commented Nov 28, 2017 • edited Loading

vidbina left a comment • edited Loading

Choose a reason for hiding this comment

vidbina commented Aug 8, 2017 •

edited

Loading

vidbina Nov 26, 2017 •

edited

Loading

vidbina commented Nov 26, 2017 •

edited

Loading

vidbina commented Nov 28, 2017 •

edited

Loading

vidbina left a comment •

edited

Loading