Add Dev Container Configuration Files #9039

benz0li · 2023-08-30T11:12:05Z

@jgm You may test online with GitHub Codespaces at https://github.com/benz0li/pandoc/tree/add-dev-containers

See also https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository#creating-a-codespace-for-a-repository

Screenshot using the default configuration:

benz0li · 2023-08-30T11:32:23Z

I tried to be minimally invasive, i.e.

Modified one line in Pandoc's .gitignore file
- Created a separate .gitignore file in the .devcontainer folder
Did not touch the LICENCE of Pandoc (the software)
- Created a separate LICENCE file in the .devcontainer folder
  ℹ️ Mentioning that The code in this directory is not part of Pandoc (the software) and,...

I hope this is fine and the instructions in the additional README.md in the .devcontainer folder are sufficient.

@jgm What do you think? Any objections? Or additions?

There are currently four configurations available:

P.S.: Tags latest and recommended come from https://www.haskell.org/ghcup/install/#supported-tools
ℹ️ Default (for use with Stack) is the default image. All configurations are based on the glcr.b-data.ch/ghc/ghc-musl image.

benz0li · 2023-08-30T12:07:15Z

Well, well, well... Somehow, statically linking does not work with GitHub Codespaces (x86_64), but does work on my MacBook Pro (AArch64) with the Dev Container using Docker Desktop. 🤷

~~Maybe @microsoft can provide more insight on this.~~

jgm · 2023-08-30T15:10:24Z

@jgm You may test with GitHub Codespaces at https://github.com/benz0li/pandoc/tree/add-dev-containers

How do I do that? I've never used Codespaces. It wasn't obvious from the link.

My main concern about this is that it's a lot of code that would need to be updated periodically, e.g. with ghc changes. Does it need to be in the pandoc repository itself?

benz0li · 2023-08-30T15:32:06Z

How do I do that? I've never used Codespaces. It wasn't obvious from the link.

https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository#creating-a-codespace-for-a-repository ff should explain.
ℹ️ You may need to add Credit card information to your account beforehand...

My main concern about this is that it's a lot of code that would need to be updated periodically, e.g. with ghc changes.

The default config (GHC 9.4.5 (for use with Stack)) should be updated according to Pandoc's stack.yaml.
ℹ️ Stackage lts-21.7 = GHC 9.4.5

For the other configs we could agree on updating 'once a week' / 'every two weeks' / 'once a month'.
ℹ️ Because GHC is pinned to the minor version, there are not so many updates.

Stack does not update that frequently. It is a different story with the Haskell Language Server, though.
👉 But in order to build Pandoc you don't need to have the latest version of those.

Does it need to be in the pandoc repository itself?

AFAIK Yes.

benz0li · 2023-08-30T15:40:32Z

In order for this to work, I must keep the GHC musl Container Registry up to date.

For example: There was a new HLS release yesterday. This means I am going to trigger a build for HLS v2.2.0.0 for each Dev Container's GHC versions asap.
👉 But this does not mean that their HLS version has to be updated immediately.

benz0li · 2023-08-30T15:41:49Z

Does it add any value to Pandoc itself: No.

@jgm am absolutely fine if you do not want to add Dev Containers to the ~~image~~ repository. That is entirely up to you.

jgm · 2023-08-30T16:41:32Z

I'm leaning against adding this just because the maintenance burden is nontrivial. But I'll leave this open; others may want to comment.

cagix · 2023-08-30T17:03:23Z

This would be an interesting addition for people using tablet devices like iPads for doing actual work. However, due to the somewhat messy configuration (at least to me) regarding spending limits and who will pay for what when, this could easily produce unwanted costs ...

benz0li · 2023-08-31T06:23:23Z

Dev Container configurations (devconatainer.json) are updated in the following cases:

Default config: Whenever a stack.yml update affects the GHC version. (mandatory)
Other configs: A new major/minor version of the latest GHC is released. (optional)
Other configs: A new version of the Haskell Language Server (HLS) is released. (optional)
Other configs: A different version of GHC is tagged as recommended¹. (optional)
Other configs: A different version of Stack is tagged as recommended¹. (optional)

👉 Updating the default config's devcontainer.json would be your (@jgm's) responsibility. The other configs devcontainer.json will be updated whenever you/I/we feel like it.

The GHC.Dockerfile is a work in progress and will reach maturity once a new release of Stack is tagged ~~both~~ latest and recommended.

That is it for now. Let us keep this a draft, then.

There will soon be an official, statically linked binary release of Stack for both Linux/x86_64 and Linux/AArch64: commercialhaskell/stack#6160 (comment).
👉 Official bindists for Alpine Linux (AArch64) will not be available anytime soon, though: https://gitlab.haskell.org/ghc/ghc/-/issues/23482

According to https://www.haskell.org/ghcup/install/#supported-tools ↩ ↩²

benz0li · 2023-09-07T06:53:30Z

FYI I downgraded package pkgconf to v1.8.1 in all glcr.b-data.ch/ghc/ghc-musl images based on Alpine 3.{17,18}: benz0li/ghc-musl@c492963

Due to haskell/cabal#8923

benz0li · 2023-09-14T11:39:04Z

@jgm Reorganise files = Point from CONTRIBUTING.md to a new wiki page 'Dev Containers' with the following content:

# Dev Containers

Pandoc's Dev Containers provide the following tools:

- The
  [Haskell Toolchain](https://www.haskell.org/ghcup/install/#supported-tools)
  ([GHC](https://www.haskell.org/ghc),
  [Cabal](https://cabal.readthedocs.io),
  [Stack](https://docs.haskellstack.org), and
  [HLS](https://haskell-language-server.readthedocs.io))
- [Git](https://git-scm.com)
- [HLint](https://hackage.haskell.org/package/hlint)
- [yamllint](https://yamllint.readthedocs.io)
- [ShellCheck](https://www.shellcheck.net)
- [hadolint](https://github.com/hadolint/hadolint)

The tools in the Haskell Toolchain are installed at `/usr/local/bin`<sup>[1](#path),[2](#alt-path)</sup>.

| :information_source: Executables installed with Cabal (at `$HOME/.cabal/bin` or `$HOME/.local/bin`) or Stack or Pip (at `$HOME/.local/bin`)<br>take precedence over the same executable installed at `/usr/local/sbin`, `/usr/local/bin`, etc.|
|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

[VS Code](https://code.visualstudio.com) is used as IDE, with the following
extensions pre‑installed:

- [Haskell](https://marketplace.visualstudio.com/items?itemName=haskell.haskell)
- [GitHub Pull Requests and Issues](https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github)
- [GitLens — Git supercharged](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens)
  - Pinned to v11.7.0 due to unsolicited AI content in recent versions.
- [Git Graph](https://marketplace.visualstudio.com/items?itemName=mhutchie.git-graph)
- [ShellCheck](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck)
- [hadolint](https://marketplace.visualstudio.com/items?itemName=exiasr.hadolint)
- [EditorConfig](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig)
- [Resource Monitor](https://marketplace.visualstudio.com/items?itemName=mutantdino.resourcemonitor)

## Parent images

Pandoc's Dev Containers are derived from the same docker images used to build
the *statically linked* Linux amd64 and arm64 binary releases of pandoc
itself.

The parent images are multi‑arch (`linux/amd64`, `linux/arm64/v8`)
<nobr>*GHC musl*</nobr> images. They are based on Alpine Linux (that is
[musl libc](https://musl.libc.org) and [BusyBox](https://www.busybox.net)).

They contain *unofficial* binary distributions of GHC (that is, ones not
released by the GHC developers). That is because:

1. the official GHC binary distributions for Alpine Linux/x86_64 have known
   bugs; and
1. there are no official binary distributions for Alpine Linux/AArch64.

Stack's global configuration (`/etc/stack/config.yaml`) sets
<nobr>`system-ghc: true`</nobr> and <nobr>`install-ghc: false`</nobr>. That
ensures that only the GHC available in the Dev Containers is used.

## Usage

For local/remote usage with VS Code, please follow the instructions at
[Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).

For use with Github Codespaces, please follow the instruction at
[Creating a codespace for a repository](https://docs.github.com/en/codespaces/developing-in-codespaces/creating-a-codespace-for-a-repository#creating-a-codespace-for-a-repository).

### Persistence

Data in the following locations is persisted:

1. The user's home directory (`/home/vscode`<sup>[3](#home)</sup>)
1. The Dev Container's workspace (`/workspaces`)

This is accomplished either via a *volume* or *bind mount* (or *loop device*
on Codespaces) and is preconfigured.

| :information_source: **Codespaces: A 'Full Rebuild Container' resets the home directory!**<br>This is never necessary unless you want exactly that. |
|:----------------------------------------------------------------------------------------------------------------------------------------------------|

## Install pandoc

### Using cabal

Addendum to [Installing pandoc &gt; Compiling from source &gt; Quick cabal method](https://github.com/jgm/pandoc/blob/main/INSTALL.md#quick-cabal-method):

Use
<nobr>
`cabal build --enable-executable-static --ghc-options '-optl-pthread' pandoc-cli`
</nobr> to
build a *statically linked* `pandoc` executable that can run on any Linux
machine of the same architecture.

### Using stack

See [Installing pandoc &gt; Compiling from source &gt; Quick stack method](https://github.com/jgm/pandoc/blob/main/INSTALL.md#quick-stack-method).

You may try
<nobr>
`stack build --ghc-options '-static -optl-static -optl-pthread' pandoc-cli`
<sup>[4](#bug)</sup>.
</nobr> to
build a *statically linked* `pandoc` executable.  
:information_source: This works on Alpine Linux/AArch64 but not x86_64.

## Haskell Language Server (HLS)

Choose <nobr>`Manually via PATH`</nobr> when asked the following question:

<img width="520" alt="Manage HLS" src="https://github.com/jgm/pandoc/assets/20233085/8d054f29-299d-4b7f-b7db-3287cafe1587">

---

<a name="path">1</a>: `PATH=$HOME/.cabal/bin:$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`

<a name="alt-path">2</a>: Installed at `/usr/bin` for GHC versions 9.2.x.

<a name="home">3</a>: Alternatively for the root user (`/root`). Use with
Docker/Podman in *rootless mode*.

<a name="bug">4</a>: `--enable-executable-static --ghc-options '-optl-pthread'`
is supposed to be equivalent to
`--ghc-options '-static -optl-static -optl-pthread'`. But somehow it is not.

benz0li · 2023-09-15T08:47:17Z

I'm leaning against adding this just because the maintenance burden is nontrivial.

@jgm I could reduce to the default Dev Container. The one that is supposed to work with the stack.yml.
ℹ️ Stackage lts-21.7 = GHC 9.4.5.

Stack is providing just two Dev Containers: For stack.yaml (Default) and stack-ghc-9.6.2.yaml (GHC 9.6.2).

Cross reference: commercialhaskell/stack#6228

- Install Stack from GitHub releases

benz0li · 2023-10-11T09:31:32Z

@jgm Commit 50e05dc also means that image glcr.b-data.ch/ghc/ghc-musl:9.8.1 is available.

- Update HLS to v2.4.0.0

- Update Stack to v2.11.1 - Keep GHC 9.2 as recommended - haskell/ghcup-hs@2c3148a is inconsistent

benz0li · 2023-11-15T07:45:00Z

Closed due to lack of interest.

jgm · 2023-11-15T16:22:07Z

Have you been able to compile pandoc with ghc 9.8.1?

benz0li · 2023-11-15T19:18:00Z

Have you been able to compile pandoc with ghc 9.8.1?

Yes.

jgm · 2023-11-17T20:41:58Z

Did you need special cabal flags? When I tried, it didn't seem that all the dependencies had compatible bounds.

benz0li · 2023-11-18T07:51:09Z

Did you need special cabal flags?

--allow-newer

When I tried, it didn't seem that all the dependencies had compatible bounds.

cabal build --allow-newer --enable-executable-static --ghc-options '-optl-pthread' pandoc-cli using image glcr.b-data.ch/ghc/ghc-musl:9.8.1 works fine:

$ docker run --rm -ti glcr.b-data.ch/ghc/ghc-musl:9.8.1 bash
605357ef37ca:/# git clone https://github.com/jgm/pandoc.git
Cloning into 'pandoc'...
remote: Enumerating objects: 123440, done.
remote: Counting objects: 100% (168/168), done.
remote: Compressing objects: 100% (119/119), done.
remote: Total 123440 (delta 57), reused 99 (delta 33), pack-reused 123272
Receiving objects: 100% (123440/123440), 59.78 MiB | 15.93 MiB/s, done.
Resolving deltas: 100% (78660/78660), done.
605357ef37ca:/# cd pandoc
605357ef37ca:/pandoc# cabal update
Config file path source is default config file.
Config file not found: /root/.config/cabal/config
Writing default configuration to /root/.config/cabal/config
Cloning into '/pandoc/dist-newstyle/src/typst-hs-659aa3821678255'...
remote: Enumerating objects: 5591, done.
remote: Counting objects: 100% (409/409), done.
remote: Compressing objects: 100% (173/173), done.
remote: Total 5591 (delta 243), reused 388 (delta 228), pack-reused 5182
Receiving objects: 100% (5591/5591), 7.12 MiB | 14.24 MiB/s, done.
Resolving deltas: 100% (3957/3957), done.
HEAD is now at 71c2494 Further improvements in parsing attachments.
Downloading the latest package list from hackage.haskell.org
Package list of hackage.haskell.org has been updated.
The index-state is set to 2023-11-18T06:48:16Z.
605357ef37ca:/pandoc# cabal build --allow-newer --enable-executable-static --ghc-options '-optl-pthread' pandoc-cli

build-log.txt

605357ef37ca:/pandoc# file dist-newstyle/build/x86_64-linux/ghc-9.8.1/pandoc-cli-0.1.1.1/x/pandoc/build/pandoc/pandoc
dist-newstyle/build/x86_64-linux/ghc-9.8.1/pandoc-cli-0.1.1.1/x/pandoc/build/pandoc/pandoc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

benz0li marked this pull request as draft August 30, 2023 16:37

benz0li force-pushed the add-dev-containers branch from d477594 to 0581dbd Compare September 1, 2023 09:55

benz0li mentioned this pull request Sep 5, 2023

Add Dev Container Configuration Files commercialhaskell/stack#6228

Merged

2 tasks

benz0li force-pushed the add-dev-containers branch 5 times, most recently from fdabfb7 to 47f0733 Compare September 13, 2023 05:47

Add Dev Container Configuration Files

4ebdaae

benz0li force-pushed the add-dev-containers branch from eba80f0 to 4ebdaae Compare September 15, 2023 08:28

benz0li marked this pull request as ready for review September 15, 2023 08:31

benz0li added 4 commits October 2, 2023 08:19

Dev Containers: Update HLS to v2.3.0.0

5c96ef4

Dev Containers: Update Stack to v2.13.1

1c21e5a

- Install Stack from GitHub releases

Dev Containers: Add 'GHC 9.8 (latest)'

50e05dc

Dev Containers: Update HLS to v2.4.0.0

3abd8fe

benz0li added 2 commits October 30, 2023 16:08

Dev Containers: Update 'GHC 9.2 (recommended)'

bb2f42e

- Update HLS to v2.4.0.0

Dev Containers: Update 'GHC 9.2 (recommended)'

a224e52

- Update Stack to v2.11.1 - Keep GHC 9.2 as recommended - haskell/ghcup-hs@2c3148a is inconsistent

benz0li closed this Nov 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Dev Container Configuration Files #9039

Add Dev Container Configuration Files #9039

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

jgm commented Aug 30, 2023

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

jgm commented Aug 30, 2023

cagix commented Aug 30, 2023

benz0li commented Aug 31, 2023 •

edited

Loading

benz0li commented Sep 7, 2023

benz0li commented Sep 14, 2023 •

edited

Loading

benz0li commented Sep 15, 2023 •

edited

Loading

benz0li commented Oct 11, 2023

benz0li commented Nov 15, 2023

jgm commented Nov 15, 2023

benz0li commented Nov 15, 2023

jgm commented Nov 17, 2023

benz0li commented Nov 18, 2023

Add Dev Container Configuration Files #9039

Add Dev Container Configuration Files #9039

Conversation

benz0li commented Aug 30, 2023 • edited Loading

benz0li commented Aug 30, 2023 • edited Loading

benz0li commented Aug 30, 2023 • edited Loading

jgm commented Aug 30, 2023

benz0li commented Aug 30, 2023 • edited Loading

benz0li commented Aug 30, 2023 • edited Loading

benz0li commented Aug 30, 2023 • edited Loading

jgm commented Aug 30, 2023

cagix commented Aug 30, 2023

benz0li commented Aug 31, 2023 • edited Loading

Footnotes

benz0li commented Sep 7, 2023

benz0li commented Sep 14, 2023 • edited Loading

benz0li commented Sep 15, 2023 • edited Loading

benz0li commented Oct 11, 2023

benz0li commented Nov 15, 2023

jgm commented Nov 15, 2023

benz0li commented Nov 15, 2023

jgm commented Nov 17, 2023

benz0li commented Nov 18, 2023

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 30, 2023 •

edited

Loading

benz0li commented Aug 31, 2023 •

edited

Loading

benz0li commented Sep 14, 2023 •

edited

Loading

benz0li commented Sep 15, 2023 •

edited

Loading