Extend .cabal format with "common" definition stanzas

[hvr]

( this is a project task description for the upcoming https://github.com/haskell/cabal/wiki/Hackathon2015 )

One complaint about the .cabal format is that there's avoidable duplication when multiple stanzas requiring to repeat some shared properties are involved (most notably build-depends)

Here's an example how common stanzas could be used:

name: thispkg
version: 1.2.3
cabal-version: >=2.0

-- ...

flag small_base
  description: Choose the new smaller, split-up base package.

-- unnamed global common stanza included implicitly in all library/executable/benchmark/test stanzas
common
  build-depends: base-orphans < 1

-- named common stanza which needs to be explicitly included
common cdef1
  hs-source-dirs: src

  build-depends: text
  if flag(small_base)
    build-depends: base >=2.1 && <5, array, containers, directory
  else
    build-depends: base >= 1.0

library
  include cdef1
  build-depends: parsec
  other-modules:

test-suite tests
  type: exitcode-stdio-1.0
  main-is: test.hs
  include cdef1
  build-depends: thispkg, tasty

benchmark bench1 
  type: exitcode-stdio-1.0
  main-is: bench1.hs
  include cdef1
  build-depends: thispkg, criterion

In the example above a single common section named cdef1 is defined, which is included by all 3 real stanzas. include cdef acts as if the respective common stanza's body was inserted in place of include cdef.

---

Possible extensions

• reserve a few special common stanza ids all, library, test-suites, and benchmark to declare common stanzas that are implicitly included by the respective named real stanza. In the example above we could have used common all instead of common cdef1 and all include cdef1 simply removed.
• allow recursive (but acyclic) includes. I.e. allow common stanzas to include other common stanzas.

/cc @dcoutts

---

@dcoutts suggests to support both,

1. named sections and explicit include at uses sites
2. unnamed global (implicit include into start of all sections) (would subsume Package global build-depends should apply constraints to all components #3404)

Moreover, we'd want this mechanism not only for .cabal files but also for cabal's config files. For non-.cabal files we want to reuse the infrastructure for file-backed include files (include-file ./some-ast-frament.inc).

---

refinement: common stanzas ought to be typed so we can catch mistakes early on during parsing already, see #2832 (comment)

[phadej]

As I can see from proposal, include commands can occur in any position.

Does this imply that they are handled by parser, i.e. not visible in GenericPackageDescription.

From tooling point of view, I'd like to see them in GenericPackageDescription, but as structure of data type doesn't easily support "include anywhere" approach: CondTree would need to be on [Either IncludeField a]. Yet "include at the beginning of the section" would be simpler: ([IncludeField], a):

Shortly, which:

-- current
condExecutables :: [(String, CondTree ConfVar [Dependency] Executable)]
-- anywhere, each block consists of includefields and "the thing"
condExecutables :: [(String, CondTree ConfVar [Dependency] [Either IncludeField Executable])]
-- only on top
condExecutables :: [(String, CondTree ConfVar [Dependency] ([IncludeField], Executable))]

[phadej]

Another comment, include commands will occur in cabal.project field to, meaning to include other file. Maybe we can think this naming thru a bit better. Probably common stanzas would be nice for project files too.

[hvr]

@phadej as to the file-including, there's two ideas on the table, either use non-overloaded include-file to refer to files, or infer include referring to a file by occurence of /s. E.g. include ./foo would refer to a file, whereas include foo would refer to a local named include-stanza. One could have optional file:/common: prefixes for overriding the detection, and allow for additional types (e.g. http:)

Then include file:foo would refer to a file foo rather than a common stanza named foo.

[phadej]

Ok, I agree, let's keep GenericPackageDescription as something rest of Cabal uses. See #3614

[phadej]

Question: should stanzas be declared before used?

If yes, then recursive acyclic stanzas are easy to implement (as stanzas overall)

I have no preference on this.

[23Skidoo]

Question: should stanzas be declared before used?

I'd say yes, it makes sense to nudge users to put common stanzas on top.

[phadej]

Motivated by possible in-parser implementation. There are two ways to parse common stanza: parse them beforehand or not.

• Parsing beforehand is difficult, because you don't know where they are inserted, i.e. which fields are valid.
• Treating them as macro is simpler from implementation pov:
  • when parsing sections, if include foo is encountered, insert stanza's fields and continue.
  • there would be no errors if stanza is not used (yet warning that it's unused)
  • warnings and errors would be reported as many times as it's used.

[phadej]

Another comment:

benchmark bench1 
  type: exitcode-stdio-1.0
  main-is: bench1.hs
  include cdef1
  build-depends: thispkg, criterion

says that include is an empty section in outline cabal-like-file view.
E.g. it would be possible to write

benchmark bench1 
  type: exitcode-stdio-1.0
  main-is: bench1.hs
  include cdef1
    something: foobar
  build-depends: thispkg, criterion

and it will be kind-of-correct (have to warn that include should be empty section)

Alternatively we could have include looking like a field:

benchmark bench1 
  type: exitcode-stdio-1.0
  main-is: bench1.hs
  include: cdef1
  build-depends: thispkg, criterion

[ttuegel]

Parsing beforehand is difficult, because you don't know where they are inserted, i.e. which fields are valid.

I don't think this needs to be the case. The common stanzas can be "typed" so that both Cabal (and the reader!) knows where they can go. For example,

common-library foo  -- only include in a library stanza
    ...

common-executable bar  -- only include in an executable stanza
    ...

common-build-info baz  -- include in any stanza that takes build-info fields
    ...

I imagine the common build-info stanza would be the most popular by far. Even if we only implemented common build-info stanzas, I think that would be a big improvement.

[phadej]

@ttuegel, yes, that's another option. And I do actually like it, except for section names; OTOH I cannot think any better ones.

[ttuegel]

And I do actually like it, except for section names; OTOH I cannot think any better ones.

Yes, I should have added, those section names are awful and shouldn't be considered for a real implementation!

A serious suggestion: we could add an ellipsis after the name to indicate a common section, i.e. library... foo for libraries, executable... bar for executables. The ellipsis would indicate that a section may be incomplete and it won't do anything until included in a real section.

[hvr]

@ttuegel While it's concise, I'm not sure I like using trailing punctuation to denote a difference (and also I don't think that the meaning of ... is obvious enough that you can guess its meaning). It'd be slightly more common to prefix keywords with special characters (e.g. $library, #library, %library) to denote macro-like definitions.

However, I'd prefer the more verbose/explicit syntax which doesn't introduce a new style of keywords by via non-alpha-nums.

And I assume common-library w/o an identifier would denote an auto-included global section that's included in all library stanzas?

Fwiw, other bikeshed names instead of common: define (and maybe use instead of include) or maybe macro.

[ttuegel]

And I assume common-library w/o an identifier would denote an auto-included global section that's included in all library stanzas?

I was only thinking about named sections... I think that's good, but I would prefer to call it something different. If it has the same syntax as a named section, I think it's too easy to accidentally leave off the name. This way, the user gets a parse error if they leave off the name accidentally. For the global sections, we could use global as a prefix. (Or a suffix?) I think any of your suggestions for the named sections is fine, but I would add fragment to the list of possibilities. My intent with fragment is to emphasize to the reader that these common sections are partial library, executable, etc. definitions.

[ghost]

Embrace hpack to cover this? Mostly a bystander, but just noting an alternative (which I am guessing everybody has already considered)

[hvr]

@kanishka-azimi Yes, I briefly considered the concept of hpack, but I've come to the conclusion that it's better to address the problem at the source, i.e. improve the few things lacking in .cabal syntax so we don't need to resort to external kludges like hpack (which would only introduce a new set of problems, YAML for starters ;-) ) to compensate for .cabal syntax's shortcomings.

[ghost]

Okay, makes sense. I ran across your yaml note last night, agree yaml, though somewhat light on noise, has ugly corners in it's definition. I created an issue in hpack to consider json, as a compromise between standard format but more limited complexity, at the cost of a little more syntax noise. I will close that issue as it probably still is more bloated than a cabal specific syntax.

The thing I like most about hpack is that people can contribute to it in relative isolation. It's intimidating seeing a repo with 800 issues, having to think about reverse compatibility, and it's a slight detour to learn a custom parser instead of a typical Json parser. Hpack will encounter the reverse compatibility issue with age... If you can keep that spirit alive, either with a second sublibrary for the cabal parser or just a well demarcated set of modules, that might increase the rate of community contributions to the revamped cabal syntax.

Look forward to seeing the future of what you all come up with.

[BardurArantsson]

Just while we're speculating, I wonder if people have though the necomer in the "config language" space, Dhall. Seems to have some very nice properties.

[hvr]

@BardurArantsson Dhall is definitely interesting in its own right (and I'm waiting for an occasion to make use of it), but it appears to me that by being actually a programming language (rather than a declarative package specification with enough structure to support automatic refactoring) it's also way too powerful for what we'd want for .cabal files. So we really want something inbetween an overly generic and inexpressive JSON data-model, and mini-programming-languages on the other end of the spectrum featuring too powerful constructs resembling user definable functions (even if not turing-complete). I really think that having a custom designed package specification language tailored to cabal's somewhat unique features and requirements is the most reasonable way to go, even if it means that we can't reuse off-the-shelf parsers.

[BardurArantsson]

(I'll stop with the semi-OT after this.)

I keep hearing about automatic refactoring (and perhaps GUI tools?), but how realistic is it to expect any such thing for Cabal? I honestly don't think it's even close to realistic enough to worry about as a goal. Plus we're talking files of (at most) a 100-200 or so lines. Who actually needs automated refactoring for that? (Especially with something like Dhall, you could probably cut down on the size drastically.)

Heck, in JVM-land even Maven had the original goal of having pom.xml files mostly being edited with a GUI tool, but as it turned nobody actually wanted that -- so almost everybody just edits the XML directly.

[hvr]

@BardurArantsson For one, haskell-mode or more generally things like haskell-ide-engine would greatly benefit from the ability to understand the structure of .cabal files, and be able to e.g. manipulate the build-depends (IOW, IDE-assisted build-depends/other-extension management).

But more importantly, I'm working on tools to assist Hackage curation by Trustees & maintainers, and one super boring, time-consuming and error-prone task is to step in and modify version bounds by hand. But more importantly, we want to make the recommended PVP-workflow more convenient, and for that imagine if we had this process automated to the point where you get presented with all information (changelog, API deltas, test/build-reports, etc) involving a new major version of your dependency, and you just have to click a button on Hackage to confirm that bumping that specific upper version bound is safe (rather than the alternative of flying blind by leaving off upper bounds altogether).

[BardurArantsson]

@hvr Ah, true, I hadn't considered the Hackage version bound thing.

[alanz]

Just a note re Dhall: It explicitly sets out to limit its power, precisely for use as a configuration language. See the original blog post.

[23Skidoo]

@phadej This can be closed as fixed by #4751 now, right?

[23Skidoo]

Closing as fixed.