Auto generating hie.yaml / better implicit configuration

[Avi-D-coder]

It seems like it should be possible in most cases to auto generate hie.yaml files given only cradle type. This would also remove the need for hie.yaml most of the time.

A .hie-bios.yaml in lowest parent directory could be used to determine stack, or cabal, if .stack-workxordist-newstylexorhie-bios` is not detected first.

We could start by looking in *.cabal for executable, library, test-suite, then any main-is and hs-source-dirs.

Caveats:

1. If you are using stack with hpack, you need to invoke stack build before we can build an implicit config.
2. In the event we observe multiple main-is or hs-source-dirs in a single module. Can we duplicate the cradle with the same name, but a different path?

if os(linux)
  hs-source-dirs: linux
if os(darwin)
  hs-source-dirs: osx
if os(windows)
  hs-source-dirs: win

cabal-helper could be used, but it's a heavy dependency for such a simple task.

Thoughts? I would implement this feature.

Update

implicit-hie

[mpickering]

The issue is that in order to do this you need to depend on the Cabal library in order to parse the cabal file which is against the premise of the library.

A better path forward would be to modify cabal/stack in order to output the correct information via a command line option which could be used to improve the implicit cradles.

[alanz]

See also haskell/haskell-ide-engine#1529

[Avi-D-coder]

I'm not suggesting a cabal dependency. I was suggesting we make a best effort to read the parts we care about from the cabal file. show-build-info certainly will definitely replace this. When do you suspect it will land?

[maralorn]

I am also a bit confused about this. Right now I put nearly the same hie.yaml in every of my projects:

cradle: {cabal: { component: "<project-name>" } }

I am even wondering why this is necessary because the README says it‘s looking for a cabal project.

(otoh: It‘s really no big hazzle, so I don‘t worry much about it.)

[fendor]

The implicit discovery mechanism of hie-bios should be able to resolve it if there is only a single component. So, you usually need it if you have multiple components, e.g. exe and lib.

[emlautarom1]

The issue is that in order to do this you need to depend on the Cabal library in order to parse the cabal file which is against the premise of the library.

A better path forward would be to modify cabal/stack in order to output the correct information via a command line option which could be used to improve the implicit cradles.

Having something like cabal components which lists all components would be ok? Something like

$ cabal components
These components were found:
library:ghcide
exe:ghcide-test-preprocessor
exe:ghcide
test:ghcide-tests

I don't know how much information should be provided for each component, but I was thinking of adding a command like this to cabal since a lot of times I want to know what's available without looking into the .cabal file.

Yet, this approach would require running an external command in order to get the desired information.

[jneira]

@emlautarom1 Something like that is needed, there are an ongoing pr (by @fendor) to add the command show-build-info which will be contain the info about components: haskell/cabal#6241

[emlautarom1]

@jneira A few questions:

• How would hie-bios and cabal exchange this information? Just run cabal show-build-info, parse the output and build a hie.yaml file?
• What about detecting changes in the .cabal file? How we make sure that hie.yaml and .cabal files are always synced?

I don't mean to sound pessimistic but this approach seems fragile (maybe I'm missing something). Since adding Cabal (lib) as a dependency is out of the question (since it couples hie-bios with Cabal), take a look at this.

I haven't read the full spec yet, but maybe using something like the build-server-protocol (#71) would ensure that hie-bios always detects the correct components and flags and stays in sync with the underlying build tool (be it cabal, stack, whatever comes next).

Correct me if I'm wrong, but the workflow would look like:

• cabal: handles .cabal files, components, dependencies, ghc flags, etc.
• hie-bios: handles a GHC session while remaining build-independent, so whatever you put on top of it doesn't know about the underlying build tool (doesn't even know about what Cabal is).
• ghcide/hls: handles type-checking, refactors, go-to definition, find usages, etc.
• code editor: any editor that supports the language-server-protocol (VSCode, Emacs, Vim, etc)

I'll look later into the build-server-protocol current "spec". I'm sure that working with the Scala guys can improve the quality of tools for both communities.

[jneira]

@emlautarom1 sorry, i missed completely your thoughtful comment

How would hie-bios and cabal exchange this information? Just run cabal show-build-info, parse the output and build a hie.yaml file?

Yeah, stringly typed programming! live the crazy unix life! 😁

What about detecting changes in the .cabal file? How we make sure that hie.yaml and .cabal files are always synced?

hie-bios cradle includes the info about what files have to been watched, client code is the responsable to use that info to invalidate caches, triggers calls to external build tool etc

build-server-protocol looks great and maybe it could be a game changer, but we should implement it for all possible tools used to build haskell source files: mainly cabal and stack but also hadrian, bazel, nix, etc, etc. Parsing cli output is surely fragile and build tool dependent but is is available right now for all of them (directly or via a script wrapper)

[emlautarom1]

@jneira Alright, so if I'm understanding this correctly, then the workflow would be something like:

1. hie-core detects if this is a cabal, stack, etc... project.
2. hie-core asks for the project components (lib, exe, tests), flags and directories to the build tool using a CLI command - currently for cabal it would be cabal show-build-info (WIP)
3. hie-core parses this output and build the appropiate hie.yaml file.
4. Users of hie-core - haskell-language-server, for ex. - are in charge of detecting when the underlying build files (.cabal, stack.yaml, etc.) are modified and notify hie-core that it needs to ask again to the underlying build tool for project components, flags and directories in order to sync the hie.yaml file.

---

build-server-protocol looks great and maybe it could be a game changer, but we should implement it for all possible tools used to build haskell source files: mainly cabal and stack but also hadrian, bazel, nix, etc, etc. Parsing cli output is surely fragile and build tool dependent but is is available right now for all of them (directly or via a script wrapper)

Absolutely, parsing plain text would be a lot faster to implement, yet I think that in the long run a more robust solution would prove to be better for maintainability. The point that worries my is 4, since now the client code on top of hie-core needs to know what build tool is being used, instead of delegating that work.

I understand that in the current situation this is the most pragmatic solution and I think that it can prove to be very usefull, but as someone said somewhere "nothing is more permanent than a temporary solution".

Currently I'm a bit busy with school, but once I'm available I'll look into building a MVP of the bsp and integrate it into cabal-install.

[maralorn]

I don‘t think that the client code needs to know what build tool is being used. I think hie-bios can pass a list of files it used to the client. The client just needs to watch them and trigger a reload on file change. It doesn‘t need to know, why it is depending on those files.

[Avi-D-coder]

see https://github.com/Avi-D-coder/implicit-hie

[jneira]

I want to take a deeper look asap to check in which cases it works (and doesnt), checking corner cases like:

• multi package setup in cabal or stack (generically if posible)
• common stanzas (they can contain hs-source-dirs)
  • see f.e. cabal-helper 😉
• stack projects with package.yaml but no cabal files
• uncommon cabal layouts (but legal for the actual parser, that takes in account the cabal spec version)

[Avi-D-coder]

• multi package setup in cabal or stack
• common stanzas: This is doable, but is not a priority for me.
• stack projects with package.yaml but no cabal files: The best bet is just to invoke stack build and then use the generated cabal file.
• uncommon cabal layouts (but legal for the actual parser, that takes in account the cabal spec version): WIP, fixed some a bit ago, but I know there are more. If needed cabal-fmt has a full parser, but it would bring in extra dependencies and is likely a lot more then we need.

The next step I'm going to look at is ingratiation with hie-bios. My experience and the experience of those I have talked to is that a handwritten hie.yaml is almost always needed even in spite of hie using cabal-helper. The current version should alleviate all the config problems beginners face, it does not cover all expert use cases.

[jneira]

Nice, imo it is suited to be integrated here cause to not have dependencies on the build tools is a hard requirement in hie-bios. So only left:

• custom raw config parsers like implicit-hie
• require build tools to give the needed info at runtime:
  • cabal will have it with show-build-info (exe:cabal part plus tests) cabal#6241
  • stack already has stack ide targets for get all targets but afaik it has not a way to give us the component of a specific source file mapping between targets and source dirs/files
    • would be that needed? (maybe @fendor could help to clarify it)
    • is there no ticket for that?
  • using other tools requires an explicit hie.yaml for now

[fendor]

would be that needed?

Yeah, that is needed.
IIRC, there is some WIP for stack to dump the same information as haskell/cabal#6241

[jneira]

stack already has stack ide targets for get all targets but afaik it has not a way to give us the component of a specific source file mapping between targets and source dirs/files

About the stack work to output build info as json (like cabal show-build-info): the branch that contains some work about, done in the bristol hackaton by @chrisdone seems to be here

[domenkozar]

For stack projects with multiple package hie currently fails since it cannot detect which Main.hs is the correct.

Running stack ide targets it could then initialize multi-crade with each component.

[jneira]

@Avi-D-coder Now we are already using implicit-hie-cradle for implicit configuration so we could close this?

[domenkozar]

How do I enable implicit-hie-cradle to use multi-project stack?

[jneira]

@domenkozar implicit-hie-cradle should use underneath the same definition you got executing gen-hie in the project root. They use the same lib (implicit-hie). So if gen-hie is not able to generate a working hie.yamlautomatically, implicit-hie-cradlewill not work neither.
does not it work out-of-the-box for your multi project?