stack new for creating new projects with templates

[radix]

I'll let others determine whether this is in of scope, but I think it would be a good idea to build in a project-creating command that sets up a basic directory/file structure.

This would be kinda like cabal init, but wider in scope -- more like lein new.

Here are some of the features I imagine it should have, perhaps to be split into separate tickets:

• stack new myproj would create a myproj directory with src, README, LICENSE, stack.yaml
• create a Main.hs (if the user specifies a command-line argument?)
• create sample unit test file
• create a directory layout within src based on module names provided by the user (as a command-line argument?)
• support plugins for different code templates.
• support specifying the template-plugin on the stack new command line without installing that plugin ahead of time, e.g. stack new myproj --template=yesod-scaffold-stack[==x.x.x] would automatically install the yesod-scaffold-stack plugin and use it to generate the new project.

(see plugin discussion at #136)

It would be nice to also have commands for manipulating a project after it's been created. New users coming to the tooling may want to just play around, and having to decide up-front what "kind" of project they should make (library and/or executable? what should the targets be?) can be a bit paralyzing -- if they make one choice, and write some code, how will they change it later? The project-creation command should make this very clear and as easy as possible to change afterward.

Khan Thompson created http://hackage.haskell.org/package/hein recently, which is basically exactly what I'm suggesting here. On the reddit discussion at http://www.reddit.com/r/haskell/comments/37wsxc/hein_an_alpha_build_tool_modelled_after_lein/crqilmw he indicated he'd be happy to deprecate his in favor of this one (implicitly, anyway).

[DanBurton]

Depending on when we get around to #138, or perhaps depending on the "template" being used, stack new myproj should also create myproj.cabal.

I like the idea of having a cli for tasks like "stack create module Foo.Bar", which creates src/Foo/Bar.hs and adds Foo.Bar to the library's exposed-modules.

[snoyberg]

Just realized I haven't commented here yet, so: I'm +1 on doing this, and thank you for writing up a detailed description. We need to think about how this will play in with some kind of a stack init for generate a stack.yaml from an existing .cabal file.

Pinging @khanage, you have interest in working on this issue?

[khanage]

I've been half starting on this for a bit now, so I'm happy to take it on (and happy to hand it over to someone a bit more experienced as well).

My vision for this is something very much like lein. One of the best stories I have demo'ing functional programming with coworkers is to get them to

1. Download lein
2. lein new compojure demo
3. lein ring server

And they've got a web site up and running, which they modify without learning much.

I'd like to replicate that (indeed that's the goal of hein), so that means for new we will create some base project with a .cabal file, a Main.hs and a stack.yaml such that stack is all ready to go.

To create that stack.yaml file - perhaps it's best to try and call stack init from this process? That would keep things coherent WRT heuristics for dealing with multiple packages etc.. It could lead to some strange interplay in the future with some esoteric template that might require certain overrides of dependencies, so perhaps I'll call stack init from the default template and leave it up to template implementations to decide on this?

[snoyberg]

Here's my recommendation: start on the general templating stuff now, not worrying too much about the interplay with stack init or something else yet. Either I or someone else will start driving the discussion forward on figuring out exactly how initialization should work, and then we can tie it in with the project templating stuff you do. I can't picture a world where the work you're starting with would be significantly impacted by any decision we make on the initialization from. Does that sound good?

[snoyberg]

I've opened #253 to try and discuss this and other related topics together

[khanage]

Just full disclosure, I'm fairly new to haskell at the level of doing things, so I'm going to be pretty slow with this.

I've made a small start on this, but my question is: how should I check this in? Should I create a branch on this repo, or fork the stack project completely? (Pretty new to open source as well ;)).

Sorry if this isn't the right forum for this, but I think it's better to be upfront about my ignorance.

[snoyberg]

No problem at all, feel free to ask questions :)

I'd recommend committing to a separate branch. I like the convention of using the issue number in the branch new, so maybe 137-template or something like that.

[gregwebs]

There is prior art in this area that should be considered. One is Michael's project-template. Another is hi https://github.com/fujimura/hi/blob/master/README.md

[snoyberg]

@khanage Just an FYI, we all seem to be in alignment now about the project initialization approach, and the description laid out here for what stack new should do fits in perfectly. Let me know if you have any questions about goals.

[khanage]

All good - like I said, I'm fairly new, so it takes me time to learn all the libraries that are in use (e.g. optparse).

I'm kind of stuck on deciding on a design for the templates - I was thinking of trying to create an ADT which represents possible actions templates can take (e.g. createFile, createDir, runCommand, assertFile) and wrapping that up in a free monad.

Any suggestions anyone? I'm open for guidance :).

[gregwebs]

why do you need a free monad? you might be overthinking things and be better off just cranking out some code right now.

[snoyberg]

After experience with the Yesod scaffolding, we eventually came down to maintaining the scaffolding in its own repo with different branches for the various flavors (which, in Yesod's case, was mostly about database backends). This made it easy to maintain multiple flavors, merge changes between them, and test them automatically. My experience with that has only been positive.

Trying to maintain a complicated scaffolding inside Haskell code itself will likely be quite difficult to manage.

[snoyberg]

@radix I read through your list most closely, and I really like your plugins idea. To elaborate: this would essentially be a collection of different templates that stack could download and offer to the user, right? If we do this well, it could completely subsume functionality like yesod init, which I'd be very happy about. In that world, it seems like people would be free to have as opinionated a project template as they'd like to provide, and stack could simply offer all of them.

@khanage I'd be happy to work through some of the architectural issues around something like that, which would leave you free to come up with a good project template itself. Does that sound like a good plan?

[khanage]

@gregwebs - I take your point, I'll get something that just works (by that I mean barely) out and we can go from there. I was thinking of using the free monad so that it provides a convenient syntax for template creators to use to create templates, but as @snoyberg points out, that might not be a scalable approach.

@snoyberg the plugins part is exactly what I'm trying to work out. I'll build something naive, and we can go from there?

[snoyberg]

Sounds good. Not to put pressure on, but do you have an ETA on something here? If it's going to be beyond this week, I'll add some brain dead code to get started so that stack is easy for new users to test out.

[khanage]

I've got a referenced commit? 8a6d987

I'm happy for anyone to just scrap that with something better, or to give me some feedback and I can get on making some changes.

[snoyberg]

It's a start, but my hope is to get to a place where we're not relying on cabal-install for anything.

[khanage]

Yeah, I had a look at the relevant cabal source and there's a comment there saying it's ad-hoc. I was hoping to use that directly rather than calling out via shell commands.

There's probably a bit of room for writing such a mapping, but I thought I'd push this up and get some feedback/assistance first before it looked like I'd abandoned it.

[snoyberg]

I was thinking we could go for something much simpler than what cabal init does: forget about trying to guess the correct build-depends, and simply have some templated projects. Adding in build-depends intelligently is a nice idea, but I really dislike how cabal does it: only at project initialization. Adding the ability to intelligently detect that would be great as a separate feature for stack at some point in the future (likely combined with #39 or #259).

[rvion]

This is how I see things:

commands

• stack new mytemplate : fetches template mytemplate and merge template folder tree in current folder
• stack new : defaults to stack new default-template

flags:

• --preview or -p to only display content about to be created in terminal. Usefull to ensure all is correct, and maybe speedup some stack tests
• --force or -f to overwrite files already present in project file tree when redefined in template

Ability to pass in key:value list in params

templates fetching

• look into below folders in following order:
  1. the .stack-templates folder of current project: this folder is for project specific templates
  2. the $HOME/.stack-templates folder: for user-specific templates. This allows anyone to version his own personnal configs in his dotfiles
  3. the $HOME/.stack/.stack-templates: for stack global templates.
• if mytemplate is a git url, clone it within the templates of the current project

templates structure

.stack-templates folders would be organized like that:

.stack-templates
    |- default-template
        |- ...
        |- stack.yml
        | -scr
            |- ...
    |- my-template
        | - ...
    |-yesod-new-controller
        |- handlers
            |- ...

possible template versioning built in:

stack new mytemplate
stack new mytemplate:minimal

.stack-templates
    | template-with-one-version-only
        |- latest
            |- ....  
    |- template-with-2-versions
        |- latest
             |- ...
        |- minimal
             |- ...

templating ?

• use some well known and already working library. (Mustache, Handlerbars ?)
• populate template dictionnary with extra variables (BASENAME for file name without extension, TIMESTAMP for.. well timestamps)

example for filenames

.stack-templates
    |-template1
         |-file-for-{{subject}}
             |- I-love-{{key3}}

example for contents

module {{BASENAME}} where

or

module MIgration{{NiceName}}{[TIMESTAMP}} where

apply = do
   ...

reverse = do
   ...

various example syntaxes

stack new yesod-handler name:launch-missiles renderer:lucid
stack new yesod-handler:lucid name:launch-missiles
stack new yesod-handler-lucid name:launch-missiles
stack new yesod-handler:lucid launch-missiles

---

What do you think ?

I think it provides enough flexibility and customization for everyone, and remains really simple
It can be used to start a new project, add licenses, add REST routes, or whatever one could want.

[cschneid]

@rvion - All the parts of templating that you suggest are implicit. Is it worthwhile to have an (optional) template configuration? It could hold things like help documentation, preprocessing commands, or other programmatic helpers?

I like the idea that the templating can be extended to the rails use case of rails generate model XYZ where it generates additional files in an existing tree.

I'm a 👍 on the proposal.

[rvion]

@cschneid can you explain what do you mean by implicit.

---

Here are some more infos:

in addition to commands like,

• stack new scotty-scaffold
• stack new https://github.com/yesodweb/yesod/tree/master/yesod-sitemap/template1
• stack new /tmp/folder/demo

we could indeed come up with a config file storing default keys/values and functions available in templating.

personnal config: $HOME/.stack-template/defaults.yml

keys:
   name: rvion
   githubpseudo: ...
tools:
   tree

stack global: `~/.stack/.stack-template/defaults.yml:

keys:
   language: haskell
tools:
   date
   stamp: date -f .....
   ...

lts-2.14 specific : `~/.stack/snapshots/x86_64-linux/lts-2.14/.stack-template/defaults.yml:

keys:
   stack: lts-2.14
   ghc_version: 7.8.4
   ...

📝, here I suggest that we add a .stack/template in ~/.stack/snapshots/x86_64-linux/lts-2.14 because if packages themselves can register templates automaticall, it makes sense

fun things ?

we can so have a metaTemplate template like that:

Main.hs:

this line will be added above the first line in Main.hs file every time the user run `stack new metaTemplate`
{{$ORIGINAL_CONTENT}}

every time the user run stack new metaTemplate, file is overwritten by a new file containig the new line + the old content 🍰

similar mechanism could be achieve for adding content in the middle of the file

eg:

{{toold.format $ORIGINAL_CONTENT}}

[chreekat]

I think @rvion's idea is really cool, but it feels like a lot of scope creep. What mechanics of stack is it using? Could it be a separate tool? Should it?

I also find the word 'new' to be unfriendly for a tool that can be used at any time to drop new templated files in. Maybe 'generate' as with rails?

[radix]

@chreekat I definitely think stack should have the project-generation functionality. I don't know so much about the modification functionality, but it seems to make some sense (especially given the concern I originally wrote in the ticket description about how it can feel paralyzing to generate a project without knowing exactly what you should decide on yet).

Also, has anyone considered wrapping existing tools like this? https://github.com/fujimura/hi

I just found it the other day.

[cschneid]

@chreekat My underlying issue that I'd like solved is much simpler than where this has gone. "I want to start a new haskell project, what am I doing".

I've asked on IRC and the answer as of now is "use cabal-install to generate the .cabal file, then start using stack".

I think a small first step that does belong in the stack project is generating a minimal compilable haskell project. So generating a stub Main.hs, a .cabal file and a stack.yml

From there, it starts snowballing as an idea to making it a templating engine, since that has proven so useful in other programming environments (rails generate, yeoman...).

[rvion]

• I think the template tool should be integrated into stack
• I think stack new is easier to remember than stack generate, but new isn't the best command name either.

I think my proposition have gone a bit too far away from the original need, but in the same time, I think a decent templating tool is a must have within stack. That's how good practice will spread and how people will becope productive.

My proposition is not (yet?) good enough: there is no drop simple way how to generate new modules (and patching cabal files, etc etc), but I feel that the core is good:

Indeed:

• simple mechanism: merge trees of files
• share the mentality of stack architecture (snapshots, user customization)
• It is simple to extend with any executables callable from within template strings
• It features backward compatible way for cabal packages to offer template generation per snapshot by just adding a .stack-templates at the root
• it allows automatic documentation of templates: what keys are missing, which holes have been filled already, by what
• the feature can be incrementally added to stack along brainstorming iterations: the basis for it to work is just to copy paste some files defined in a .stack-template folder

[rvion]

Here is a shorter / better proposal:

1. create template with stack new template_id key1:val1 key2:val2
2. If keys are missing, nice message about what must be given to template. (list of necessary keys are done parsing the template folder)
3. locations for templates and default key values are preject specific: (.stack-template/...) , then user: (~/.stack-template/...), then snapshot (~/.stack/snapshots/arch/lts-xxx/.stack-template/...), then global(~/.stack/.stack-template/...).
4. Keys can be ommited if already defined in a .stack-template/defaults.yml (see above for locations)
5. if a cabal packages contains a .stack-template/ folder, it will be added in the snapshot .stack-template/ folder for automatic discovery.
6. templates are folders (with arbitrary levels of subfolders, and files), whose files and subfolders use markup both in name and content
7. templating engine will replace expressions inside the template brackets:
   • env.xxx is replaced by specific dynamic content (env.current_file_content, env.imports, etc.)
   • func.ppp is call pre-built scripts defined in stack (func.format.module, func.lowercase, etc.)
   • key.aaa.bbb.ccc is replaced by values specified in stack new params or defaults.yml file
   • tool.xxx param1 param2 key.paramfromkey is replaced by the result of tool xxx called with params param1, param2, amd the key found for key.paramfromkey
8. options could be:
   • --preview or -p to only display content about to be created in terminal. Usefull to ensure all is correct, and maybe speedup some stack tests
   • --force or -f to overwrite files already present in project file tree when redefined in template
   • --project or -p to specify in which sub project should the template run

All uses-cases I can think of are covered:

a template to create a module could be done like that:

module
    |- scr
        |-{{func.format.path key.prefix}
            |-{{func.format.module key.name}}.hs
                   "module {{func.format.module key.name}} where
                   "import Imports
                   "

and called via stack new module prefix:foo/bar/baz name:test

---

@radix @DanBurton @chreekat @cschneid @khanage @snoyberg : does it feels logic and simple enough ?
Do you have use cases I could test the model against ?

[snoyberg]

I was in the middle of a response when my computer ran out of battery, so I may have forgotten something when rewriting it.

I'm worried about the perfect becoming the enemy of the good here, and in the process scaring people away from writing something useful. In other words: I like the ideas I see here, but let's get something into people's hands quickly and iterate.

I've implemented something very simple for now. It could certainly use improvement, but serves the purpose of getting someone started quickly. Let's make it better.

[snoyberg]

Can you put that in a separate issue? It doesn't seem relevant to what's
being discussed here.

On Fri, Jun 26, 2015, 11:59 PM aceLren [email protected] wrote:

I'm still getting this error on OSX (works fine on Windows). It handles
stackage.org fine (0.1.0.0 didn't, but upgrading to 0.1.1.0 fixed that),
but now it dies on ghc:

Preparing to download ghc-7.8.4 ...TlsException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))

—
Reply to this email directly or view it on GitHub
#137 (comment)
.

[DanBurton]

Self-assigning; I'm going to flesh this out a little bit more.

[DanBurton]

(And by that I mean, I'm going to flesh out the code.)

Tentative plan:

• Use mustache to define templates, hastache to interpret them
• only works if current directory is empty
• new-template names project according to current directory name

[DanBurton]

Logic was already in place for checking the absence of files to be written. I implemented the other things I mentioned.

Intended next steps:

• command line override of arbitrary variables (such as name)
• add some optional variables to the new-template, such as author

And then:

• command line accept "template" argument
• host templates in a github repo (commercialhaskell/stack-templates?)
• automatically download templates if the needed template is not found

[DanBurton]

Per @snoyberg's suggestion, I switched stack new templates over to the project-template format. Templates are now hosted at https://github.com/commercialhaskell/stack-templates. I believe my work here is done for now.

@chrisdone kindly review Stack/New.hs, and the associated parts of Stack/Options.hs (newOptsParser) and Main.hs (newCmd).

Potential future work: template default args in a yaml file, other "calculated" default args such as timestamps.

[gibiansky]

+1 for new-template names project according to current directory name instead of just new-template.cabal (if this hasn't already been implemented in latest Github stack, I'm testing with Hackage version)

[DanBurton]

@gibiansky it has! You can easily get the "latest Github stack" if you wish via stack upgrade --git.

[DanBurton]

Closing this issue. If people still want particular features described here but not yet implemented, feel free to open up a new issue and ping me.