Support main as the default branch for templates

[majjoha]

I've been looking into setting up a template for my Stack projects that I want to share on GitHub, and I am currently running version 2.5.1.

When referencing a template with either a URL or a file on my computer, I am able to scaffold projects successfully with the following two commands:

stack new <project-name> <url>

stack new <project-name> <file>

However, if I try to scaffold a project by referencing a template from my GitHub repository, I get the following error:

Expand/collapse log output

~/Dropbox/Kode/Haskell % stack new --verbose test-project majjoha/simple
  Version 2.3.3 x86_64
  Compiled with:
  - Cabal-3.2.0.0
  - Glob-0.10.1
  - StateVar-1.2
  - aeson-1.4.7.1
  - annotated-wl-pprint-0.7.0
  - ansi-terminal-0.10.3
  - ansi-wl-pprint-0.6.9
  - array-0.5.4.0
  - asn1-encoding-0.9.6
  - asn1-parse-0.9.5
  - asn1-types-0.3.4
  - async-2.2.2
  - attoparsec-0.13.2.4
  - attoparsec-iso8601-1.0.1.0
  - auto-update-0.1.6
  - base-4.14.0.0
  - base-compat-0.11.1
  - base-compat-batteries-0.11.1
  - base-orphans-0.8.2
  - base16-bytestring-0.1.1.7
  - base64-bytestring-1.1.0.0
  - basement-0.0.11
  - bifunctors-5.5.7
  - binary-0.8.8.0
  - bitarray-0.0.1.1
  - blaze-builder-0.4.1.0
  - blaze-html-0.9.1.2
  - blaze-markup-0.8.2.7
  - bytestring-0.10.10.0
  - casa-client-0.0.1
  - casa-types-0.0.1
  - case-insensitive-1.2.1.0
  - cereal-0.5.8.1
  - clock-0.8
  - colour-2.3.5
  - comonad-5.0.6
  - conduit-1.3.2
  - conduit-combinators-1.3.0
  - conduit-extra-1.3.5
  - connection-0.3.1
  - containers-0.6.2.1
  - contravariant-1.5.2
  - cookie-0.4.5
  - cryptohash-sha256-0.11.101.0
  - cryptonite-0.27
  - cryptonite-conduit-0.2.2
  - data-default-class-0.1.2.0
  - deepseq-1.4.4.0
  - digest-0.0.1.2
  - directory-1.3.6.0
  - distributive-0.6.2
  - dlist-0.8.0.8
  - easy-file-0.2.2
  - echo-0.1.3
  - ed25519-0.0.5.0
  - either-5.0.1.1
  - exceptions-0.10.4
  - extra-1.7.4
  - fast-logger-3.0.1
  - file-embed-0.0.13.0
  - filelock-0.1.1.5
  - filepath-1.4.2.1
  - filtrable-0.1.5.0
  - fsnotify-0.3.0.1
  - generic-deriving-1.13.1
  - ghc-boot-th-8.10.1
  - ghc-prim-0.6.1
  - githash-0.1.4.0
  - hackage-security-0.6.0.1
  - hashable-1.3.0.0
  - hfsevents-0.1.6
  - hi-file-parser-0.1.0.0
  - hourglass-0.2.12
  - hpack-0.34.2
  - hpc-0.6.1.0
  - http-api-data-0.4.1.1
  - http-client-0.7.1
  - http-client-tls-0.3.5.3
  - http-conduit-2.3.7.3
  - http-download-0.2.0.0
  - http-types-0.12.3
  - infer-license-0.2.0
  - integer-gmp-1.0.3.0
  - integer-logarithms-1.0.3
  - libyaml-0.1.2
  - lifted-base-0.2.3.12
  - lukko-0.1.1.2
  - megaparsec-8.0.0
  - memory-0.15.0
  - microlens-0.4.11.2
  - microlens-mtl-0.2.0.1
  - microlens-th-0.4.3.5
  - mime-types-0.1.0.9
  - mintty-0.1.2
  - monad-control-1.0.2.3
  - monad-logger-0.3.34
  - monad-loops-0.4.3
  - mono-traversable-1.0.15.1
  - mtl-2.2.2
  - mustache-2.3.1
  - neat-interpolation-0.5.1.1
  - network-3.1.2.0
  - network-uri-2.6.3.0
  - old-locale-1.0.0.7
  - old-time-1.1.0.3
  - open-browser-0.2.1.0
  - optparse-applicative-0.15.1.0
  - optparse-simple-0.1.1.2
  - pantry-0.4.0.2
  - parsec-3.1.14.0
  - parser-combinators-1.2.1
  - path-0.8.0
  - path-io-1.6.0
  - path-pieces-0.2.1
  - pem-0.2.4
  - persistent-2.10.5.2
  - persistent-sqlite-2.10.6.2
  - persistent-template-2.8.2.3
  - pretty-1.1.3.6
  - primitive-0.7.1.0
  - process-1.6.8.2
  - profunctors-5.5.2
  - project-template-0.2.1.0
  - random-1.2.0
  - regex-applicative-0.3.4
  - regex-applicative-text-0.1.0.1
  - resource-pool-0.2.3.2
  - resourcet-1.2.4.2
  - retry-0.8.1.2
  - rio-0.1.18.0
  - rio-orphans-0.1.1.0
  - rio-prettyprint-0.1.1.0
  - rts-1.0
  - safe-0.3.19
  - safe-exceptions-0.1.7.0
  - scientific-0.3.6.2
  - semigroupoids-5.3.4
  - semigroups-0.19.1
  - silently-1.2.5.1
  - socks-0.6.1
  - split-0.2.3.4
  - splitmix-0.1.0.1
  - stm-2.5.0.0
  - stm-chans-3.0.0.4
  - streaming-commons-0.2.2.1
  - syb-0.7.1
  - tagged-0.8.6
  - tar-0.5.1.1
  - tar-conduit-0.3.2
  - template-haskell-2.16.0.0
  - temporary-1.3
  - text-1.2.3.2
  - text-metrics-0.3.0
  - th-abstraction-0.3.2.0
  - th-expand-syns-0.4.6.0
  - th-lift-0.8.1
  - th-lift-instances-0.1.17
  - th-reify-many-0.1.9
  - time-1.9.3
  - time-compat-1.9.3
  - tls-1.5.4
  - transformers-0.5.6.2
  - transformers-base-0.4.5.2
  - transformers-compat-0.6.5
  - typed-process-0.2.6.0
  - unicode-transforms-0.3.6
  - unix-2.7.2.2
  - unix-compat-0.5.2
  - unix-time-0.4.7
  - unliftio-0.2.13
  - unliftio-core-0.2.0.1
  - unordered-containers-0.2.12.0
  - uuid-types-1.0.3
  - vector-0.12.1.2
  - vector-algorithms-0.8.0.3
  - x509-1.7.5
  - x509-store-1.6.7
  - x509-system-1.6.6
  - x509-validation-1.6.11
  - yaml-0.11.4.0
  - zip-archive-0.4.1
  - zlib-0.6.2.2
Warning: this is an unsupported build that may use different versions of

dependencies and GHC than the officially released binaries, and therefore may

not behave identically.  If you encounter problems, please try the latest

official build by running 'stack upgrade --force-download'.
2020-11-04 12:06:13.026373: [debug] No project config file found, using defaults.

2020-11-04 12:06:13.037678: [debug] (SQL) SELECT COUNT(*) FROM "last_performed" WHERE ("action"=?) AND ("timestamp">=?); [PersistInt64 1,PersistUTCTime 2020-11-03 11:06:13.037466 UTC]

2020-11-04 12:06:13.039377: [debug] Opening local template: "majjoha/simple.hsfiles"

2020-11-04 12:06:13.039639: [info] Downloading template "majjoha/simple" to create project "test-project" in test-project/ ...

2020-11-04 12:06:13.039782: [debug] Downloading /majjoha/stack-templates/master/simple.hsfiles

2020-11-04 12:06:13.236572: [debug] Opening local template: "/Users/mathias/.stack/templates/majjoha/simple.hsfiles"

2020-11-04 12:06:13.237237: [error] The template "majjoha/simple" is invalid and could not be used. The error was: InvalidInput "404: Not Found"

/Users/mathias/.stack/templates/majjoha/simple.hsfiles:   14.00 B downloaded...

From what I can gather, the issue is that stack new <project-name> <github-username>/<template-name> assumes that the template exists on the master branch (see src/Stack/New.hs#L176:L182).

Is it possible to reference main as the default branch at the moment, and if it isn't, would you consider supporting it?

Thank you for your great work!

[qrilka]

@majjoha I don't see stack-templates among your repositories so it looks to me that the error above is not about main. But supporting main seems to be a sensible feature.

[majjoha]

@qrilka: Oh. It was public when I ran into the issue but I removed the repository afterwards. I've put it on GitHub again so you can reproduce the issue. Sorry for the inconvenience!

[qrilka]

Good then. I think PR fixing this will be appreciated.

[majjoha]

Having only read Learn You a Haskell for Great Good! and currently working my way through Haskell Programming from First Principles, I believe I am still too much of a novice to implement this (at least, without a bit of guidance).

Also, what would be the ideal behavior from your perspective? Off the top of my head, I see three possible implementations:

1. replace master with main in the code referenced in my original post which will most likely cause problems to many people.
2. attempt to pull the template from the main branch if a template cannot be found on master. If this also fails, then throw an error.
3. require users to explicitly state a branch as a parameter. This could be on the format username/branch/template or with an optional argument --branch main.

What are your thoughts?

[qrilka]

I think 2 makes sense probably with 3 as an enhancement.
The 1st part looks to be not hard to implement: urlFromRepoTemplatePath could be converted (and renamed) to a function which would return a URL with an optional backup URL and downloadFromUrl would need to have this "backup" implemented. Let me know if you want to attempt implementing this. If not I could find time to do that myself. I'd be glad to answer your questions if you have any.
But still this could only appear or you could try to get Stack version built from git.
Meanwhile the obvious workaround for now is to add a master branch mirroring main.

[majjoha]

Thank you so much for your willingness to help. I greatly appreciate it.

I've decided to give it a try, although I might admit that I am struggling quite a bit.

What I am currently trying to achieve is to keep the existing behavior of how templates are downloaded but add support for a branch field in RepoTemplatePath. I am mostly doing this because I believe it will make it easier to implement point 3 mentioned above at a later point if it becomes necessary. Once this is in place, I want to figure out how to retry downloading a template from GitHub with a different RepoTemplatePath where the branch is set to main if it cannot find a template on the master branch.

In https://github.com/majjoha/stack/commit/8d3180a83ac6381e7345eefb017bd44f1dc6360f, you can see the progress I've made so far. Does it seem like I am on the right track, and if so, where would you suggest that I go next?

At the moment, one of the tests in the test suite is failing with the following error:

src/test/Stack/Types/TemplateNameSpec.hs:17:9:
1) Stack.Types.TemplateName.TemplateName.parseTemplateNameFromString parses out the TemplatePath
     expected: RepoPath (RepoTemplatePath {rtpService = Github, rtpUser = "user", rtpTemplate = "name.hsfiles", rtpBranch = Just "master"})
      but got: RelPath "github:user/name.hsfiles" "github:user/name.hsfiles"

Do you have any pointers on how I can fix the failing test, and also, how do I run the version of Stack that I am currently working on?

Thank you once again!

[qrilka]

Your appreach could help with the point 3 but it doesn't seem to address 2.
As for the error you're seeing - in https://github.com/majjoha/stack/commit/8d3180a83ac6381e7345eefb017bd44f1dc6360f#diff-928fe810cc7900b1b94c905c78c35e9f16df89a35208cdecc7594b4a26ee38acR134-R135 you're matching either 1 component or 3 components separated by / (actually ignoring the 3rd component). If you want to support name, use/name and user/name/branch then you should have 3 cases there.
Hopefully that will help you to go further.

[jeffhappily]

Actually, GitHub does provide an API to get the information of the repository, including the direct download link (with the default branch).

For example, GET https://api.github.com/repos/majjoha/stack-templates/contents/simple.hsfiles would return

{
    "name": "simple.hsfiles",
    "path": "simple.hsfiles",
    "sha": "ee253ece365710a3f334ebd4d63eb82dd2fd948c",
    "size": 3098,
    "url": "https://api.github.com/repos/majjoha/stack-templates/contents/simple.hsfiles?ref=main",
    "html_url": "https://github.com/majjoha/stack-templates/blob/main/simple.hsfiles",
    "git_url": "https://api.github.com/repos/majjoha/stack-templates/git/blobs/ee253ece365710a3f334ebd4d63eb82dd2fd948c",
    "download_url": "https://raw.githubusercontent.com/majjoha/stack-templates/main/simple.hsfiles",
    "type": "file",
    "content": "ey0jIFNUQVJUX0ZJTEUgLmdpdGh1Yi93b3JrZmxvd3MvY2kueW1sICMtfQpu\nYW1lOiBDSQpvbjogW3B1c2gsIHB1bGxfcmVxdWVzdF0KCmpvYnM6CiAgYnVp\nbGQ6CiAgICBuYW1lOiB0ZXN0cwogICAgcnVucy1vbjogdWJ1bnR1LWxhdGVz\ndAogICAgc3RlcHM6CiAgICAgIC0gbmFtZTogQ2xvbmUgcmVwb3NpdG9yeQog\nICAgICAgIHVzZXM6IGFjdGlvbnMvY2hlY2tvdXRAdjIKICAgICAgLSBuYW1l\nOiBTZXQgdXAgSGFza2VsbAogICAgICAgIHVzZXM6IGFjdGlvbnMvc2V0dXAt\naGFza2VsbEB2MS4xLjMKICAgICAgICB3aXRoOgogICAgICAgICAgZ2hjLXZl\ncnNpb246IDguOC40CiAgICAgICAgICBlbmFibGUtc3RhY2s6IHRydWUKICAg\nICAgICAgIHN0YWNrLXZlcnNpb246IDIuNS4xCiAgICAgIC0gbmFtZTogUnVu\nIHRlc3RzCiAgICAgICAgcnVuOiBzdGFjayB0ZXN0Cgp7LSMgU1RBUlRfRklM\nRSAuZ2l0aWdub3JlICMtfQoqLmNhYmFsCi5zdGFjay13b3JrCgp7LSMgU1RB\nUlRfRklMRSBwYWNrYWdlLnlhbWwgIy19Cm5hbWU6ICAgICAgICAgICAgICAg\nIHt7bmFtZX19CnZlcnNpb246ICAgICAgICAgICAgIDAuMS4wLjAKZ2l0aHVi\nOiAgICAgICAgICAgICAgInt7Z2l0aHViLXVzZXJuYW1lfX17e15naXRodWIt\ndXNlcm5hbWV9fW1hampvaGF7ey9naXRodWItdXNlcm5hbWV9fS97e25hbWV9\nfSIKbGljZW5zZTogICAgICAgICAgICAgSVNDCmF1dGhvcjogICAgICAgICAg\nICAgICJ7e2F1dGhvci1uYW1lfX17e15hdXRob3ItbmFtZX19TWF0aGlhcyBK\nZWFuIEpvaGFuc2Vue3svYXV0aG9yLW5hbWV9fSIKbWFpbnRhaW5lcjogICAg\nICAgICAgInt7YXV0aG9yLWVtYWlsfX17e15hdXRob3ItZW1haWx9fW1hdGhp\nYXNAbWpqLmlve3svYXV0aG9yLWVtYWlsfX0iCmNvcHlyaWdodDogICAgICAg\nICAgICJ7e2NvcHlyaWdodH19e3teY29weXJpZ2h0fX1Db3B5cmlnaHQgKGMp\nIHt7eWVhcn19e3teeWVhcn19MjAyMHt7L3llYXJ9fSB7e2F1dGhvci1uYW1l\nfX17e15hdXRob3ItbmFtZX19TWF0aGlhcyBKZWFuIEpvaGFuc2Vue3svYXV0\naG9yLW5hbWV9fXt7L2NvcHlyaWdodH19IgoKZXh0cmEtc291cmNlLWZpbGVz\nOgotIFJFQURNRS5tZAotIExJQ0VOU0UKCmRlc2NyaXB0aW9uOiAgICAgICAg\nIFBsZWFzZSBzZWUgdGhlIFJFQURNRSBvbiBHaXRIdWIgYXQgPGh0dHBzOi8v\nZ2l0aHViLmNvbS97e2dpdGh1Yi11c2VybmFtZX19e3teZ2l0aHViLXVzZXJu\nYW1lfX1tYWpqb2hhe3svZ2l0aHViLXVzZXJuYW1lfX0ve3tuYW1lfX0jcmVh\nZG1lPgoKZGVwZW5kZW5jaWVzOgotIGJhc2UgPj0gNC43ICYmIDwgNQoKbGli\ncmFyeToKICBzb3VyY2UtZGlyczogc3JjCgp0ZXN0czoKICB7e25hbWV9fS10\nZXN0OgogICAgbWFpbjogICAgICAgICAgICAgICAgU3BlYy5ocwogICAgc291\ncmNlLWRpcnM6ICAgICAgICAgdGVzdAogICAgZ2hjLW9wdGlvbnM6CiAgICAt\nIC10aHJlYWRlZAogICAgLSAtcnRzb3B0cwogICAgLSAtd2l0aC1ydHNvcHRz\nPS1OCiAgICAtIC1XYWxsCiAgICBkZXBlbmRlbmNpZXM6CiAgICAtIHt7bmFt\nZX19CiAgICAtIGhzcGVjCiAgICAtIGhzcGVjLWRpc2NvdmVyCgp7LSMgU1RB\nUlRfRklMRSBSRUFETUUubWQgIy19CiMge3tuYW1lfX0KIVtDSV0oaHR0cHM6\nLy9naXRodWIuY29tL3t7Z2l0aHViLXVzZXJuYW1lfX17e15naXRodWItdXNl\ncm5hbWV9fW1hampvaGF7ey9naXRodWItdXNlcm5hbWV9fS97e25hbWV9fS93\nb3JrZmxvd3MvQ0kvYmFkZ2Uuc3ZnKQoKIyMgTGljZW5zZQpTZWUgW0xJQ0VO\nU0VdKGh0dHBzOi8vZ2l0aHViLmNvbS97e2dpdGh1Yi11c2VybmFtZX19e3te\nZ2l0aHViLXVzZXJuYW1lfX1tYWpqb2hhe3svZ2l0aHViLXVzZXJuYW1lfX0v\ne3tuYW1lfX0vYmxvYi9tYWluL0xJQ0VOU0UpLgoKey0jIFNUQVJUX0ZJTEUg\nTElDRU5TRSAjLX0KQ29weXJpZ2h0IChjKSB7e3llYXJ9fXt7XnllYXJ9fTIw\nMTl7ey95ZWFyfX0ge3thdXRob3ItbmFtZX19CgpQZXJtaXNzaW9uIHRvIHVz\nZSwgY29weSwgbW9kaWZ5LCBhbmQgZGlzdHJpYnV0ZSB0aGlzIHNvZnR3YXJl\nIGZvciBhbnkgcHVycG9zZQp3aXRoIG9yIHdpdGhvdXQgZmVlIGlzIGhlcmVi\neSBncmFudGVkLCBwcm92aWRlZCB0aGF0IHRoZSBhYm92ZSBjb3B5cmlnaHQg\nbm90aWNlCmFuZCB0aGlzIHBlcm1pc3Npb24gbm90aWNlIGFwcGVhciBpbiBh\nbGwgY29waWVzLgoKVEhFIFNPRlRXQVJFIElTIFBST1ZJREVEICJBUyBJUyIg\nQU5EIFRIRSBBVVRIT1IgRElTQ0xBSU1TIEFMTCBXQVJSQU5USUVTIFdJVEgK\nUkVHQVJEIFRPIFRISVMgU09GVFdBUkUgSU5DTFVESU5HIEFMTCBJTVBMSUVE\nIFdBUlJBTlRJRVMgT0YgTUVSQ0hBTlRBQklMSVRZIEFORApGSVRORVNTLiBJ\nTiBOTyBFVkVOVCBTSEFMTCBUSEUgQVVUSE9SIEJFIExJQUJMRSBGT1IgQU5Z\nIFNQRUNJQUwsIERJUkVDVCwKSU5ESVJFQ1QsIE9SIENPTlNFUVVFTlRJQUwg\nREFNQUdFUyBPUiBBTlkgREFNQUdFUyBXSEFUU09FVkVSIFJFU1VMVElORyBG\nUk9NCkxPU1MgT0YgVVNFLCBEQVRBIE9SIFBST0ZJVFMsIFdIRVRIRVIgSU4g\nQU4gQUNUSU9OIE9GIENPTlRSQUNULCBORUdMSUdFTkNFIE9SCk9USEVSIFRP\nUlRJT1VTIEFDVElPTiwgQVJJU0lORyBPVVQgT0YgT1IgSU4gQ09OTkVDVElP\nTiBXSVRIIFRIRSBVU0UgT1IKUEVSRk9STUFOQ0UgT0YgVEhJUyBTT0ZUV0FS\nRS4KCnstIyBTVEFSVF9GSUxFIFNldHVwLmhzICMtfQppbXBvcnQgRGlzdHJp\nYnV0aW9uLlNpbXBsZQptYWluID0gZGVmYXVsdE1haW4KCnstIyBTVEFSVF9G\nSUxFIHNyYy9NYWluLmhzICMtfQptb2R1bGUgTWFpbiB3aGVyZQoKbWFpbiA6\nOiBJTyAoKQptYWluID0gZG8KICBwdXRTdHJMbiAiaGVsbG8gd29ybGQiCgp7\nLSMgU1RBUlRfRklMRSBzcmMvU2ltcGxlLmhzICMtfQptb2R1bGUgU2ltcGxl\nIHdoZXJlCgp7LSMgU1RBUlRfRklMRSB0ZXN0L1NpbXBsZVNwZWMuaHMgIy19\nCm1vZHVsZSBTaW1wbGVTcGVjIChzcGVjKSB3aGVyZQoKaW1wb3J0IFNpbXBs\nZQppbXBvcnQgVGVzdC5Ic3BlYwoKc3BlYyA6OiBTcGVjCnNwZWMgPSB1bmRl\nZmluZWQKCnstIyBTVEFSVF9GSUxFIHRlc3QvU3BlYy5ocyAjLX0Key0jIE9Q\nVElPTlNfR0hDIC1GIC1wZ21GIGhzcGVjLWRpc2NvdmVyICMtfQo=\n",
    "encoding": "base64",
    "_links": {
        "self": "https://api.github.com/repos/majjoha/stack-templates/contents/simple.hsfiles?ref=main",
        "git": "https://api.github.com/repos/majjoha/stack-templates/git/blobs/ee253ece365710a3f334ebd4d63eb82dd2fd948c",
        "html": "https://github.com/majjoha/stack-templates/blob/main/simple.hsfiles"
    }
}

and the download_url is what you want. But that means you'll have to change it into an IO action, which could possibly fail, and you will probably need to do the same for Gitlab and Bitbucket, not sure if this is a good idea, just a suggestion.

[qrilka]

This looks to be an interesting way to resolve the situation. BTW download_url doesn't seem to be necessary - content already contains the needed template. So the only extra step would be to extract the field and decode its text.

[majjoha]

Thank you for your helpful feedback, @qrilka. I like the simplicity of your suggestion a lot, @jeffhappily, and it seems that running curl -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/majjoha/stack-templates/contents/simple.hsfiles" | jq -r .content | base64 --decode expands to the expected template correctly.

Since I am still fairly inexperienced, I think this might be too big a mouthful for me, though, but I'd love to either join somebody for a pairing session to see how they'd go about implementing it or look at a diff at some point.

[qrilka]

@majjoha I don't think it is very complicated - at the moment it looks to me that the main step would be to change

 ["https://raw.githubusercontent.com", "/", user, "/stack-templates/master/", name]

to

["https://api.github.com/repos/", user, "/stack-templates/contents/", name]

and then add an extra step which for Github will extract the field "content" and decode it and for Gitlab/Bitbucket will leave the contents as is.
Parsing JSON is quite easy with aeson and for base64 Stack already uses base64-bytestring.
Does this sound like good starting points for a PR implementing this new approach?

[qrilka]

@majjoha as there were no reply from you I went ahead and created a PR implementing what I have described above - #5447

[majjoha]

@qrilka: My apologies for the late reply. Work has taken up a lot of my time lately, so I have not been able to spend too much time on GitHub.

Thank you for looking into a solution to the issue. It is incredibly helpful and a good learning experience to have the code available to learn from.

[qrilka]

Sure @majjoha - no problem.
Thanks for raising the issue and spending your time on it.