Create Homebrew tap?

[agilgur5]

Problem

Installation currently requires not just downloading pre-built binaries via GitHub Releases, but also cloning the repo or otherwise downloading the docopts.sh helper.

Suggested Solution

Distributing via a package manager could make it much easier to get started with docopts, especially for Shell projects that don't have a language-specific package manager to use.

On macOS at least, Homebrew is the defacto default package manager (Brew also works on Linux, though there are better package managers available there). Can create a repo under the docopt organization, homebrew-docopt which will serve as the Homebrew Tap. Taps are just git repos (with Ruby code), so it's relatively straightforward in that sense.

An example of a simple Homebrew tap is Instrumenta's (creator of Conftest, kubeval, etc).
As you can see there, a simple Formula can just reference a GitHub Release, acting as a simple wrapper around the existing pre-built binaries.

Contributing

Would be happy to contribute building this or even build a standalone repo myself and transfer it to the docopt organization if that is acceptable.

Misc

I used docopts at my last job and it helped a ton with both documentation and arg/option parsing, but installation was my team's main gripe with it. Awesome tool that I would love to make easier to install!

[Sylvain303]

Hello @agilgur5

Thanks reporting. I'm not a mac user, so for giving help to support mac stuff, I need a way to access a machine that would behave like a mac (either a contributor with a mac testing stuff together, or a mac VM that I could ssh to, or a container if such exists).

In the past I was using travis CI but the service API or the license changed and it doesn't work anymore. I didn't spent time on it. See #54

I don't use brew neither, I have to learn it before I could provide something.

[doctops] installation was my team's main gripe

Ah, virus are everywhere those time. 😉 So you created an issue, good point.

As you may have read from the README: doctops works without docopts.sh as a lower level command line tool.

That was it first design. I introduced docopts.sh by sharing some script I made, especially from building more and more examples. May be the doc is not relevant anymore, and should be refactored to push the docopts.sh wrapper first now.

Or I could embed docopts.sh into the binary and output it on the fly. I was reluctant to embed more code to the docopts output as the eval becomes to have huge impact on your shell code. And could be seen as unsafe from sysadmin point of view, especially if you use docopts for root privileges script.

Having a packaged version of docopts including binary, man page, and an extra shell lib is a long running wish, see #22.

Actually I'm involved in a huge docopts Go code refactoring, not spending time about packaging.

This repos is only for docopts the shell version for bash, so I can't help about other language support for docopt. Are you asking for multiple language support of some homebrew package or only one for shell version?

How can I help you?

I mean, given you mention a top-level repos homebrew-docopt but also boot strapping Shell projects.

I can accept a pull request for docopts about a sub-directory for handling an homebrew formula for mac.

The doc you linked says:

Tap formulae follow the same format as the core’s ones, and can be added under either the Formula subdirectory, the HomebrewFormula subdirectory or the repository’s root. The first available directory is used, other locations will be ignored. We recommend use of subdirectories because it makes the repository organisation easier to grasp, and top-level files are not mixed with formulae.

We can also review the documentation for explaining things differently on how to install docopts, as docopts is barely installed only in PATH would suffice. Including doctops.sh. Or refactor documentation about explaining more distinctly the difference between the binary and the helper.

etc.

Please explain your goal more precisely on what would be achieved, given the docopts scope.

[agilgur5]

Thanks for the quick response @Sylvain303 !

Can definitely help on the Mac front

I need a way to access a machine that would behave like a mac (either a contributor with a mac testing stuff together,

Yea of course, I'd be willing to fully contribute and test this myself 🙂
Per my opening comment, Brew does work on Linux (and WSL), so it can be tested without a Mac too.

In the past I was using travis CI but the service API or the license changed and it doesn't work anymore. I didn't spent time on it. See #54

Ah I went through this change myself recently in some of my OSS repos so I can actually help with that too! I'll comment there with more details

Ah, virus are everywhere those time. 😉

I did actually start that job in the middle of lockdown in 2020 haha 😅

Sidebar -- docopts.sh entrypoint wrapper

As you may have read from the README: doctops works without docopts.sh as a lower level command line tool.

Yup, I've used both! docopts.sh is a convenient wrapper with its auto parsing, and since the examples reference it, it made it a bit easier to write code using the wrapper since it would resemble the examples.

Or I could embed docopts.sh into the binary and output it on the fly. I was reluctant to embed more code to the docopts output as the eval becomes to have huge impact on your shell code.

Ah I see, yea those are trade-offs for sure. I do think parts of docopts.sh could be rewritten in Go to reduce the output to eval but 🤷 trade-offs are hard.

I think there's actually a way around that though! You could have docopts just output docopts.sh with a command. So then the user can either put it on disk somewhere (e.g. on $PATH) or use it on the fly if they don't care as much about preference.

This technique is actually really similar to how some binaries output shell completions nowadays. For instance, in Kubernetes-land, you can use kubectl completion bash to output Bash completions for the kubectl CLI. Per the linked doc, you can add it to your .bashrc simply with echo 'source <(kubectl completion bash)' >>~/.bashrc.

So you could add a command, like docopts wrapper docopts.sh or something and it would output the script. You can see kubectl's Go code for this here -- it basically just outputs a Shell file. I actually took a similar approach at my current job for an internal CLI too, so I'm well acquainted with this approach now, happy to help with that approach, giving you more details or maybe writing a prototype PR for it! We might want to split this discussion into a separate issue then

And could be seen as unsafe from sysadmin point of view, especially if you use docopts for root privileges script.

One of my roles is as a Security Engineer, so to some extent I agree, but to another, you have to use eval either way, so 🤷 . The method I outlined above can get around eval'ing the whole wrapper though when output to disk (on $PATH or otherwise sourced).

Back to Homebrew and packaging / distribution

Having a packaged version of docopts including binary, man page, and an extra shell lib is a long running wish, see #22.

Ah, I hadn't seen that issue. I have a tiny bit of experience with Debian packages. May or may not be able to help with that. I remember seeing one repo with a fully automated and simple Debian publishing process, so I've been meaning to find that anyway. I'll link it there if I find it again.

This repos is only for docopts the shell version for bash, so I can't help about other language support for docopt. Are you asking for multiple language support of some homebrew package or only one for shell version?

Just Shell!
Per my opening comment, the other languages have their own language-specific package managers, so they wouldn't need Brew etc. Since Shell has no package manager, people generally use the OS package manager, and Brew is macOS's defacto one.

I mean, given you mention a top-level repos homebrew-docopt but also boot strapping Shell projects.

Ah I see where the confusion is here. This is actually a Homebrew convention to use such a top-level named repo that references other repos. From the Taps docs:

• brew tap <user/repo> makes a clone of the repository at https://github.com/<user>/homebrew-<repo>

Repository naming conventions and assumptions

On GitHub, your repository must be named homebrew-something to use the one-argument form of brew tap. The prefix “homebrew-“ is not optional. (The two-argument form doesn’t have this limitation, but it forces you to give the full URL explicitly.)

So this is just a standard convention in Brew and required to use the one argument form, e.g. brew tap docopt/docopt.

A top-level repo can also hold Formulae for other, related Shell packages, like for #35 / docopt.sh. A "Tap" is just a list of packages basically. The subdirectories contain "Formulae" for how to install each package. So as a "list", it usually makes more sense as a separate repo.

That being said, from that issue it seems like you don't have permission to make top-level repos in the docopt org? So that might be a blocker in and of itself.

I can accept a pull request for docopts about a sub-directory for handling an homebrew formula for mac.

It isn't required to use the convention, so we can use a subdirectory instead. There are two caveats to that:

1. Installing the Tap (before getting the Formula) would require the two argument command: brew tap docopt/docopts https://github.com/docopt/docopts. From there (once you have the Tap/the list), you can install docopts itself with brew install docopts.
2. Installation of the Tap (just the list of packages) would clone this whole repo as opposed to a tiny repo that just references the releases. A good bit more inefficient in that sense.

Let me know what you think of those trade-offs. I can still make a PR for a subdirectory if that's what you would prefer as opposed to a separate repo.

[agilgur5]

Hmm if this is considered "Acceptable" by Homebrew, I might be able to PR it to the main Homebrew repo instead actually. Small libraries and owners usually aren't acceptable but docopts might be large enough and I'm not the owner, so that might work out. They do require builds though as opposed to binaries, so that's a bit more complicated. I might file an issue there and reference this then. Would be good for you to be informed of that either way.

Regardless of the outcome of that, I can still help with embedding docopts.sh and CI in separate issues etc 🙂

[Sylvain303]

Hello @agilgur5

A lot of stuff here. But it's OK because of the discussion around the distribution mechanism and related issues digs quite a lot of question and thinking. 😉

Lets push discussion and sample of embedding docopts.sh in #60. I created a dev-branch with a sample of code.

We will definitely discuss CI and mac related testing in #54.

You said, « Brew does work on Linux » great but tools differ from mac and Linux (sed, awk, realpath, bash, etc.) So re-creating a mac like environment is mandatory, see other issues mac related...

But we would fix that together. I also could explore, having a docker VM for that:
https://www.linuxuprising.com/2021/03/install-macos-big-sur-or-catalina-in.html

Back on track about homebrew related steps

docopt Organization: long story short

I'm not a member of docopt Organization right, so I can't create toplevel repository myself. docopt story came with the python version first, which is at abandon, with a fork solely for python.

It took me two years to be granted as new docopts maintainer. #5

I probably could ask more privileges. I will do it when my new Go parser will be ready.
Still a huge amount of work, on my spare time. 😉

Typo docopts and docopt without S

docopt without S is the lib. It has been re-implemented in many language all based on python version first. Some has evolved.

docopts is a tool that behaves like a lib for bash (some other shell could probably also eval some form of output). I suppose the name is a play upon word with getopts the internal bash option parser based on C lib.

So I suppose you missed an S here:

So this is just a standard convention in Brew and required to use the one argument form, e.g. brew tap docopt/docopt.

So I should read: brew tap docopt/docopts

And probably the same (or not) with toplevel repository naming convention:

homebrew-docopt ?

But may I have to learn deeper Homebrew convention to understand.

docopt.sh cousin

docopt.sh from #35 is an interesting approach that generates the shell code fixed into the generated script. It uses python docopt lib and somewhat catches the AST at some point and generates some shell code that will be embedded into a fixed docopt syntax validator in the resulting shell script. With all the mechanism of auto-modifying the generated code.

docopt.sh is not part of docopts project. I made some cross generator code for exploring how it behaves but we suspended our exchanges at some out-of-time spare period, I suppose.

The generated shell code is standalone, as the docopts binary provides, without the need of python nor associated lib. A dev mode exists which self re-generate or something.

Writing a docopt language parser in pure bash is not possible for performance reason. So both approach docopts and docopt.sh uses programming languages to handle the huge parsing task and output text. Both could be seen as text processor. docopt.sh goes farther with handling destination file manipulation routine.

docopt.sh is distributed via pip as a python tool, so I suppose we could omit brew install for that. So it may become more clear for me what we are trying to accomplish here.

Go pre-build binary distribution

I rewrote docopts in Go to get rid of python dependency #9, and because the Go lib was available and the easily produced statically linked binaries. Which now brings us to the distribution mechanism. 😉 I crafted a tool for producing github releases: deploy.sh which I moved outside the docopts repository at some point for reusing.

This all are some kind of workaround while I didn't yet generate a valid debian package for that. Which is a quest on its own about learning package building, debian workflow and golang package building the debian way. And also about statically linked vs dynamically linked conflict that could raise about packaging and docopts own goal to be standalone.

So in the mean time, I'm pushing some cross-compiled binaries to github releases. 😄

binary distribution and security concern

As you mentioned, other docopt project are language oriented and provide their own packaging for distribution. On the shell side there's no such distribution, or it should be packaged via distribution packaging system. As sysadmin I use a lot of shell script for managing root privileges tasks. So a docopts binary must become a point of trust. Which should be accomplished by installing an official package, or by trust delegation, or building your own binary.

This is because of the eval, which is not part of other lib version in other languages. I explored some other way not using eval, but the shell API comfort and performance decrease.

And as you also mentioned some are ready to trust source <(kubectl completion bash). And as it could be saved and checked, it some kind of acceptable compromise. I will never accept a curl-pipe-sudo-bash on the other side. It's another interesting debate that will go elsewhere.

But all those delayed me to provide robust trusted distribution of pre-built binaries.
That's also why I encourage people to build it manually via a Go developer of their friend.

All that said, let's go back to homebrew packaging.

homebrew steps

For now you seem to know brew far better than me and you can test it too, especially on mac side. Which the purpose here. For Linux side, I'm not sure about the benefit of such package compared to an official package. May be only the time involved to build it, the learning curve.

So I agree that you could explore both having the Formulae hosted in some repos of your first or in a sub-folder in the docopts repository. Having the drawback of supplying the repository path at install time if I understood well.

Or it could be a single orphan branch (git clone --single-branch -b homebrew [email protected]:docopt/docopts.git homebrew-docopts) if brew supports it. Some repository uses orphan branches mechanism to store things (doc for example)? 🤔

Then we will ask to create a toplevel repos in docopt organization when it becomes stable.

We should still define what we want to become the brew package content.

I let you come back with some suggestion here.

[agilgur5]

macOS testing

You said, « Brew does work on Linux » great but tools differ from mac and Linux (sed, awk, realpath, bash, etc.) So re-creating a mac like environment is mandatory, see other issues mac related...

Very true! I meant more as like a "smoke test" of sorts that if the Brew package works on Linux, there's a good chance it'll work on macOS with the proper binary.
I have seen some of the other Mac issues, which I did notice some are due to the new ARM64 architecture, which is a whole other compatibility issue. I don't have an ARM Mac either for testing that locally.

But we would fix that together. I also could explore, having a docker VM for that:
https://www.linuxuprising.com/2021/03/install-macos-big-sur-or-catalina-in.html

Oh I think I've heard of this before. There's also Docker for ARM which I have used a handful of times for virtualization in robotics contexts.

docopt/docopts history

Org

I'm not a member of docopt Organization right, so I can't create toplevel repository myself.

Gotcha

docopt story came with the python version first, which is at abandon, with a fork solely for python.

Yep I remember this. I didn't know there was an updated fork of the Python version though, that's good to hear! I was sad when I saw a lot of the docopt libraries in this org weren't maintained despite it being such a great tool 😢

Still a huge amount of work, on my spare time. 😉

I'm an open-source maintainer and contributor myself (see my profile), so I definitely get the struggle 😅
And the company where I used docopts last at imploded and laid off all staff 1.5 years ago, so I'm doing this in my spare time too. Because it's a great tool!

s vs. no s

docopt without S is the lib. It has been re-implemented in many language all based on python version first. Some has evolved.

Yup I'm aware of that

So I should read: brew tap docopt/docopts

Ah sorry for the confusion. If it were a top-level Tap that could potentially hold multiple Formulae, I think docopt (no s) might make more sense. You would do brew tap docopt/docopt && brew install docopts.
But if it were a subdirectory in this repo and only had this one Formula for docopts, then yes, I agree docopts (with s) probably makes more sense as there's only one thing in it anyway. So then it would be brew tap docopt/docopts https://github.com/docopt/docopts && brew install docopts

Hopefully that's less confusing now 😅

cousin

docopt.sh from #35 is an interesting approach that generates the shell code fixed into the generated script

Yea I checked it out 2 years ago too and had a similar opinion. Very interesting. But didn't fit the use case of what I working on as we had like ~20 tiny scripts, so that would be a heck of a lot code to generate as opposed to just calling docopts in each one.

docopt.sh is distributed via pip as a python tool, so I suppose we could omit brew install for that.

Yea I suppose that's true since it does depend on the Python version too. I honestly hadn't looked at it in a while so I totally forgot it used pip already. Thanks for pointing that out!

Go build

I rewrote docopts in Go to get rid of python dependency #9, and because the Go lib was available and the easily produced statically linked binaries

Yea I thought that was a great idea! Much easier to work with statically linked binaries for shell scripts!

I crafted a tool for producing github releases: deploy.sh which I moved outside the docopts repository at some point for reusing.

Ohhhh. I was looking at the Makefile earlier and saw it referenced deploy.sh but I couldn't find a deploy.sh script in this repo. Now I know where it is!

Which is a quest on its own about learning package building, debian workflow and golang package building the debian way

Yea Debian packaging is definitely more complicated than Brew at least 😅

So in the mean time, I'm pushing some cross-compiled binaries to github releases. 😄

It's a great solution for now! 👍 👍
And honestly, good to produce binaries with GH releases anyway, even if other distributions exist in the future -- having it from the source is always a good option.

So a docopts binary must become a point of trust. Which should be accomplished by installing an official package, or by trust delegation, or building your own binary.

Agreed

It's another interesting debate that will go elsewhere.

haha that it would

That's also why I encourage people to build it manually via a Go developer of their friend.

Ah I was wondering why the README was written the way it was.

So, interestingly enough, homebrew/core requires building the binary as part of the installation, so it effectively becomes an automated build.
As you might suspect though, there's a lot of trade-offs to building libraries locally, and that's probably one of the reasons why Homebrew can sometimes be a very finicky package manager to use.

For third-party Taps, this isn't required though, so can start with binaries and then could expand to doing a full build.

Brew

Which the purpose here. For Linux side, I'm not sure about the benefit of such package compared to an official package. May be only the time involved to build it, the learning curve.

Yea I don't think there's much "real" benefit for Linux either.

So I agree that you could explore both having the Formulae hosted in some repos of your first or in a sub-folder in the docopts repository. Having the drawback of supplying the repository path at install time if I understood well.

Ok, sounds like a plan then!

Or it could be a single orphan branch (git clone --single-branch -b homebrew [email protected]:docopt/docopts.git homebrew-docopts) if brew supports it. Some repository uses orphan branches mechanism to store things (doc for example)? 🤔

That's an interesting idea 🤔 Unfortunately it does not seem like Brew supports this from what I could find online:

• Two old Brew issues
• This SO answer shows an unofficial way of installing a Formula from a commit hash. But that's a Formula, not a Tap. And it's not officially supported either way (I believe it breaks the updating functionality of Brew and potentially some other things).
• The three argument form of brew tap where you specify the URL is the only other form. There's no other variation to that has a branch name argument. Branches are supported in Git Hub URLs, but I don't think can be made into a git URL (something like your command is necessary, as far as I recall at least).

We should still define what we want to become the brew package content.

1. My plan is to start with just the binaries at first.
2. Then can add docopts.sh (embedded docopts.sh wrapper into docopts binary #60 may obviate the need for that)
   • Possibly as an option (e.g. brew install docopts --with-docopts-sh) or just automatically for everyone. Notably options aren't supported by homebrew/core
3. Then can go for a Formula that does a whole build. That was why I was looking at the Makefile and deploy.sh etc.
4. That one I could then submit to homebrew/core for possible acceptance in the core Tap.

Notably, I saw that goreleaser supports Brew, but it's possibly a bit too automated (it looks like it pushes to git for you?). And it just links the binaries, so it's pretty simple to do without as well. Just might want some automation to change versions and SHAs automatically.

• It also has a "Note" at the bottom saying it "does not generate a valid homebrew-core formula" and it "will not be accepted in any of the official homebrew repositories". It is just meant for third-party Taps: "generated formulas are meant to be published as homebrew taps"

[Sylvain303]

goreleaser

goreleaser

I didn't know this software, may it could replace deploy.sh

As mentioned I didn't spent time investigating on releases. And rereading some code I produced earlier, I see I was lost in writing shell script for Bash expert / hacker / PoC. 🤔

Some cleanup will come in comment and tools, that's OK.

brew build / go build

You will need to build the code, you can either try make or setup deploy.sh and do a deploy.sh build I will hack and post some update in some alternative Dockerfile so it could show you the required command to set it all.

You may ultimately have a full build env in a docker at the end. I didn't try it since month, I've to look at it first.

updating doc and rephrase

[binary distribution and security concern]

Ah I was wondering why the README was written the way it was.

Following our exchange I will document that too, in order to explain my choice and why I discourage getting not signed / reviewed pre-build binaries, for a script that could possibly eval random code with root privileges.

your side

I still didn't explore yet homebrew more than the few quote we exchanged.

Neither I did on github Actions, nor of setting up macos like VM.

I will let you know when I will have levelup ➕ 🆙

Let me know your blocker I will investigate my code, to remove barrier, if any. Or give you quick key to speedup your exploration.

[agilgur5]

Hi @Sylvain303, I'm back after a long delay (see #54 (comment) for more details on the delay) and did a good bit of work on this repo today!

goreleaser

I wrote up a PR migrating from deploy.sh to goreleaser in #65!

brew build / go build

Per the PR, I ended up replacing the whole custom dependency management with the now built-in Go modules (i.e. go.mod). This made it a lot easier to get started and I was able to do so on a Mac that I hadn't even installed Go on yet! Hooray for standardization of Go deps finally 😅

So I ended up not needing Docker or anything, though this process can simplify / standardize the Dockerfile (and could create a scratch image for docopts too -- goreleaser even has built-in publishing of Docker images too!).

3. Then can go for a Formula that does a whole build. That was why I was looking at the Makefile and deploy.sh etc.
4. That one I could then submit to homebrew/core for possible acceptance in the core Tap.

Now that I generally understand how this repo works, builds, and the dependency management, I think I should be able to make a PR directly to homebrew-core and avoid a third-party tap.
I should be able to copy another Go formula (e.g. just by random, found this one) and since the build system / dependency management is the same as most Go formula now, it should be relatively straightforward to adapt for docopts 🤞

Neither I did on github Actions, nor of setting up macos like VM.

I wrote up a PR earlier today for that as well, see #62 🙂

Let me know your blocker I will investigate my code, to remove barrier, if any. Or give you quick key to speedup your exploration.

All your explanations here were plenty! Even 4 months after I was able to come back and reference them -- glad that we both have detailed comments 🙂

[agilgur5]

testing tap (+ core coming soon)

and since the build system / dependency management is the same as most Go formula now, it should be relatively straightforward to adapt for docopts 🤞

For testing / interim purposes, I was able to make a really simple Formula in my personal tap here. This can be installed with:

brew install agilgur5/tap/docopts

I had to patch in the go.mod there, but otherwise it's fairly straightforward. I'll be submitting a PR to homebrew-core soon, and I'll see if they allow that patch while #65 / go.mod changes are pending here. They might choose to wait until the go.mod exists, however.

building from source

If you run the command above, you'll see it actually runs go build and will download + install go if needed too (via Go's Brew Formula). So I think this satisfies your security concern well too, as core Brew is designed to build from source 🙂

docopts.sh

I also added docopts.sh to the bin path, which is generally on $PATH, so source docopts.sh can be used from anywhere.

This is only one type of installation pattern, so I still think that #60 would be great to implement for the same reason that kubectl completions bash and other similar patterns exist