-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rewrite in golang lolz =) #126
Comments
I personally agree that rewriting the project sounds interesting. Just being curious, what's the potential benefit of using golang? |
@rami3l I don't really know. Maybe it helps to attract some developers. Now looking at Bash code: it's pretty hard to maintain and I don't think there are too many people who're using .bash. I think bash/shell scripting is the best tool for this kind of job anyway. The original idea is to support limited environment where .bash is not available generally. I tried to rewrite the tool in
Btw, this is already a 10-year project. When I started I couldn't think I would go far like this. :)) |
@icy If I were you, maybe I'll go for Python because:
I'd like to experiment... Although I'm not quite sure of what you are trying to say here:
|
@icy A simple proof of concept has been made public here. PS: I wrote the |
@rami3l Thanks a lot for your contribution and your time. I really appreciate that. I think Python or any higher level language can help a lot. However, we need to consider one of the main advantages of the current I did try the Dlang-version, one of the reasons was it's a compiled language. I know there is someway to do the same in Python (PyInstaller -- which I use a lot in my work), but from my experience it's not very portable. Please note this is my little experience, as I'm not a heavy Python programmer. Let's me write down some goals in my mind now, before we can go further:
Do you think we would obtain the same thing (esp. the 3rd goal) when we rewrite the tool in Python and/or foobar? |
@icy I also have doubts concerning those aspects... TestingI'm not quite sure what tests should be integrated a wrapper like this (due to my lack of similar experience), but I'm sure that any "modern" language (which meets with your 1st point) has a proper way of doing tests far better than what we have now... And we will do those tests on which platforms? If this project needs to be cross-compiled, then the problem might be more complex. PortabilityYour point reminds me of the fact that I don't know quite much about the potential use cases of this project. Surely, if we want to maximize the portability, we should really choose to compile the project. But when I came up with the idea of using Python, what I referred to was to run Python script with an interpreter, i.e to use Python as a scripting language, so that this project may run on any machine that is capable of running a Python 3 interpreter. BTW: PyInstaller is for "freezing" a Python application, and a frozen app is something quite different in terms of portability. So maybe I should ask, what exactly are the special cases in which, for example, the CPython interpreter is not available or at least not convenient to use? (From my experience, an example might be an OpenWRT router with limited flash storage. Installing bash is possible in this case, while Python might look a bit heavy.) It is important to specify these special cases in order to make a good choice between languages. PS: I am learning Golang right now, and I think it's a promising choice if we finally go for a compiled language. Of course, the portability must be considered before making the final decision... |
@rami3l Thanks for your input. That's great to discuss these things TestingWe have unit tests that can help to modify code easily. Writing some tests in bash is not very comfortable. We also need integration / smoke tests because we need to make sure the behavior of the tool is correct on the target machines. For this part the current way can be used (e.g, https://github.com/icy/pacapt/blob/ng/tests/dpkg.txt). PortabilityYes I really want to use the tool on busybox-based environment, where bash (or even .sh) is not generally available. There are also many limited environments: router (OpenWrt), some Pi machines, some bsd machines, docker containers. Golang can be built on many target machines as I know, and it's also easy to build on different targets (see for example https://www.digitalocean.com/community/tutorials/how-to-build-go-executables-for-multiple-platforms-on-ubuntu-16-04) Fyi, there is already a C-based project https://github.com/emilengler/sysget . It's doing well it's just not pacman-liked though |
@icy Maybe we can start trying to refactor this project using Golang, and I think |
@icy I have trouble translating my Python script in Golang in the PS: Here I choose to import a library. |
@rami3l Thanks a lot for your initial work. Let me write down some important aspects of pacman..., which I tend to forget now. As far as I remember, if the parser can support multiple counting (number of appearances of any option). I have a lot of time now, to spend a bit on pacman. Give me a little time to sell out my guitar :D PS: TLDR; In my dlang code, it's very easy to do that (https://github.com/icy/pacapt/blob/nd/dmain/internals.d#L467). During the last few weeks, I discussed quite a lot with a team mate, to answer to the question, does that really make sense to write everything in I don't know. Last week, I just wrote 300 lines of Bash code that cover most important features of this k8s/cd tool : https://github.com/fluxcd/flux , and it also can do a lot more useful things for my company (to be honest, I also need to write 2 jiffy tools in golang, one of them is the prometheus exporter; but they are optionally). If the problem is that we can't write some tests in Bash, why don't people try to solve that problem? Oh please! |
@icy Yep! Flag problems seem to have been solved, including flag counters. As for language use, I still think a modern language will help, though it's important to help a bash-based project have a smooth transition. I wrote that "all should be written in Still, for experimental reasons, I'll rewrite at least |
@icy By the way, do you think that we can utilize GitHub Actions to simplify the testing? |
@rami3l Github Action is also fine, though it's a bit scary to use some 3rd party actions. For small projects, I don't think there is any huge difference between GithubActions and Travis/CircleCi. You may go with any tool you feel comfortable then ;) |
@icy I also referenced your Dlang version, I personally know nothing about Dlang, yet it seems like a transcription with the same global variable logic as the |
Hi @rami3l , sorry again with my belated response. And thank you a lot for spending your time on the project. I really appreciate that. The dlang version was simply a direct port of the bash scripts, with some minor features. Let me write down some important things of the orginal (1) understand user inputScripts:
These contain the main argument parsing steps, and the goal is to have almost the same option sets as the ArchLinux pacman (see This part is heavily relying on ( Starting with another goal would be a lot easier (e.g, (2) understand the system targetNow when the script understand user input, it needs to translate them into the target's commands. Hence the next step is to understand the current environment: https://github.com/icy/pacapt/blob/ng/lib/00_core.sh . This script basically detects the type of the OS/package manager, and it's quite easy to rewrite them in golang/python Imho. Nowadays (3) connect/invoke system package managers (and/or transform the output)The remained parts in the Basically, in this step, you will issue the system commands and capture their output. Sometimes an important thing is to deal with interactive command: For example, when you run As you can see in my dlang version, I just slightly modify the scripts under Line 728 in f3ae5f8
Line 775 in f3ae5f8
Line 342 in f3ae5f8
pass-through option when you want to just skip all pacapt features (e.g, pacapt -v -- yum install is exactly yum install ) -- that's a crazy thing you can ignore for now. To me, that way of invoking external scripts is efficient, unsafe, and yeah, it's still bash script so people may hate that ;)
Feel free to ask me any questions:100: |
@icy Thanks a lot for your reply! It turns out that even though I don't know much about the original project structure, I still broke my task into exactly these different pieces. So, I'd also like to share with you some details of my current work, so that we might come up with better ways to solve problems that might emerge... 1) Argument ParsingThis is not the difficult part for me. The argument parsing library I used is argparse, but in a way far from elegant. This library does not support flags as subcommands (only words, but so as many other argument parsers, so I chose just to stick with this one), so I managed to strip all the flags away from the package names, store the names directly for later, and let the parser deal with the flags only: // stripTargets distinguishes between pacapt flags and package names.
func stripTargets(args []string) (cmd []string, keywords []string) { ... } So if I want to add 2) Platform DetectingThis is where I got a bit confused. For a long time I just left this part blank and hardcoded the detection result to be "Homebrew": // DetectPackManager detects the package manager in use
// TODO: Make this function REALLY detect package managers
func DetectPackManager(dryRun bool, noConfirm bool) PackManager {
switch runtime.GOOS {
case "windows":
return &Chocolatey{dryRun, noConfirm}
case "darwin":
return &Homebrew{dryRun}
default:
return &Unknown{dryRun, noConfirm}
}
} Maybe now we can start to really do this part, but what's the most elegant way? What are the "special" cases to be aware of? Should we suppose that our users all have their package managers in their PATH, so that I can just use exec.Lookpath()? 3) Invoke Package ManagerAs for the dispatch, I was suggested by a friend to use an interface (if it was in Python, maybe I'll just go for metaprogramming). I also created an Unknown instance of the interface, so that new package managers can quickly be cloned and modified out of it. The interface looks really ugly, but it gets things done (is there a better way?): // PackManager represents a PACkage MANager
type PackManager interface {
Q([]string) error
Qc([]string) error
Qe([]string) error
...
} Also, after some research, I found out that it's not that worthwhile to stick with bash scripts. So instead, I offered the following functions to facilitate things... // RunIfNotDry prints out the command if DryRun, else it runs the command.
func (hb *Homebrew) RunIfNotDry(cmd []string) (err error) { ... }
// CheckOutput runs the command and returns its output both to a string and to Stdout (ignored if DryRun).
func (hb *Homebrew) CheckOutput(cmd []string) (out string, err error) {
...
p.Stdout = io.MultiWriter(os.Stdout, &outBuf)
p.Stderr = io.MultiWriter(os.Stderr, &outBuf)
...
out = outBuf.String()
...
} Please note that the stdout/stderr is redirected (and tee'd if we want to check the output), so the output won't be a problem. If the command is interactive, maybe we just need to set the stdin similarly. // Qu lists packages which have an update available.
func (hb *Homebrew) Qu(kw []string) (err error) {
return hb.RunIfNotDry(append([]string{"brew", "outdated"}, kw...))
} Finally, if really necessary, it's always possible to call 4) TestingI implemented a --dryrun option so that the program might work like this: $ go run main.go -S curl --dryrun
>> brew install curl
$ go run main.go -S gimp --dryrun
>> brew cask install gimp ... though I become quite unsure about how the testing can be done, especially automatically, and across platforms. 5) Homebrew Implementation: Extra SubcommandsI first saw the original homebrew_Rs() function, and then I realized that this part is more difficult than it seems, and there has been heated discussions, like here and here. So should I just call Similarly, I think it's a good idea to invoke Any advice is welcomed :) |
hi @rami3l , there are a lot of details. I will need to read that again. I will now have some quick feedback hoping that helps: 5) HomebrewIt's great if we can support them partially without other dependency. But if the subcommand provides safety and exactness we can just ask the user to install them. 4) TestingWe would go with some smoke tests. As I know, Travis has some macos support, we can add them if we need. For linux-based machines, I generate test scripts thanks to a ruby program: https://github.com/icy/pacapt/tree/ng/tests . The idea is simple: execute the command, and match them against some known pattern. This is the best strategy to make sure the program won't be broken at user runtime 3) System command invocationYes I also like the idea of using some system call instead. It would take a little effort to filter out the output (see https://github.com/icy/pacapt/blob/ng/lib/dpkg.sh ) |
@icy It's nice to see your reply! It's the weekend again so I can continue working on this project. The current structure of my project has made it rather easy to add basic support (i.e. the one-liner part) for a new package manager with some copy-paste work. However, as for the non-one-liner functions, I realized that my lack of knowledge about bash scripting has become a problem for me when it comes to translation without resorting to the I am trying to make sense of bash, though it might be a bit slow, as I am just an amateur trying to learn something during my spare time @_@. It will be so nice if you wish to lend me a hand implementing some of the functions, like Again, feel free to ask anything :) |
@rami3l I think you have a good framework to start now. You may want to have some tests, and start porting the bash things. That's quite a lot of work 💯 |
@icy It's been a month and things are going quite well! I think the best way to keep the momentum is to get happier while coding (I kind of hate the copy-paste approach in my original Golang implementation). So it took me quite a while to decide what to do, and I have finally finished rewriting the whole Golang code base in Rust, the language of my preference, and a language which handles multi-platform tests quite well AFAIK. (Another reason is that clap.rs is said to add pacman-style argument parsing support in the near future.) I'm actually using this Rust version of wrapper on my Mac on a daily basis. Also, I came up with a way to do integration tests. I'll write them once I have the time. Let's see how far I can go from here! PS: The project has been renamed to |
Amazing and congratulation for the first success 💯 I don't know Rust much (At my work people are using Golang). I used to play with Dlang in my free time. However I know Rust is a better choice for many purposes. I'm happy that you've found the right tool and also get tool worked on your daily basis. PS: I don't have much free time at the moment, it's hard for me to work on some hobby projects :( |
@icy I managed to build utilities to check stdout here, and you can see that I borrowed a bit of the test pattern from this project. (Thanks!) Although I'm not quite sure, it looks like we have test cases for just a small part of package managers in this repo, so how's everything else guaranteed... or are they (?_?) PS: It'll be so nice if you could add pacaptr to the Related Projects section of README.md :) |
@rami3l Thanks a lot, and congratulations. I've got some tough problems at my work and was not able to catch up with you. Feel free to open a PR to mention your project there. I believe you don't need to rely on the original idea of pacapt, as that way you have more freedom. When your project is cool and stable enough, we can decide what to do with the current script then. Regarding the tests, it's relying on contributors' works. Many of them have been contributing to the project, and as requiring tests in bash is not an easy task, tests are not enforced. I believe in the contributors , and it's also not very hard to spot the issues in the library functions -- most of them are very small . (The hardest part, in main or core, was heavily tested from my side) Thank you very much , and bests |
Fixed in #131 |
I'd like to get back to this ticket with some updates. Now most parts in the library Your project is now mentioned a new section: https://github.com/icy/pacapt/blob/ng/README.md#similar-projects (There is also We have now more tests and also support some custom Docker image for testing purpose (https://github.com/icy/pacapt/tree/ng/tests). You may find any easy way to sync. them in your project. (kudo to @mondeja who has made a great effort to fix some issues, add new tests and operators.) The project is using quite a bunch of different things https://github.com/icy/pacapt/blob/ng/ARCHITECTURE.md#history-languages . For learning purpose, it's just ... awesome 👍 Regarding the future of the project, I'd suggest
As the general setup hasn't changed much, I hope you can find some easy way to update your project accordingly when applicable (tests, operation implementation). Thanks. |
@icy Thanks for the update! In fact, it was your idea of remaking What you've done is a really nice work, congrats! I still agree that technically, nothing beats At the same time, working on the Rust port is also an enriching experience for me: although I still haven't finished porting all your scripts, I brought your project to Windows, learned a lot about Rust, made a shiny interface, and even contributed to my upstreams. Most importantly, I finally have something written for myself to be used on a daily basis. All of this started with this tiny issue, and this is just amazing! Thank you! |
Thanks for your sharing @rami3l . The tool and the idea may vary, but the open spirit would be the same and that I love to read from your message. Shell is hard. I keep my eyes on this project https://github.com/oilshell/oil, but I've not seen any [work] place I can be ready for it. Give it more time. Maybe you want to help that project too. What I am surprised from your Rust port is how it deals with the test DSL quickly/easily. I started with Dlang for a reason (it's quite friendly to work with new DSL in Dlang), and it's also easy to write new extensions for |
@icy In terms of usage, how is a bash extension (dylib?) different from an executable binary? |
It'd be faster, and give some additional feature to Bash tool. One of the great examples is
That said I've not used the feature heavily. It's because people tends to avoid bash in that situation. 🗡️ |
I've tried the tests (the
|
@icy If this dylib is exposing nothing more than a few "extern C" stuff, then I can't see anything stopping one from doing the same thing in Rust... PS: Speaking of coreutils, we now have a cross-platform Rust rewrite of the GNU coreutils at https://github.com/uutils/coreutils. |
Got it. Thanks for the clarification.
Great. I think I learnt about that from HackerNews, but I haven't tried that yet. |
@rami3l Would you mind recommending some book on system programming in Rust? Thanks. |
@icy You really don't need to buy any book. |
I have a problem. I can't read anything from laptop or reader device (which I don't have.) Yes I do but that's not how I want to spend time with my hobby 👯 |
@icy Lucky for you, the tutorial (a.k.a The Book) also comes with a printed version (and quite expensive in Europe, huh?), but please remember that Rust is a quickly evolving language, and the Book, as I remembered, is already a bit different from what I have read when I was learning Rust. |
Oh I will try again. Yes in Europe books are expensive 🗡️
Yes I know. I already have some out-of-date books here, but it's fine. When I have new thing you can compare with old ones. With online resources it's a little to hard to do that (or I don't know how to do) Basically I have a few weeks off soon, and I need to take a break with new things :P |
@icy I always forget to add |
Thanks for your suggestion. I'm happy to hear that. Please give sometime to write |
No description provided.
The text was updated successfully, but these errors were encountered: