Command ecosystem #572
Comments
|
Makes sense to me. My one hesitation is that we are somewhat duplicating what composer does. We have to think about how Drush (and Drupal) use Composer. That doesn't have to block better github integration. Maybe dl could assume Github is the project name contains a slash. |
|
Well, for one I assume that Drush commands downloaded from Github might use composer to install their dependencies. In general, I don't think the average user would use composer to manage their Drush commands, but that doesn't mean that Drush couldn't use it internally. But if we rely on composer, the question whether to use packagist or something arises. It would be nice to have some standard repository for Drush commands. As it is, they're pretty much unfindable on d.o, nevermind the double work of maintaining a project on d.o too. |
|
I would love to ask all drush commandfiles to include a composer.json with a keyword=drush and list themselves on packagist. unfortunately, i don't see a way on Packagist to list all packages that have a given keyword. It might be that Drush should host its own plugin directory. I might be interested in that. Github webhooks make that not overly difficult. |
|
Actually, Packagist does have a decent search API. The listings are ugly still, unfortunately. |
|
We could make our own nice plugin directory just by using the services of packagist. For listing: https://packagist.org/search.json?tags[]=drush And https://packagist.org/packages/drush/drush.json gives pretty much the info we need, I think. So, I figure that it shouldn't be that hard to implement a packages.drush.org that simply asks packagist and renderes the result nicely. My boss might even throw some designer/themer hours after it just for the fun of it. At least we could make a drush command to get the listing and pretty print the result of second . I'll look into how difficult it would be making drush dl understand packagist vendor/project. |
|
|
Ok, I don't understand point 2. Multiple versions of Drupal in the same directory? |
|
I said that poorly. I think the best practice for Github Drush projects will be files like
|
|
I still don't understand the issue, I must be missing something. For Drush commands related to a module (like rules), the command files For other Drush commands, they're basically version independent per So I don't see the need for versioning commands on Github or packagist?
|
|
OK, I guess I am making this excessively complicated. My concerns make more sense for modules that choose to move to Github and not push their code+releases back to drupal.org. Maybe we don't have to worry about those in this issue. Looking at commands/pm/download.pm.inc, it is pretty awkward IMO hack in downloading of Github projects. I think it would be better if folks edited the composer.json that comes with Drush and ran |
|
One could argue that modules belong on d.o anyway, so I'd say we can ignore I don't like the composer.json way, it requires that any random Drush One could argue for a new extension management command, but then it should Keeping things simple both from a code and end-user perspective might be
|
|
I haven't perused it all, but the wp-cli tool has something called Community Packages which can be installed with composer, among other ways https://github.com/wp-cli/wp-cli/wiki/Community-Packages |
|
I've had some time to look a bit more at composer, and using a composer.json file (rather than Drushs, which gets overwritten when you upgrade Drush) could be the way. As it is now, one can go It means that we're leaving d.o hosted Drush commands in the dust, but that's up to the Drush team to decide. Personally, I think it's fair enough as it's only pure Drush commands that's affected (coder module being the odd case suggesting to install the module in .drush, but even that have a composer version now). We could make a thin Drush wrapper for the Any thoughts? |
|
We are in perfect agreement.
|
|
Good. :-)
Anyway, it should work as it is. I just checked, and Drush uses drush_scan_directory() to find command files, which mean that even though the command file lies below
I'll try making a packagist package of a Drush command and see how/if it works. |
|
Yes, I think some sites will add Drush to their composer.json so that all Since you are interested in these issues, I want to draw your attention to |
|
I've just tried installing https://packagist.org/packages/xendk/bandaid using composer require, and it works just fine, both on master and 6.2. So.. This issue seems to be more about documenting this as the suggested way to install drush commands, and build something that'll provide a nice listing of packagist drush commands. "add Drush to their composer.json"... I'm not familiar with the practice of using composer for Drupal sites, so I'm not sure how things will get laid out (and how that'll affect this issue). Got a link to someone explaining the details? |
|
I'm not sure how they did it but it seems like it could be fairly simple to set up something like https://plugins.drush.org based on http://plugins.roundcube.net/ The downside is that it's another bit of infrastructure to maintain. |
|
@kostajh Looks like they're hosting their own version of the packagist code base. We could do that, but as you said, it's another bit of infrastructure. We've discussed creating a plugins.drush.org as a simple wrapper that just asks packagists for packages with the drush tag, which might even be possible to do clientside. The advantage is that by using packagist, you can simply do a composer require, without having to tell composer about our custom repositiory. |
|
Aah, I guess it shouldn't just be that easy. So, having a command installable through composer, I figured I'd fix up its tests to work with the new autoloaded test classes. The obvious solution would be to require-dev drush so composer would fix up the autoloading, however then we'll end up with a full Drush inside .drush/vendor, which is pretty much guaranteed to cause havoc at some point. Of course I could work around the problem by keeping composer.json elsewhere and symlinking my commands directory into .drush, but that's rather bad DX. |
|
Devel solves this with a by using the --configuration option for |
|
That fixes it. Sidenote: With a bit of fudging, it still possible to run the same tests on Drush 5-7: xendk/bandaid@58fd7b3 which is nice. So we're back to "green" on this issue. |
|
@xendk - would be wonderful to make some progress on a listing app for Drush projects on Packagist. keep us posted! |
|
Actually we have a working site, locally, that just needs to get some of the rougher edges sanded off. But then our resident themer got ill and other things took precedence, but I'll try to get it up somewhere soon. As it is, the entire catalog is 3 extensions, where I suspect that one is dead. But that should change. |
|
There: http://drackage.xen.dk/ Code is here: https://github.com/reload/drackage |
|
@weitzman Had time to check it out? (forgot to @mention) |
|
Sorry for the delay - I lost track of this. I'm a bit sad that the package listing is a Drupal site. Would it be possible to code up a client side version? I think it would be OK to just focus on listings and link to packagist.org for package detail. |
|
Also, I got the idea from http://blog.atom.io/2014/04/25/ci-for-your-packages.html that would could document a sample travis.yml so that Drush commands can easily test themselves using our Unish test framework. |
|
If we leave the package display up to packagist, we loose the copy-pastable command for installing (not all drush users are intimate with composer yet). And while I get the current trend that's making people use Jekyll and the like instead of Drupal for their personal blog, I'm also a bit too old to really care. I picked Drupal as it makes sense to use something that anyone that might end up working on the site should feel at home in, and I picked D8 to be a bit ahead of the curve and try using it for something practical (I know that goes a bit agaist point one, but that's mostly a matter of time). +1 for the CI instructions. And I'd also suggest some pages giving new users an overview of how to manage their drush commands with composer. |
|
Good comments! First, my replies to @weitzman:
If we do this, then we can be sure that all Drush extensions loaded via 'composer require' will be found in the "drush" folder of the Drupal root. This will greatly limit the number of places we need to look for commandfiles.
You are right, the goal here is to have Drush be loaded via composer as part of each Drupal project. So, you install Drush in your Drupal project via 'composer require drush/drush', and it ends up in vendor/drush/drush. Any composer libraries brought in by Drush's composer.json file will be part of the Drupal site's autoloader. The main reason for having a global Drush is so that you can use it for your old, pre-Composer Drupal sites. If you only have one Drupal 8 site on a machine, then just put vendor/drush/drush in your $PATH, and you're done. As Moshe said, we wouldn't have to allow global extensions to be loaded from a site-specific Drush; if we did, though, it could only be extensions that did not use Composer. You are absolutely right about the dependency problem. For Composer libraries such as Guzzle, that are used in Core, it's easy: everyone has to use the version that Drupal requires, or you'll get a Composer error. For minor revisions, contrib module writers will have a choice. If they require very specific dependencies, then they are likely to get a lot of Composer errors as Drupal upgrades. If they are lax, then things will "just work", until they don't, and then the user will get strange errors. I expect that everyone will eventually settle on being lax for contrib, and hopefully the library writers of the world will be very good about not breaking backwards compatibility, except with a major version upgrade. Then there will be no strange errors (crossing fingers). Major version upgrades are going to be a larger problem, especially for "discovered" libraries that are not used in Drupal core, but become very popular in contrib. If you have a bunch of modules using foo 4., and then foo 5. comes out and is adopted by some contrib modules, then Drupal users will not be able to use certain modules together. Composer will tell you when this happens, but most people won't have any recourse. If this problem is left unchecked, then a site could be blocked from applying a Drupal security update in a crisis. This would be bad. Regardless, though, Drupal is going down this road. Drush can either follow or not; if we don't, then we're essentially saying don't use composer libraries in your Drush extensions, because they won't work. That's the worse decision, in my mind. |
|
@greg-1-anderson But after some internal discussions here, I don't understand the original problem. You can have two autoloaders, and they can load the same lib. Bandaid and Drush both pull in YAML, and that works fine. You do have a problem if the first registered autoloader loads a version that user of the second doesn't work with, though. I might have been gripped by late night fatalism, but I don't think the only options are "use composer like this or not at all". |
|
Haven't had time to investigate here. I was having trouble with Drush IQ's use of Goutte conflicting with Drupal 8's use of Goutte; I need to go back and see what exactly are the limitations with version matching when two autoloaders pull in the same library. I think both were using the same major version of Goutte. I agree that ultimately, supporting global commandfiles in a non-crippled way is important. We just need to come up with the best rules and conventions for selecting library versions and including the autoloader. |
|
Stumpled upon this issue, as I want to rework my master drush command. I just read through the issue, but don't get my head around everything. So I try to summarize the uses cases we've got with Drush and composer. Currently I see the following:
Before I make any (new) suggestions, can you help me out, if there are more uses cases we want to cover? FYI: I just referenced this discussion in the meta page of our Drupal Composer group. |
|
I echo @darthsteven's excitement for managing Drush extensions with Composer, as it should greatly simplify custom Aegir installations. That said, I really like the idea of
Is that an approach the Drush maintainers would be open to? If so, I'd be happy to start working on a PR. |
|
@ergonlogic - I'm open to it, but can't guarantee a merge. Wrappers have to be thin and foolproof in order to be an improvement over |
|
Copying here a discussion that happened outside this issue (related specifically to the proposal to have a "global" Drush that finds and re-launches the site-specific Drush): There is another alternative for finding the right Drush that I don't think is discussed here. For folks who use the 'use' command, we could make a variant that alters the PATH such that the drush that goes with the site you selected via 'use' appears first. The issue here, of course, is that you can't set one of the shell's environment variables from a subprocess, so this would only work if 'use' was a shell alias that first called 'drush use', and then set the PATH if it was necessary to do so. This could work in conjunction with a global Drush, that would appear first in the PATH when not 'use'-ing a site, or when the 'use'd site is remote. Also, regarding the idea of one Drush calling some other Drush, this is essentially what happens already when you run 'drush @Remote cc all' -- except in this case, the local Drush uses ssh to call the remote Drush w/ a "cc all" command. In the composer case (per-site, composer-managed Drush, that is), we'd do the same thing, but we'd just call Drush directly. |
|
As mentioned in #1132, I think we should standardize on using "drupal-drush" as our composer type, rather than "drush-extension". |
|
I made some progress in #1138. With this PR, I also set the --local flag, which effectively "breaks" global Drush extensions, unless you copy them into each of your sites. However, we do not need to stop here; there are other things we can do to make global Drush extension management easier -- maybe even automatic. I can continue in the same PR, or commit this one and continue with separate PRs for each improvement. |
|
Okay, here is my new idea. This replaces most of what I suggested in 1) - 4) above, while maintaining the original vision (a) - (c).
I think this should make everyone happy. If you want to keep using your global Drush extensions the same way you always have, because the autoloader will "usually work" like this, then you can. If you run into a situation where the autoloader does NOT work like this, then you can just add your global extension into the composer.json of your Drupal site, run composer install, and either Composer will grab the right version for you, or you'll get a useful set of error messages about the specific version incompatibilities that you are experiencing. Should be just a little more code, and we'll be done. I think I can wrap it up early next week. |
|
I realized that (4) and (5) are a mistake. Rather than adding magic behavior to --local, we can just have folks add additional commandfile include locations in their site-local drushrc.php files. This does not work today, because Drush only checks for commandfile / config includes once, and it is too late to modify these by the time the site-local config is sourced. We could change this behavior, though; all we would need to do is remember which locations we already sourced, and test to see if any more were added every time we source another config file. |
|
Interesting side note. If you use the composer packagist repository http://drupal-packagist.webflo.io/, it automatically inserts dependencies from Drupal modules to Drupal core, and from Drush extensions to Drush. See: See: drupal-composer/drupal-parse-composer#3 Their model is, if you use composer to install a Drupal module, then it is also expected that you are using composer to install Drupal core, and these are part of the same composer project. This also extends to Drush extensions and Drush--if you install your Drush extension from http://drupal-packagist.webflo.io/, the assumption is that you are using a site-local Drush. Other packagist repositories do not have this assumption, but I thought it was interesting to mention this here. |
…ser libraries can interact with Drush via Drush\Drush::autoload().
…ser libraries can interact with Drush via Drush\Drush::autoload().
|
More discussion on local-vs-global commandfile management in #1340. |
|
I recently had to resolve a difficult problem I was having with site-local Drush and Drupal 8. When I used the d8 site with the global Drush, it worked fine, but the site-local Drush was crashing in the terminate() method. It turned out that the problem was a After this experience (which other people have also experienced), I now firmly and strongly believe that Drush extensions, Drupal modules, and any other manner of plugin should never include any autoload.php file. Only top-level components (the front controller, Drupal, Drush, Behat, etc.) should include the autoloader. Plugins should always live in the same autoloader as the app they plug in to. So, how should a Drush extension insure that its autoloader is included? The current answer is to call the function I am working on strategies for managing Drush extensions from Composer in #1347, but this effort is still in progress. |
|
I've been pondering this lately. I've done a few global Drush commands, and lately a few Symfony Console based ones. I've actually considered re-doing a few of my global Drush commands as stand-alone Symfony Console commands, as their reliance on Drush could be handled as calling Drush as an external command (Symfony Process FTW). There's something to be said about installing Drush in the site code (as anyone that's tried drush sql-sync from a local, global, Drush 7 to a remote Drush 6 will know). My viewpoint might be skewed, but from where I'm looking, global Drush commands is increasingly looking like a wart (a historically useful one, but none the less). I don't know what others have been using global Drush commands for, but in my own experience it's mostly been for commands that could just as well been standalone shell commands that called Drush for some "low-level" Drupal work. Back when Drush was still young, creating a command line PHP script was pretty involved, and Drush commands was a welcome shortcut. But with Symfony and Composer, it's become much easier. I suspect that if you take the total sum of Drush commands, subtract those in modules, and those which is global due to the convenience of not installing it into every site, you'll end up with a small set of global commands that could just as well be done as standalone shell commands. So I'll back up and suggest letting Drush do what it's named for, being a Drupal shell, not being a Drupal shell sometimes being a generic command runner. Drop global commands, and only have a global configuration file for configuration items that make global sense (overriding the ssh command line and that kind of stuff). It'll simplify the code, and some of Drush internals might be rolled out into separate packages that other commands might use (parsing alias files and ssh'ing to aliases to run command is a few I could find a use for). Someone might even create a composer template for shell commands relating to Drupal that'll include a lib for easy running of Drush in backend mode. It might seem a bit like a step backwards, but I think it'll focus Drush more on it's core value, and I don't think the loss of the global context is that great in the long run. |
|
I think that it's a good idea to factor some useful libraries out of Drush, for use in other stand-alone cli applications. |
|
I made an attempt at making Drush detect and report "dependency hell" situations in #1520. Moshe and I just discussed this, and the end conclusion we reached was that this is a hard problem, and Composer was not really designed to solve it. Moving forward, it looks like we are going to completely disable support for any sort of global Drush commandfile in Drush 8, maybe with some way to opt-in for this behavior, maybe not. This will formalize the recommendation above as actual Drush policy. Global commandfiles should not be Drush commands, but should instead be written with the Symfony Console component, or something similar, and 'exec' Drush when needed to operate on a Drupal site. |
|
So... this issue started off as a request to be able to 'drush dl' extensions directly from github. Now it appears to be deprecating global commandfiles entirely. Is this just limited to installation via Composer, as #1520 suggests? Maybe I'm mis-reading this issue (and #1513 (comment)), but wouldn't that make commands like FWIW, I'm fine with limiting automatic search paths, so long as this is still valid in |
|
We don't have a firm implementation in mind yet, although allowing opt-in of global locations, e.g. via You are correct that the general idea would be to deprecate the concept of providing commands such as The theory is:
Note that the Drupal Console project gets around this problem in a different way: it requires that your version of the Drupal Console must match exactly with the version of Drupal that you are running it with. If you have different versions of Drupal on the same system, then you'll need to install different versions of the Drupal Console in order to use them successfully with your different sites. Drush needs to work with multiple versions of Drupal, so this is not an option for us. If a command never needs to bootstrap or target a Drupal site, it is again our theory that such commands do not need to be Drush commands; they could be Symfony console commands. Drush make could be a Symfony console command, but it is in Drush core, so it does not need to be ported. |
|
This issue has been about several interrelated issues. I think we need to close it and discuss elsewhere.
|
|
See also #2103. |
Having written a few Drush commands, I've repeatedly run into the problem of how/where to host/release them.
Github being the first obvious choice, it's quick, you can run tests on Travis, and seems natural as Drush itself is here.
However, Drupal.org does have the very nice advantage of supporting
drush dl <drush_command>.Basically, what I'd really like would be being able to tell people to
drush dl <something>a Github hosted command.I guess that it's quite doable to make Drush understand
drush dl xendk/bandaid, and do some Github magic, or implement a drush extend xendk/bandaid, but what's the Drush core teams thoughts?The text was updated successfully, but these errors were encountered: