Feature request: Add docker exec to aliases remote

[iBobik]

Currently it is able to execute remote drush by alias like this:

$aliases['stage'] = array(
    'remote-host' => 'mystagingserver.myisp.com',
    'remote-user' => 'publisher',
);

But today we run Drupal also in Docker container, so it is good to run Drush commands also inside it. It could be configured like this:

$aliases['stage'] = array(
    'docker-machine' => 'aws01',
    'docker-container' => 'drupal_web_1',
    'docker-user' => 'www-data',
);

And also conbined to connect to server and there into container:

$aliases['stage'] = array(
    'remote-host' => 'mystagingserver.myisp.com',
    'remote-user' => 'docker',
    'docker-container' => 'drupal_web_1',
    'docker-user' => 'www-data',
);

So we will be able to use it intuitively like this:

drush sql-sync @site.local @site.stage

Workaround (how we have to to it today to sync into container):

drush @site.dev sql-dump | docker exec -i —user=www-data drupal_web_1 drush sql-cli

[weitzman]

you can connect via ssh-options. see http://wadmiraal.net/lore/2015/03/27/use-docker-to-kickstart-your-drupal-development/ and https://www.chapterthree.com/blog/drupal-development-docker-compose

[iBobik]

Running SSH server inside Docker container for using it’s shell is
antipattern and has many other disadvantages.

Jan Pobořil

2016-01-13 18:26 GMT+01:00 Moshe Weitzman [email protected]:

Closed #1910 #1910.

—
Reply to this email directly or view it on GitHub
#1910 (comment).

[weitzman]

If you don't want to run SSH, then I suggest you put your alias file into the container and run docker exec drush @foo sql-*

[iBobik]

Strongest use case for this feature request is sql-sync between containers.
There is no easy way.

Jan Pobořil

2016-01-13 23:53 GMT+01:00 Moshe Weitzman [email protected]:

If you don't want to run SSH, then I suggest you put your alias file into
the container and run docker exec drush @foo sql-*

—
Reply to this email directly or view it on GitHub
#1910 (comment).

[weitzman]

Yeah, its kind of hard to transfer data between environments that are deliberately so isolated from one another. Note that sql-sync has options available that you can use to script partial operations. See --no-dump --no-sync. With those, and drush_invoke_process() you can potentially compose a new command.

[iBobik]

Btw, it should be possible to do something like:

docker exec -i source_container drush sql-dump | docker exec -i destination_container drush sql-cli

But it will be much harder to do it if containers are on different machines.
Or if I want to gzip transfer.
Or if remote machine is available only by SSH (not Docker Machine).
Of want to show progress (as drush pipe can).
…

[frederickjh]

Is this something like a Bastion server with docker being run inside a remote host accessed via ssh? Drush has commands for that, if that is indeed the case. Remote Operations on Drupal Sites via a Bastion Server

[weitzman]

sql-sync across containers is an interesting challenge. would love to see some discussion and/or code.

[beeradb]

This feels like a really easy thing to add which would reduce pain for people trying to use drush in docker based setups.

We already have site aliases, so why not allow them to work with Docker, which is increasingly the direction infrastructure is moving?

[weitzman]

What exactly feels easy? I'm not sure what precisely is being requested. Is sql-sync across containers really the pain point?

[beeradb]

I apologize, Moshe. I should not have used the word easy, it was a poor word choice that likely belittles the work that would actually be involved.

With that said, here is the use case I'm trying to solve. I've got a docker-compose based local development environment with (typically) two containers, nginx/php-fpm, and mysql. Because of the way docker-compose networking works, the DB URL/port is different from within the docker container and the host machine (db:3306 vs 127.0.0.1:32801, for example).

If these were just local dev containers, I might just bite the bullet and break best practices by installing a SSH daemon within our containers. I wouldn't love it, but I'd probably live with it. In this case though, the nginx container being used is the same container as prod, so that's a non-starter. My best workaround right now is to check $_SERVER['argv'] and include a conditional drush.settings.php file which overrides the $databases variable (yuck) when drush is invoked. I could also (and do) include drush in the container, but the reality is that eventually, people are going to find something that causes them to want to change version of drush or upgrade faster than I can keep up, so at least being able to support native usage of drush would be a big win. I'd much prefer to do this with aliases over my fix.

I realize most of what is being asked here is working around docker idiosyncracies, but at the same time, drupal sites are increasingly being ran in containers for both production and dev, and it feels like allowing that workflow to be a first class citizen for aliases would be a nice win.

[beeradb]

Also, fwiw, even though I found this issue first, it looks to me like #2367 is a better solution to this. Since it could support arbitrary commands such as docker exec, kubectl exec, rkt enter, etc.

[mc0e]

In the case I'm currently looking at, drupal is running under docker, which is managed by kubernetes. Ssh is nowhere in the command I want to run to fire drush remotely. I don't know which host(s) drupal is running on, and that will vary dynamically.

Still, I can provide a command prefix that can be put before the drush command, and the remote drush will get run, with all the usual I/O streams hooked up. The parts of remote drush functionality I want most would be covered by being able to specify such a command prefix in a drush @alias. Being able to specify a wrapper script could probably do the same thing.

$aliases['stage'] = array( 
    'remote-command' => 'KUBECONFIG=/my/project/.kube/config kubectl exec -it mypod mycontainer',
);

The value of 'remote-command' can be prepended to 'drush [drush args]' and most stuff will work fine. If support for setting environment variables was not there, I could use a wrapper script to get that, and call it in the same manner.

I understand that there are some file copying operations that might not work this way. That's not an issue for me.

I really want to be able to specify my manner if connection from within the alias, so I can have a heterogeneous collection of drupal instances accessed different ways, and still be able to do things like loop over aliases, connect to each drupal instance, and return info on outstanding security issues.

[mortenson]

I know this doesn't fix the aliasing problems, but if you're just tired of running docker-compose exec SERVICE drush ... every day, this could help: https://gist.github.com/mortenson/48d16568147a710bedf94506f837733a

[weitzman]

Greg sketched out custom transports at #3017 (comment)

[weitzman]

I rather like the tiny shell script from @mortenson. Its not a replacement for transports, but it can be extended easily.

[mc0e]

As @mortenson says, it doesn't solve the alias problem.

But lets get creative... maybe instead of putting an alternative drush in the path, we could put an alternative ssh in the path, which reads the drushrc file(s) and makes use of a 'transport' parameter if it's in the alias, and does the right thing.

Or instead of parsing the drushrc files, the ssh replacement/wrapper could recognise an extra option that we could pass via drush's ssh-options facility, and use that to substitute a different transport.

hmm, actually that sounds like it might be what @greg-1-anderson was talking about with Drupal Console in #3017 (comment)

[weitzman]

We added a layer called 'Transports' No support for Docker yet but its getting closer if anyone is motivated.

[greg-1-anderson]

We'd need a docker transport to supplement the existing ssh transport; that would handle docker containers on the local machine. Then we'd also need another transport that would use the ssh transport and the docker transport to make an ssh+docker transport.

[ElijahLynn]

Is this something like a Bastion server with docker being run inside a remote host accessed via ssh? Drush has commands for that, if that is indeed the case. Remote Operations on Drupal Sites via a Bastion Server

@frederickjh This is good for a bastion, but it doesn't yet have instructions for a docker container running inside the destination host (after the bastion). From what I am reading, there isn't a solution to sync from a remote that is a container inside that host (even a workaround). Is that correct?

My use case is trying to get a lando drush sql-sync @stg @self working, with the remote being a container inside an ec2 instance (using AWS ECS).

[ElijahLynn]

We'd need a docker transport to supplement the existing ssh transport; that would handle docker containers on the local machine. Then we'd also need another transport that would use the ssh transport and the docker transport to make an ssh+docker transport.

I'd be interested in working on this if you would want to mentor me in a video conference (which we could record and post back here) for 30-60 minutes giving me some direction and more understanding on what needs to be added to the Transport layer. I have a pressing need for this right now, and thus some motivation.

[greg-1-anderson]

The most important part of this is deciding how to extend transports. At the moment, the implementation is hardcoded to only allow local and ssh transports -- that is, if the alias is not local, then the ssh transport is hardcoded.

See:

https://github.com/consolidation/site-process/blob/master/src/SiteProcess.php#L97

The next level of goodness here would be to add a couple more transports (as suggested above), and use them if certain attributes exist in the alias. We'll want to consider this implementation carefully to ensure that we don't build anything too limiting. I've seen your name on a number of related ssh issues, so I think you'd be a good person to advise here. I have not designed a solution yet.

A transport is just a class that wraps a commandline in some other commandline, e.g. making ls into ssh <user>@<host> ls, handling escaping et. al.

To implement a transport, just look at the existing one:

https://github.com/consolidation/site-process/blob/master/src/Transport/SshTransport.php

Not sure when I could hack on this, but please give it a try and ping me in slack if you get stuck. We can jump on a quick ad-hoc call if I have time.

[ElijahLynn]

Thanks @greg-1-anderson, my immediate tasks have changed a bit to get something working for now, I'll reach out to you on Slack when I get to this.

[sjerdo]

Hi @greg-1-anderson, are there any plans to rewrite RsyncCommands to use the TransportInterface? Currently the remote shell (-e / --rsh) option is hardcoded set to ssh and the ssh options.

[greg-1-anderson]

TransportInterface is for building commandlines for starting processes. ssh is one way to start a process.

Rsync might also use ssh when copying files; however, the way that works is unrelated to TransportInterface. It's not a bad idea to allow rsync to use other options besides just ssh; however, that would be a separate effort.

[weitzman]

I will soon have a for for this issue.

[weitzman]

Lets continue in #3835