Warn user when programs are missing like mysql, mysqldump, rsync, sqlite3

[alberto56]

Background:

I have a Docker file which installs Drush (latest dev version), runs drush site-install, and then attempts to enable certain modules. At the last step, there is a "needs higher bootstrap" error.

You can see the resulting output at https://circleci.com/gh/alberto56/docker-drupal/1.

Note the following:

• At Step 34, Drupal is successfully installed.
• At Step 39, An attempt to use a drush command on the environment results in "needs higher bootstrap"; I am expecting the command to recognize the newly-created Drupal site and enable the modules

Steps to reproduce:

• Install Docker or use a CoreOS VM
• git clone --branch 8.0.x-dev-1.x https://github.com/alberto56/docker-drupal.git
• checkout commit 1fd4c3a12f7430066d171c48eecd72df1ceaa1ce
• docker build .
• docker run -d -v 80:80 [image hash]
• you can now visit your site on a browser.
• docker pml or docker en still results in "needs higher bootstrap"

This will result in the error.

Possibly related to: #1336

[alberto56]

A note on the above:

Drupal seems to use an absolute positioning for the database, so to see the site you need to change the permissions on the database. For example if your container is f25, you could type:

docker exec f25 bash -c 'chown -R www-data:www-data /db'
docker exec f25 bash -c 'chmod -R 777 /db'

Now, the site will be visible in a browser. However the following still will say "needs higher bootstrap", even though the site is fully live:

docker exec f25 bash -c 'cd /srv/drupal/www/ && drush pml'

[alberto56]

I ran the command with --debug, and the problem was that sqlite3 was not available.

This is not immediately obvious, especially taking into account the following factors:

• sqlite3 is not required with "drush site-install", and the site works fine when installed with drush, even if sqlite3 is not available.
• sqlite3 was not required for drush commands to run with Drupal 7 and/or previous versions of drush.

Proposed feature request:

Instead of failing with:

Drush was not able to start (bootstrap) the Drupal database.             [error]
Hint: This may occur when Drush is trying to:
 * bootstrap a site that has not been installed or does not have a
configured database. In this case you can select another site with a
working database setup by specifying the URI to use with the --uri
parameter on the command line. See `drush topic docs-aliases` for
details.
 * connect the database through a socket. The socket file may be
wrong or the php-cli may have no access to it in a jailed shell. See
http://drupal.org/node/1428638 for details.

The failure could be something like:

Drush was not able to start (bootstrap) the Drupal database, because
the database is a sqlite database and sqlite3 is not currently installed on
the system. Please install sqlite3 and try again.

[weitzman]

Similarly, I dont think we validate to make sure that other key programs are available like mysql, mysqldump, rsync, etc.

[jonhattan]

Reference: https://www.drupal.org/node/1820166

[jerrac]

I just spent a good hour or so figuring out that drush requires the mysql client due to the lack of information in that error message. I think this is the second time I've had to figure this out... So I'd love to see more informative error messages.

[weitzman]

The link @jonhattan posted is a good starting point. Other relevant examples that I have seen are composer diagnose and brew doctor

[skyred]

Yes, and the current error msg is misleading:

Command pm-enable needs a higher bootstrap level to run - you will need to invoke drush from a more functional Drupal     [error]
environment to run this command.
Drush was not able to start (bootstrap) the Drupal database.                                                              [error]
Hint: This may occur when Drush is trying to:
 * bootstrap a site that has not been installed or does not have a configured database. In this case you can select
another site with a working database setup by specifying the URI to use with the --uri parameter on the command line. See
`drush topic docs-aliases` for details.
 * connect the database through a socket. The socket file may be wrong or the php-cli may have no access to it in a jailed
shell. See http://drupal.org/node/1428638 for details.

[pantaoran]

I also ran into this problem. I have separate Docker containers for php and database servers, so the php container doesn't have mysql client installed, which makes Drush fail with the obscure error message.

[juliencarnot]

Wow, I'm so glad I finally found this issue! drush status was giving me the right info about the database, so I didn't notice that my php container was lacking mysql-client...

Running apt-get install mysql-client did fix the problem, but is it the best way to do this while preserving the benefits (safety, performance, etc.) of containerization?

[GroovyCarrot]

I've just had to spent some time debugging this issue also, in the context of using a mysql docker container. I'm not really understanding why this dependency is necessary? PHP can already communicate with the database, running in another container on the network just fine, without the need for mysql-client to be installed.

I had thought that perhaps it was checking for a socket, as the message suggests, however the directory is empty, and would be useless anyway:

ls -l /var/run/mysqld/
total 0

As the above comments, it would be preferable to not have to bake this package into the PHP container when it is seemingly unnecessary.

[GroovyCarrot]

After some more investigation, it seems that Drush isn't written to use native PHP mysql drivers at all, and calls the client directly.

class Sqlmysql extends SqlBase {
  public function command() {
    return 'mysql';
  }
...

Is used to construct a shell command:

  /**
   * Execute a SQL query.
   *
   * Note: This is an API function. Try to avoid using drush_get_option() and instead
   * pass params in. If you don't want to query results to print during --debug then
   * provide a $result_file whose value can be drush_bit_bucket().
   *
   * @param string $query
   *   The SQL to be executed. Should be NULL if $input_file is provided.
   * @param string $input_file
   *   A path to a file containing the SQL to be executed.
   * @param string $result_file
   *   A path to save query results to. Can be drush_bit_bucket() if desired.
   *
   * @return
   *   TRUE on success, FALSE on failure
   */
  public function query($query, $input_file = NULL, $result_file = '') {
    ...
    $parts = array(
      $this->command(),
      $this->creds(),
      $this->silent(), // This removes column header and various helpful things in mysql.
      drush_get_option('extra', $this->query_extra),
      $this->query_file,
      drush_escapeshellarg($input_file),
    );
    $exec = implode(' ', $parts);
    ...
    $success = drush_shell_exec($exec);
    ...
  }

The solution really is to just use PHP to connect to the database?

[ArtuGit]

#2410
php-xml extension is necessary, but there are not any warnings.

[mikeyp]

I mean I can understand this for the sql-dump sqlc and other commands, but why does drush cc all and drush cron need the mysql client?

[weitzman]

Thats part of Drush's progressive bootstrap. Been that way for many years. Its true that some commands dont need it but enough do thats its not that helpful to bootstrap differently for different types of commands.

[rickysarraf]

Thanks for adding me to this bug. I've been using the old drush (5.10.0) so far, up till now. Now that that version shows its age, is unmaintained, and out of Debian; I chose to follow the new trend of using software directly from the upstream projects. Because as I remember, the new drush was known to be hard to package for a distribution.

Not having a package (and pacakge metadata/relationship/dependency) is what led to not noticing the missing dependency. It'd be nicer to have such self reliant tools to have a better error message.

[sebas5384]

There's any progress on this issue? mysql client shouldn't be a dependency since php could handle all the mysql communication. Maybe we could leave this as an option.

I had the case which I can only use the RHEL7 Docker images, but the images of PHP lack on that dependency. https://access.redhat.com/containers/#/search/php

[weitzman]

@sebas5384 there are no plans to get rid of dependency on mysql. its helpful to avoid php in many cases, and essential for commands like sql-dump and sql-cli

[mikeyp]

I don't think anyone is saying that mysql client shouldn't be required for sql-dump and sql-cli. The problem is when it's required for commands like cache-rebuild. This is becoming more of an issue with technologies like Docker, Kubernetes, Openshift, Heroku, etc becoming more popular for Drupal hosting, since containers tend to be very compact and while they often have more than enough requirements to run Drupal, things like cache-rebuild or cron should be able to run as well.

[sebas5384]

@weitzman I think @mikeyp said it all.

I agree with you @weitzman, but only for commands like sql-cli or sql-dump, because there are other commands like updb or config-import those shouldn't needed the mysql-client as dependency.

[andypost]

but --report=diff needs difftools or sort of that

[agerard]

Having just gone through a frustrating hour with this uninformative error message to finally determine that I needed postgresql-client to let drush work with my external db; I understand it doesn't make sense to try to diagnose all missing dependencies, but adding the simple reminder to "try running drush with --debug to diagnose dependency problems" would go a long way.

[weitzman]

Just added a warning in drush master for this.

[weitzman]

Drush master now uses PHP to connect to the DB when bootstrapping (#3800). So now a lot of commands don't require mysql-client. The sql:* commands still do, as does site:install.