Nitro extremely slow on macOS Big Sur

[jan-thoma]

Description

On macOS Big Sur the nitro containers are extemly slow. For example:

• Uncached Graphql Queries can take up to 20s in comparsion to a shared hosting where the same query take less than > 1s
• CP Call are slower than excpected
• Installing updates and plugins sometimes ends in a timeout

We use a external database container (MySQL) on a docker server on the same network, the connection from the dev machines are 1GB/s or even 10GB/s.

Running Craft instances based on the provided docker images with the same project and and the external database or internal server yields good performance.

Are the nitro containers somehow restricted in ressource usage or are there any limits which can be leveraged?

Steps to reproduce

1. Spin up nitro container
2. Watch it beeing slow

Additional info

• Nitro version: 2.0.8
• Docker version: 20.10.8, build 3967b7d
• OS: macOS Big Sur 11.5.2 (20G95)

[timkelty]

Running Craft instances based on the provided docker images with the same project and and the external database or internal server yields good performance.

@jan-thoma are you able to share details about this setup where you're seeing good performance? (e.g. is it docker-compose based, something else?)

Also if you're able to send over something reproducible (project config files, composer.json/lock, nitro config) that would help me do a better test.

[jan-thoma]

@timkelty

Here's an docker-compose file we use in production. We use traefik as reverse proxy for our containers. The Fronted is a Vite-SSR container which shouldn't matter much.

We use this setup primarily on a 16C/32T Machine with 64GB RAM and standard SATA-SSD, which runs fast as hell. Even on a Macbook Pro 2016 with 4C/8T and 16GB Ram it runs magnitudes faster than the nitro containers.

version: '3.2'
services:
    node:
        image: registry.io.t-k-f.ch/tkf/tkf-next-3-frontend:stable
        restart: always
        networks:
            - external
        depends_on:
            - craft
        labels:
            - "traefik.enable=true"
            - "traefik.docker.network=${TRAEFIK_NETWORK}"

            - "traefik.http.routers.${TRAEFIK_NAME_FRONTEND}.rule=Host(${TRAEFIK_HOST_FRONTEND})"
            - "traefik.http.routers.${TRAEFIK_NAME_FRONTEND}.entrypoints=websecure"
            - "traefik.http.routers.${TRAEFIK_NAME_FRONTEND}.tls.certresolver=letsencrypt-tls"
            - "traefik.http.routers.${TRAEFIK_NAME_FRONTEND}.middlewares=${TRAEFIK_NAME_FRONTEND}-www-to-non-www"

            - "traefik.http.middlewares.${TRAEFIK_NAME_FRONTEND}-www-to-non-www.redirectregex.regex=^https://www.(.*)"
            - "traefik.http.middlewares.${TRAEFIK_NAME_FRONTEND}-www-to-non-www.redirectregex.replacement=https://$${1}"
            - "traefik.http.middlewares.${TRAEFIK_NAME_FRONTEND}-www-to-non-www.redirectregex.permanent=true"

            - "traefik.http.services.${TRAEFIK_NAME_FRONTEND}.loadbalancer.server.port=${TRAEFIK_PORT_FRONTEND}"
        environment:
            - VITE_APP_TITLE=${VITE_APP_TITLE}
            - VITE_APP_GRAPHQL_API=${VITE_APP_GRAPHQL_API}
            - VITE_APP_CACHE_PAGES=${VITE_APP_CACHE_PAGES}
            - VITE_APP_CACHE_AGE=${VITE_APP_CACHE_AGE}
    craft:
        image: registry.io.t-k-f.ch/tkf/tkf-next-3-backend:stable
        restart: always
        networks:
            - external
            - internal
        volumes:
            - backend:/app/web/data
        depends_on:
            - mysql
        labels:
            - "traefik.enable=true"
            - "traefik.docker.network=${TRAEFIK_NETWORK}"

            - "traefik.http.routers.${TRAEFIK_NAME_BACKEND}.rule=Host(${TRAEFIK_HOST_BACKEND})"
            - "traefik.http.routers.${TRAEFIK_NAME_BACKEND}.entrypoints=websecure"
            - "traefik.http.routers.${TRAEFIK_NAME_BACKEND}.tls.certresolver=letsencrypt-tls"
            - "traefik.http.routers.${TRAEFIK_NAME_BACKEND}.middlewares=${TRAEFIK_NAME_BACKEND}-www-to-non-www"

            - "traefik.http.middlewares.${TRAEFIK_NAME_BACKEND}-www-to-non-www.redirectregex.regex=^https://www.(.*)"
            - "traefik.http.middlewares.${TRAEFIK_NAME_BACKEND}-www-to-non-www.redirectregex.replacement=https://$${1}"
            - "traefik.http.middlewares.${TRAEFIK_NAME_BACKEND}-www-to-non-www.redirectregex.permanent=true"

            - "traefik.http.services.${TRAEFIK_NAME_BACKEND}.loadbalancer.server.port=${TRAEFIK_PORT_BACKEND}"
        environment:
            - DB_DRIVER=${DB_DRIVER}
            - DB_SERVER=${DB_SERVER}
            - DB_PORT=${DB_PORT}
            - DB_DATABASE=${DB_DATABASE}
            - DB_USER=${DB_USER}
            - DB_PASSWORD=${DB_PASSWORD}
            - DB_SCHEMA=${DB_SCHEMA}
            - DB_TABLE_PREFIX=${DB_TABLE_PREFIX}
            - APP_ID=${CRAFT_APP_ID}
            - ENVIRONMENT=${CRAFT_ENVIRONMENT}
            - SECURITY_KEY=${CRAFT_SECURITY_KEY}
            - TIMEZONE=${CRAFT_TIMEZONE}
            - WEB=${CRAFT_WEB}
    mysql:
        image: mysql
        restart: always
        networks:
            - internal
        volumes:
            - database:/var/lib/mysql
        environment:
            - MYSQL_RANDOM_ROOT_PASSWORD=${DB_PASSWORD}
            - MYSQL_DATABASE=${DB_DATABASE}
            - MYSQL_USER=${DB_USER}
            - MYSQL_PASSWORD=${DB_PASSWORD}
networks:
    external:
        external: true
    internal:
        driver: bridge
volumes:
    database:
    backend:

Here's the corresponding composer.json

{
  "require": {
    "craftcms/cms": "3.7.10",
    "vlucas/phpdotenv": "^3.4.0"
  },
  "require-dev": {
    "yiisoft/yii2-shell": "^2.0.3"
  },
  "autoload": {
    "psr-4": {
      "modules\\": "modules/"
    }
  },
  "config": {
    "sort-packages": true,
    "optimize-autoloader": true,
    "platform": {
      "php": "7.2.5"
    }
  },
  "scripts": {
    "post-root-package-install": [
      "@php -r \"file_exists('.env') || copy('.env.example', '.env');\""
    ]
  }
}

The Docker file which build the craft container

FROM composer:2 as vendor
COPY composer.json composer.json
COPY composer.lock composer.lock
RUN composer install --ignore-platform-reqs --no-interaction --prefer-dist

FROM craftcms/nginx:7.4

WORKDIR /app

COPY --chown=www-data:www-data --from=vendor /app/vendor/ /app/vendor/
COPY --chown=www-data:www-data . .

RUN chmod +x bootstrap.sh
RUN mkdir storage

ENTRYPOINT ["./bootstrap.sh"]

And the bootstrap file to apply pending changes

#!/bin/sh

./craft migrate/all
./craft project-config/apply
./craft clear-caches/cp-resources

/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisor.conf

[timkelty]

@jan-thoma perfect, thanks! Will do some testing and get back to you.

In the meantime, can you try adding a bind mount to your craft service, and see if you see the same slowdown?
If you do, that means the bottleneck is Docker Desktop's Hypervisor virtualization (which is what I suspect).

        volumes:
            - backend:/app/web/data
            - .:/app
            

Can you also check Docker Desktop settings, and see what the value is for this:

[jan-thoma]

Docker Desktop settings are:

[jasonmccallister]

@jan-thoma one of the major impacts for local development is mounting a local filesystem (bind mounts) to a container, especially on Docker for Mac as macOS does not have native container support in the OS. Docker for Mac creates a virtual machine and installs Docker tools inside the VM, when you mount a filesystem/directory it mounts into the VM.

Also, if you are having issues with MySQL on macOS, that image is notorious for having issues with the default resources assigned to Docker Desktop. You can change those settings under Docker > Preferences > Resources.

What do you currently have assigned for the CPU and Memory on Docker Desktop?

[jan-thoma]

@jasonmccallister The MySQL Server on runs on a Linux machine and not on the developers macOS machines. I would rule this out for beeing the culprit. Docker Desktop has assigned the following Ressources

CPU: 8
Memory: 8GB

[jasonmccallister]

@jan-thoma just to be very clear, when you say:

On macOS Big Sur the nitro containers are extemly slow

You are referring to the database containers (mysql, postgres) running on Nitro or the site containers (that actually run the Craft application)?

[jan-thoma]

@jasonmccallister

Craft Containers are running locally on macOs
MySQL Containers are running on an external server Ubuntu 20.04 in the same LAN

[jan-thoma]

@timkelty

The bind mount slows down the container, significantly. The docker-compose setup still seems to approx 4x times faster in subsequent queries.

docker-compose setup:

nitro setup:

(edit) The Craft Container was accessed without a reverse proxy

[jan-thoma]

Maybe this project is worth a look. It syncs the data bidirectional between native volumes and the local filesystem instead of using bind mounts

https://mutagen.io/

I just did a little test and it works well so far

[jan-thoma]

An other option which could increase performance could be to exclude the runtime and maybe other folders from the bindmount

[jasonmccallister]

Maybe this project is worth a look. It syncs the data bidirectional between native volumes and the local filesystem instead of using bind mounts

We went down this road after Nitro v1 and considered it for Nitro v2. However, after working out the nit-picky details of including and orchestrating another tool that need to be installed to use Nitro we decided against it. There was a point the Docker team was looking at including mutagen (not sure of a reference) with Docker, but for some reason they went with the gRPC fuse route.

An other option which could increase performance could be to exclude the runtime and maybe other folders from the bindmount

We also went down this route and opted against it. Nitro can be used to develop other frameworks like Laravel, Symfony and etc. Some people also drastically modify their project directory structure and we would not want to force anyone into a specific structure. We have discussed making that interactive during the nitro add command but have not made much progress.

Another problem with excluding specific directories in the bind mount is we would need to bind every file and directory individually which would cause problems when someone adds a new directory or file in a project and its not mapped into the container. As far as I know, there is not a .dockerignore for bind mounts... Although that would be helpful!

[jasonmccallister]

Also, is this impacting everyone using Nitro or only a few developer machines? For reference, I am running Nitro with the following resources and not seeing (and have not run into it yet) performance issues like you mention (granted I do not have a lot of data in this site but the database).

What do you have under file sharing? I only have private and my user folder to reduce the directories that are shared to Docker.

[jan-thoma]

We have two developer machines which are affected booth Macbook Pros, but i can't speak for everybody.

I removed most of the folders the after your suggestion, but i need to check when i'm back in the office on monday. Accessing the database over a VPN Connection doesn't make the performance better...

Another problem with excluding specific directories in the bind mount is we would need to bind every file and directory individually which would cause problems when someone adds a new directory or file in a project and its not mapped into the container. As far as I know, there is not a .dockerignore for bind mounts... Although that would be helpful!

You can use Anonymous volumes to point to a subdirectory of a bind mount to exclude it, but i never used it to be honest.

[jan-thoma]

@jasonmccallister i can't see a differnce in performance when restrictig the file sharing ressources

[diluno]

I also have extremely bad performance, makes it almost unusable. Saving in the control panel can take up to a minute.

[Kethatril]

There are issues with docker for mac that occur after the computer wakes up from sleep. It majorly affects performance when xdebug is enabled, but still affects performance without it, just to a lesser extent.
More details: docker/for-mac#3455

[heymarkreeves]

I'm finding it increasingly difficult to endorse anything Docker-based for mac-based web developers.

I'd used Lando earlier this year after a new version of MAMP broke things again. That was a bit finicky to maintain. I tried Nitro next and it was great! Zero config and it was fast. Then I went away for a few months and worked with some in-house Docker setups that worked well for the most part, but required ongoing support for some folks. Now I'm here, my OS is updated to Big Sur, Nitro's updated to the latest, and I've got roaring fans and 10-30 second waits to load a small site's control panel screen on a 2-year old mac mini with a 3 GHz 6-Core Intel Core i5, 32GB RAM, and very little else running.

I just can't push the overhead of maintaining this or time spent waiting for things to load onto my clients.

I think from here on out, I'm just going to manage PHP with Homebrew and use local MySQL. Docker has its place for large, complex projects and teams, but I don't think that's what Nitro is targeting.

[jasonmccallister]

I completely understand the frustration and get where you are coming from, Docker for Mac has been notorious for slow load times and we are actively working on ways to improve that. While this is a fix that needs to happen in Docker Desktop, we are trying to address some load times with this PR which adds the ability to exclude directories on a site specific config... The larger the number of files the slower Docker Desktop becomes.

Excluding directories like vendor, node_modules, and even storage has giving us some pretty consistent improvements to load times with Craft and Nitro.. the PR has a few screenshots with a minimal Nitro config but excluding those specific directories and the load times are consistently around the 200ms range. This is a change that will come with 3.0 but a majority of the team has been using the pre-alpha build internally with really good results.

Hopefully this type of change will help improve the development experience when using Nitro, I know its been a pain for Docker on Mac and there are many "hacks" to try and improve performance but this is one I think most people can benefit from when using Nitro and Craft.

[heymarkreeves]

@jasonmccallister Thanks for the update and insights! I'll look forward to those changes.

[xavianaxw]

@jasonmccallister is it advised to try out v3 of nitro now or it's still too early? My nitro is really buckling with Big Sur now and it's really frustrating :(

[jasonmccallister]

@xavianaxw I would not recommend it yet, although there are build instructions to get it working on the 3.0 branch. We have some breaking changes we need to get in before it can become a beta.

We have a milestone where we keep track of whats left here: https://github.com/craftcms/nitro/milestone/6

While it says 3 percent complete, almost all of those are done but will not be closed out until we cut a beta and the community is using those features. That is our definition of "done".

Also, the Docker Desktop team has a roadmap and they are actively looking at making the filesync more performant on macOS so we should see some benefits from that soon (crossing my fingers).

[chrisrowe]

Are there any expected timelines for cutting a beta release? I'm struggling to get work done currently and weighing up moving back to Valet or building my own v3

[timkelty]

@chrisrowe Craft 4 is taking precedence right now, so we haven't committed to a release date yet.
Thank you for your patience!

[S-n-d]

While I greatly appreciate all the work you guys have done with Nitro, the dev experience on macOS (M1 pro + Monterey, in my case) is really off-putting. Control panel load times are atrocious. Sometimes I feel like I'm spending more time waiting than actually developing 😜 I'm really hopeful for v3 but let's face it... Docker on macOS won't be fast anytime soon. I used to work with MAMP and loved the transition to Nitro regarding workflow, but damn this slowness is frustrating. It even makes me rethink my choice of developing on a Mac, and linux is becoming more and more attractive.

Also, this: https://pen.so/2021/09/02/docker-on-macos/

[heymarkreeves]

@S-n-d Don't give up on the macOS developer experience! I've found Laravel Valet to be a delight. And you can couple it with phpmon for a GUI experience.

[chrisrowe]

Yep, I've gone back to Valet for speed. With phpmon and dbngin it's not a bad setup. I'd still ultimately love to use Nitro but for now, for me at least this is way faster.

[jasonmccallister]

Thank you for the feedback @S-n-d! There is always going to be some tradeoffs when syncing files (homestead, docker, end etc.) but I'm using the M1 and not seeing load times on the control panel (even for large sites) that are far off from having a VPS. Although the load times are better in v3 because we added a new feature that Docker Compose does not have, the ability to exclude directories during when mounting an app.

The really frustrating part is Docker and its file performance. As an example, the steps we recommend to improve performance for one user we have to tell another user to do the opposite (gRPC fuse specifically). How does that make sense? In an ideal world, Docker would fix the syncing issues and we could all just move on. However, Nitro is made as an alternative for local development, I would want it to work best for everyone but not at a sacrifice of developer productivity.

I understand why some have moved back to Valet, and we can (probably) never be as fast as running on bare metal (think VPS versus a bare metal server - one wins but it has its trade offs over the other). The problem with tools like Valet is they don't account for Windows or Linux users... In my eyes, Linux users have always been left in the dark for local development tools requiring the user to completely understand Linux, Containers, VM's just to get started and that is not always the case.

@S-n-d if you don't mind sharing your Docker Desktop settings we can try to troubleshoot, might be easier to send an email to [email protected] with your details and mention me.

Note: I have also been frustrated with macOS and switch between my MacBook Pro and a desktop PC running Zorin OS. I always end up back on macOS because there are some features (hardware) that you can't get on PC or Linux :)

[S-n-d]

Thanks @heymarkreeves and @chrisrowe for the tips! I've just tried Valet in conjunction with phpmon and dbngin. Like you guys said this is a VERY speedy setup and a delight to use. I'm going to stick with this setup. My M1 finally feels like an M1 😅

@jasonmccallister I would like to thank you for taking the time to craft (hehe) such an extended response 🙏 I gave Docker more than enough resources (6 cpu's, 8GB memory) but it still didn't perform well. CP load times were lackluster, like 10+ sec to load a simple entries list. For comparison on Valet this happens near instant (< 1sec). I'm quite sad to have to part from Nitro, I really like the workflow and in general the idea behind it, but alas in reality performance matters. I will try to follow the project though, hoping one day it can approach bare metal 😉 I do have a desktop PC running Arch, might try Nitro on that when I find the time, but i'm quite reluctant to use that for development. I really need my macbook for mobility.

[brettburwell]

For anyone following along with this thread, it sounds like there's good news on the horizon from the Docker side of things.

As @jasonmccallister mentioned, and from what little I admittedly understand about how Nitro and Docker work under the hood, it sounds like the major bottleneck is on the file syncing front. There's an open Github issue on the Docker repo that's been active for almost 2 years (yikes!), and within the past few weeks there's been some notable developments.

If you're curious you can find benchmarks and details towards the bottom of the thread, but the TL;DR is there's an experimental build with a new approach to file syncing (that's tied to macOS 12.3) that's bringing pretty dramatic speed improvements.

Fingers crossed we'll see a major performance boost sooner rather than later.

[jan-thoma]

Just tested the new experimental build on Macbook Pro M1 Max (12.2.1)

With Virtualization Framework and VirtioFS enabled in the Experimental Features section. Seeing some dramatic improvements in speed. GraphQl that took above 6 seconds in the old setup no execute under 1s if not cached.

The VirtioFS is greyed out on 12.2.1 but can be activated as mentioned here docker/roadmap#7 (comment)

[mediabeastnz]

Just tested the new experimental build on Macbook Pro M1 Max (12.2.1)

With Virtualization Framework and VirtioFS enabled in the Experimental Features section. Seeing some dramatic improvements in speed. GraphQl that took above 6 seconds in the old setup no execute under 1s if not cached.

The VirtioFS is greyed out on 12.2.1 but can be activated as mentioned here docker/roadmap#7 (comment)

This also helped with improving speed quite dramatically. I'm using a Macbook Pro M1 Pro (16gb) and without enable this setting it was almost unusable.

[mijewe]

My experience with enabling virtiofs has not been so positive - my containers will now hang several times a day, with the only resolution being to restart Docker.

Attempting a nitro restart (as opposed to restarting Docker) eventually yields the error message Error: unable to restart container x: Error response from daemon: Cannot restart container x: tried to kill container, but did not receive an exit event

I'm sure I saw this commented elsewhere but I'm struggling to find it

[jasonmccallister]

Hi, we are closing this issue as we have decided to retire Nitro, so no additional work will occur on this project. You can read the official blog post here https://craftcms.com/blog/retiring-craft-nitro. We appreciate everyones feedback and involvement and we look forward to refocusing our efforts on Cloud!

If you're looking for a new local development environment, we recommend DDEV and have a knowledge base article to help you with the transition: https://craftcms.com/knowledge-base/migrating-from-craft-nitro-to-ddev.