I can't flash my device when I use a Docker container (IDFGH-9230) #10617

TheElectronicMusicProject · 2023-01-25T21:55:01Z

Answers checklist.

I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v4.4

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

CMD

Development Kit.

ESP32-WROOM-32

Power Supply used.

USB

What is the expected behavior?

With a simple project, where I can build/flash/monitor my application, I want to test it on a Docker container. The device is detected on COM3.

I created a container that is working. The version in the image is v5.1-dev-3025-g49551cc48c.

I'm able to build it correctly, however when I try to flash it, I find some errors.
In particular, no COM serial port can be found (by default is on /dev/ttyUSB1 but it's not a valid COM port).

What is the actual behavior?

This is the error I obtain:

 *  Executing task: /opt/esp/python_env/idf5.1_py3.8_env/bin/python /opt/esp/idf/components/esptool_py/esptool/esptool.py -p /dev/ttyUSB1 -b 460800 --before default_reset --after hard_reset --chip esp32 write_flash --flash_mode dio --flash_freq 40m --flash_size 2MB 0x1000 bootloader/bootloader.bin 0x10000 esp1.bin 0x8000 partition_table/partition-table.bin 

esptool.py v4.5.dev2
Serial port /dev/ttyUSB1

A fatal error occurred: Could not open /dev/ttyUSB1, the port doesn't exist

I'm not able to set up Docker in order to make it see the COM ports available.

Steps to reproduce.

I attach my project folder:
esp1.zip

Debug Logs.

No response

More Information.

No response

The text was updated successfully, but these errors were encountered:

KaeLL · 2023-01-25T22:21:45Z

https://docs.docker.com/engine/reference/commandline/run/#-add-host-device-to-container---device

TheElectronicMusicProject · 2023-01-26T18:12:15Z

https://docs.docker.com/engine/reference/commandline/run/#-add-host-device-to-container---device

I added/changed few lines in devcontainer.json. I used the GUID class I found under "Device Manager" attachet to my COM3 port.

	"forwardPorts": [],
	"runArgs": [
		/*"--privileged",*/
		"--isolation=process",
		"--device=class/4d36e978-e325-11ce-bfc1-08002be10318"
	]

however when I click on "Reopen in Container", I obtain an error due to a Command failed.
Here the log:

[2023-01-26T18:07:46.914Z] Dev Containers 0.266.1 in VS Code 1.74.3 (97dec172d3256f8ca4bfb2143f3f76b503ca0534).
[2023-01-26T18:07:46.914Z] Start: Resolving Remote
[2023-01-26T18:07:46.927Z] Setting up container for folder or workspace: c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:46.932Z] Start: Check Docker is running
[2023-01-26T18:07:46.933Z] Start: Run: docker version --format {{.Server.APIVersion}}
[2023-01-26T18:07:47.289Z] Stop (356 ms): Run: docker version --format {{.Server.APIVersion}}
[2023-01-26T18:07:47.289Z] Server API version: 1.41
[2023-01-26T18:07:47.289Z] Stop (357 ms): Check Docker is running
[2023-01-26T18:07:47.290Z] Start: Run: docker volume ls -q
[2023-01-26T18:07:47.677Z] Stop (387 ms): Run: docker volume ls -q
[2023-01-26T18:07:47.756Z] Start: Run: docker ps -q -a --filter label=vsch.local.folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --filter label=vsch.quality=stable
[2023-01-26T18:07:48.415Z] Stop (659 ms): Run: docker ps -q -a --filter label=vsch.local.folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --filter label=vsch.quality=stable
[2023-01-26T18:07:48.416Z] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:49.080Z] Stop (664 ms): Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:49.081Z] Start: Run: C:\Users\User\AppData\Local\Programs\Microsoft VS Code\Code.exe --ms-enable-electron-run-as-node c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js up --user-data-folder c:\Users\User\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data --workspace-folder c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --workspace-mount-consistency cached --id-label devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --log-level debug --log-format json --config c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1\.devcontainer\devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2023-01-26T18:07:49.632Z] (node:4576) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[2023-01-26T18:07:49.632Z] (Use `Code --trace-deprecation ...` to show where the warning was created)
[2023-01-26T18:07:49.637Z] @devcontainers/cli 0.25.2. Node.js v16.14.2. win32 10.0.22000 x64.
[2023-01-26T18:07:49.637Z] Start: Run: docker buildx version
[2023-01-26T18:07:50.289Z] Stop (652 ms): Run: docker buildx version
[2023-01-26T18:07:50.290Z] github.com/docker/buildx v0.7.1 05846896d149da05f3d6fd1e7770da187b52a247
[2023-01-26T18:07:50.290Z] 
[2023-01-26T18:07:50.290Z] Start: Resolving Remote
[2023-01-26T18:07:50.295Z] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:50.767Z] Stop (472 ms): Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:50.773Z] Start: Run: docker inspect --type image espressif/idf
[2023-01-26T18:07:51.233Z] Stop (460 ms): Run: docker inspect --type image espressif/idf
[2023-01-26T18:07:51.237Z] local container features stored at: c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\node_modules\vscode-dev-containers\container-features
[2023-01-26T18:07:51.240Z] Start: Run: tar --no-same-owner -x -f -
[2023-01-26T18:07:51.314Z] Stop (74 ms): Run: tar --no-same-owner -x -f -
[2023-01-26T18:07:51.318Z] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f C:\Users\User\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1674756471234\Dockerfile-with-features -t vsc-esp1-fa484f7ddd0aaa6140db4250f0577e54 --target dev_containers_target_stage --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1\.devcontainer
[2023-01-26T18:07:51.550Z] 
[2023-01-26T18:07:52.083Z] 
[+] Building 0.0s (0/0)

[2023-01-26T18:07:52.116Z] 

[2023-01-26T18:07:52.350Z] 
[+] Building 0.0s (0/1)
 => [internal] load build definition from Dockerfile-with-features         0.0s
 => => transferring dockerfile: 1.69kB                                     0.0s
[2023-01-26T18:07:52.473Z] 
[+] Building 0.1s (2/3)
 => [internal] load build definition from Dockerfile-with-features         0.0s
 => => transferring dockerfile: 1.69kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/espressif/idf:latest            0.0s
[2023-01-26T18:07:52.536Z] 
[+] Building 0.2s (10/10) FINISHED
 => [internal] load build definition from Dockerfile-with-features         0.0s
 => => transferring dockerfile: 1.69kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/espressif/idf:latest            0.0s
 => [dev_container_auto_added_stage_label 1/5] FROM docker.io/espressif/i  0.0s
 => CACHED [dev_container_auto_added_stage_label 2/5] RUN apt-get update   0.0s
 => CACHED [dev_container_auto_added_stage_label 3/5] RUN update-alternat  0.0s
 => CACHED [dev_container_auto_added_stage_label 4/5] RUN wget --no-verbo  0.0s
 => CACHED [dev_container_auto_added_stage_label 5/5] RUN echo "source /o  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:34589692b191618aff0b6653e58be02b2aa986405c232  0.0s
 => => naming to docker.io/library/vsc-esp1-fa484f7ddd0aaa6140db4250f0577  0.0s
 => exporting cache                                                        0.0s
 => => preparing build cache for export                                    0.0s
[2023-01-26T18:07:52.689Z] Stop (1371 ms): Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f C:\Users\User\AppData\Local\Temp\devcontainercli\container-features\0.25.2-1674756471234\Dockerfile-with-features -t vsc-esp1-fa484f7ddd0aaa6140db4250f0577e54 --target dev_containers_target_stage --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1\.devcontainer
[2023-01-26T18:07:52.693Z] Start: Run: docker events --format {{json .}} --filter event=start
[2023-01-26T18:07:52.726Z] Start: Starting container
[2023-01-26T18:07:52.726Z] Start: Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount source=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1,target=/workspaces,type=bind --mount source=extensionCache,target=/root/.vscode-server/extensions,type=volume --mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --isolation=process --device=class/4d36e978-e325-11ce-bfc1-08002be10318 --entrypoint /bin/sh vsc-esp1-fa484f7ddd0aaa6140db4250f0577e54 -c echo Container started
[2023-01-26T18:07:53.007Z] 
[2023-01-26T18:07:53.311Z] docker: class/4d36e978-e325-11ce-bfc1-08002be10318 is not an absolute path.
See 'docker run --help'.
[2023-01-26T18:07:53.466Z] Stop (740 ms): Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount source=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1,target=/workspaces,type=bind --mount source=extensionCache,target=/root/.vscode-server/extensions,type=volume --mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --isolation=process --device=class/4d36e978-e325-11ce-bfc1-08002be10318 --entrypoint /bin/sh vsc-esp1-fa484f7ddd0aaa6140db4250f0577e54 -c echo Container started
[2023-01-26T18:07:53.467Z] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:53.476Z] Stop (783 ms): Run: docker events --format {{json .}} --filter event=start
[2023-01-26T18:07:53.886Z] Stop (419 ms): Run: docker ps -q -a --filter label=devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1
[2023-01-26T18:07:53.886Z] Error: Command failed: docker run --sig-proxy=false -a STDOUT -a STDERR --mount source=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1,target=/workspaces,type=bind --mount source=extensionCache,target=/root/.vscode-server/extensions,type=volume --mount type=volume,src=vscode,dst=/vscode -l devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --isolation=process --device=class/4d36e978-e325-11ce-bfc1-08002be10318 --entrypoint /bin/sh vsc-esp1-fa484f7ddd0aaa6140db4250f0577e54 -c echo Container started
[2023-01-26T18:07:53.887Z] trap "exit 0" 15
[2023-01-26T18:07:53.887Z] exec "$@"
[2023-01-26T18:07:53.887Z] while sleep 1 & wait $!; do :; done -
[2023-01-26T18:07:53.887Z]     at Aoe (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1893:1355)
[2023-01-26T18:07:53.887Z]     at uT (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1893:1291)
[2023-01-26T18:07:53.888Z]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[2023-01-26T18:07:53.888Z]     at async Poe (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1899:2128)
[2023-01-26T18:07:53.888Z]     at async Zf (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:1899:3278)
[2023-01-26T18:07:53.888Z]     at async aue (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:2020:15276)
[2023-01-26T18:07:53.888Z]     at async oue (c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js:2020:15030)
[2023-01-26T18:07:53.899Z] Stop (4818 ms): Run: C:\Users\User\AppData\Local\Programs\Microsoft VS Code\Code.exe --ms-enable-electron-run-as-node c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js up --user-data-folder c:\Users\User\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data --workspace-folder c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --workspace-mount-consistency cached --id-label devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --log-level debug --log-format json --config c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1\.devcontainer\devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2023-01-26T18:07:53.900Z] Exit code 1
[2023-01-26T18:07:53.906Z] Command failed: C:\Users\User\AppData\Local\Programs\Microsoft VS Code\Code.exe --ms-enable-electron-run-as-node c:\Users\User\.vscode\extensions\ms-vscode-remote.remote-containers-0.266.1\dist\spec-node\devContainersSpecCLI.js up --user-data-folder c:\Users\User\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-containers\data --workspace-folder c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --workspace-mount-consistency cached --id-label devcontainer.local_folder=c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1 --log-level debug --log-format json --config c:\Users\User\Documents\Espressif\Learn-ESP32\Environment_Setup\esp1\.devcontainer\devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root true
[2023-01-26T18:07:53.906Z] Exit code 1

KaeLL · 2023-01-26T19:08:59Z

I assume you tried COM3 before trying the GUID? Because on Linux, this is how I'd do it
$ docker run --rm -v $PWD:/project --device=/dev/ttyUSB0:/dev/ttyUSB0 -w /project espressif/idf:release-v4.4 idf.py flash

fhrbata · 2023-02-06T10:22:32Z

@TheElectronicMusicProject Hello, have you been able to figure this out or find some working solution? It seems that currently it's not possible to share windows devices with linux containers in windows docker. At least according to these docker/for-win#1018 and https://learn.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/hardware-devices-in-containers. Maybe usbipd-win can be used to workaround this somehow. Anyway I for sure may be missing something, but I'm not sure if we can do anything about this.

TheElectronicMusicProject · 2023-02-07T19:17:55Z

I haven't been able to communicate through a serial COM port by using Docker. Maybe using Linux could solve this problem but I won't try now.

What I think it is possible, but I haven't tried this method, is to compile (build) the code and catch the .bin files. Then pass the files to the Flash download tool installed on the Windows system and program the ESP32.

If there are other possibilities please let me know!

fhrbata · 2023-02-08T07:39:00Z

@TheElectronicMusicProject Hello, yes, using the container to build your app and then flashing directly from within window should IMHO work without problems, even though I understand it's not very convenient way. Unfortunately at this point I'm not aware about other possibility :(. Would it be ok with you if we close this as IIUC this is not currently possible to do from within a docker? Thank you

igrr · 2023-02-08T09:21:21Z

One other option to try might be to run the esp_rfc2217_server.py tool on Windows host side, then inside the Docker container specify the port as rfc2217://<ip>:<port>, and pass that TCP port into the container. This should allow you to use the "flash" and "monitor" features of ESP-IDF over an RFC2217 connection. More information can be found here: https://docs.espressif.com/projects/esptool/en/latest/esp32/esptool/remote-serial-ports.html.

(Esptool project releases contain a standalone esp_rfc2217_server.exe program which you can run on Windows without having to install Python.)

fhrbata · 2023-02-08T09:32:38Z

Ah, I was not aware about the esp_rfc2217_server.py. Thank you very much @igrr for jumping in and provide this suggestion!

TheElectronicMusicProject · 2023-02-09T20:19:43Z

Thank you @igrr for your suggestions.

One other option to try might be to run the esp_rfc2217_server.py tool on Windows host side, then inside the Docker container specify the port as rfc2217://<ip>:<port>, and pass that TCP port into the container.

Firstly, I copied in my project esp_rfc_2217_server.py and esptool.py.

I opened a terminal and I ran (my ESP32 was connected to COM3):
python esp_rfc2217_server.py -p 4000 COM3

Then, the terminal showed the following lines and seemed to work:

INFO:root:RFC 2217 TCP/IP to Serial redirector - type Ctrl-C / BREAK to quit
INFO:root:Serving serial port: COM3
INFO:root:TCP/IP port: 4000

I opened the Docker container and added a new port with port number 4000 and the local address became localhost:4000
So I ran (inside the container):
python esptool.py --port rfc2217://localhost:4000 flash_id

However this generated an error:

esptool.py v3.3.2-dev
Serial port rfc2217://localhost:4000
Traceback (most recent call last):
  File "/opt/esp/python_env/idf5.1_py3.8_env/lib/python3.8/site-packages/serial/rfc2217.py", line 416, in open
    self._socket = socket.create_connection(self.from_url(self.portstr), timeout=5)  # XXX good value?
  File "/usr/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "esptool.py", line 5399, in <module>
    _main()
  File "esptool.py", line 5392, in _main
    main()
  File "esptool.py", line 4699, in main
    esp = esp or get_default_connected_device(ser_list, port=args.port, connect_attempts=args.connect_attempts,
  File "esptool.py", line 110, in get_default_connected_device
    _esp = ESPLoader.detect_chip(each_port, initial_baud, before, trace,
  File "esptool.py", line 363, in detect_chip
    detect_port = ESPLoader(port, baud, trace_enabled=trace_enabled)
  File "esptool.py", line 320, in __init__
    self._port = serial.serial_for_url(port)
  File "/opt/esp/python_env/idf5.1_py3.8_env/lib/python3.8/site-packages/serial/__init__.py", line 90, in serial_for_url
    instance.open()
  File "/opt/esp/python_env/idf5.1_py3.8_env/lib/python3.8/site-packages/serial/rfc2217.py", line 420, in open
    raise SerialException("Could not open port {}: {}".format(self.portstr, msg))
serial.serialutil.SerialException: Could not open port rfc2217://localhost:4000: [Errno 99] Cannot assign requested address

Any idea how can be solved?

dobairoland · 2023-02-13T08:37:54Z

I opened the Docker container and added a new port with port number 4000 and the local address became localhost:4000

Are the ports the same on both the host and guest? Port 4000 inside the docker need to be connected to the same 4000 port on the host.

dobairoland · 2023-02-13T09:11:33Z

... and please try rfc2217://localhost:4000?ign_set_control including the "?ign_set_control" as shown in the documentation.

fhrbata · 2023-02-13T13:21:31Z

Hello @TheElectronicMusicProject ,
I think the problem might be related to your connection string rfc2217://localhost:4000 passed to/used in the container. The container is running in separate network namespace and there is IMHO nothing listening on the 4000 port. You can try to use the host IP directly or host.docker.internal as described here. I tested this and it works for me. On the host I'm running esp_rfc2217_server.py -v -p 4000 COM3 and the following commands are working. Note 192.168.33.242 is the host IP.

C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf esptool.py --port rfc2217://192.168.33.242:4000?ign_set_control flash_id
C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf esptool.py --port rfc2217://host.docker.internal:4000?ign_set_control flash_id
C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

So maybe this can help with your problem. Thank you

EDIT: I tested with current IDF master branch only

TheElectronicMusicProject · 2023-02-13T22:23:43Z

Hello,

I'm close to the solution, but I need a clarification.

@fhrbata you say that I should run one of the three bash commands, so I used the second one. In my case is:
docker run espressif/idf esptool.py --port rfc2217://host.docker.internal:4000?ign_set_control flash_id

From the host side I used: esp_rfc2217_server.py -v -p 4000 COM3

Then, from the terminal I see:

Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements are satisfied.
Requirement files:
 - /opt/esp/idf/tools/requirements/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.1_py3.8_env/bin/python
Added the following directories to PATH:
  /opt/esp/idf/components/esptool_py/esptool
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp-elf-gdb/12.1_20221002/xtensa-esp-elf-gdb/bin
  /opt/esp/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin
  /opt/esp/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-2022r1-11.2.0/xtensa-esp32s2-elf/bin
  /opt/esp/tools/xtensa-esp32s3-elf/esp-2022r1-11.2.0/xtensa-esp32s3-elf/bin
  /opt/esp/tools/riscv32-esp-elf/esp-2022r1-11.2.0/riscv32-esp-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
  /opt/esp/tools/cmake/3.24.0/bin
  /opt/esp/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin
  /opt/esp/python_env/idf5.1_py3.8_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

esptool.py v4.5.dev2
Serial port rfc2217://host.docker.internal:4000?ign_set_control
Connecting...
Device PID identification is only supported on COM and /dev/ serial ports.
......
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting...
Device PID identification is only supported on COM and /dev/ serial ports.
.
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: ec:94:cb:7b:2b:a4
Uploading stub...
Running stub...
Stub running...
Manufacturer: 20
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

However the ESP32 continues running the previous code, and not the one of this project (I observe this by looking at the terminal).

Also, why am I not able to run this command inside the container (from Dev Containers: Reopen in Container option)? It gives me errors that aren't present by running docker from the terminal as you've done.

fhrbata · 2023-02-14T07:18:57Z

Hello @TheElectronicMusicProject ,

However the ESP32 continues running the previous code, and not the one of this project (I observe this by looking at the terminal).

The flash_id is mostly a testing command, which reads the SPI flash manufacturer, it doesn't flash the device. Anyway from your output you are able to communicate with the device from within the docker image, which is a good sign.

To actually flash the device, you should run the idf.py flash command as shown in the example 3.
C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

With this you should be hopefully able to flash your app.

Also, why am I not able to run this command inside the container (from Dev Containers: Reopen in Container option)? It gives me errors that aren't present by running docker from the terminal as you've done.

IMHO there should be no difference between running the commands interactively(-it) or not. From the info it's hard to say, at least for me, where the problem could be.

EDIT: Maybe try to use the host IP instead of the host.docker.internal, but this is just a wild and most probably wrong, guess of mine.

HTH

TheElectronicMusicProject · 2023-02-15T18:03:46Z

Thank you for your patience @fhrbata .

I used docker run espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

But from the terminal I see:

Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements are satisfied.
Requirement files:
 - /opt/esp/idf/tools/requirements/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.1_py3.8_env/bin/python
Added the following directories to PATH:
  /opt/esp/idf/components/esptool_py/esptool
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp-elf-gdb/12.1_20221002/xtensa-esp-elf-gdb/bin
  /opt/esp/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin
  /opt/esp/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-2022r1-11.2.0/xtensa-esp32s2-elf/bin
  /opt/esp/tools/xtensa-esp32s3-elf/esp-2022r1-11.2.0/xtensa-esp32s3-elf/bin
  /opt/esp/tools/riscv32-esp-elf/esp-2022r1-11.2.0/riscv32-esp-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
  /opt/esp/tools/cmake/3.24.0/bin
  /opt/esp/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin
  /opt/esp/python_env/idf5.1_py3.8_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

Executing action: flash
CMakeLists.txt not found in project directory /

Any idea what's happening?

dobairoland · 2023-02-16T09:45:39Z

CMakeLists.txt not found in project directory /

Most likely you are not running a command from an ESP-IDF project directory.

fhrbata · 2023-02-16T10:14:07Z

@dobairoland @TheElectronicMusicProject
Hello, I think Roland is right. Please note that you should also use -v %cd%:/<project> -w /project options for the docker.
The hint is the CMakeLists.txt not found in project directory / message

-v %cd%:/<project> - this will bind current host dir %cd% in the container as /<project>. Please replace <project> with your project directory
-w /<project> - this will set your binded/mounted <project> dir as a working directory inside the container

So for <project> == hello_world and the proper docker options I got the following and the chip is flashed.

C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash
Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements are satisfied.
Requirement files:
 - /opt/esp/idf/tools/requirements/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.1_py3.8_env/bin/python
Added the following directories to PATH:
  /opt/esp/idf/components/esptool_py/esptool
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp-elf-gdb/12.1_20221002/xtensa-esp-elf-gdb/bin
  /opt/esp/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin
  /opt/esp/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-2022r1-11.2.0/xtensa-esp32s2-elf/bin
  /opt/esp/tools/xtensa-esp32s3-elf/esp-2022r1-11.2.0/xtensa-esp32s3-elf/bin
  /opt/esp/tools/riscv32-esp-elf/esp-2022r1-11.2.0/riscv32-esp-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
  /opt/esp/tools/cmake/3.24.0/bin
  /opt/esp/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin
  /opt/esp/python_env/idf5.1_py3.8_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

Executing action: flash
Running ninja in directory /hello_world/build
Executing "ninja flash"...
[1/5] cd /hello_world/build/esp-idf/esptool_py && /opt/esp/python_env/idf5.1_py3.8_env/bin/python /opt/esp/idf/components/partition_table/check_sizes.py --offset 0x8000 partition --type app /hello_world/build/partition_table/partition-table.bin /hello_world/build/hello_world.bin
hello_world.bin binary size 0x30ed0 bytes. Smallest app partition is 0x100000 bytes. 0xcf130 bytes (81%) free.
[2/5] Performing build step for 'bootloader'
[1/1] cd /hello_world/build/bootloader/esp-idf/esptool_py && /opt/esp/python_env/idf5.1_py3.8_env/bin/python /opt/esp/idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x0 /hello_world/build/bootloader/bootloader.bin
Bootloader binary size 0x5100 bytes. 0x2f00 bytes (37%) free.
[2/3] cd /opt/esp/idf/components/esptool_py && /opt/esp/tools/cmake/3.24.0/bin/cmake -D IDF_PATH=/opt/esp/idf -D "SERIAL_TOOL=/opt/esp/python_env/idf5.1_py3.8_env/bin/python;;/opt/esp/idf/components/esptool_py/esptool/esptool.py;--chip;esp32s3" -D "SERIAL_TOOL_ARGS=--before=default_reset;--after=hard_reset;write_flash;@flash_args" -D WORKING_DIRECTORY=/hello_world/build -P /opt/esp/idf/components/esptool_py/run_serial_tool.cmake
esptool esp32s3 -p rfc2217://host.docker.internal:4000?ign_set_control -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 80m --flash_size 2MB 0x0 bootloader/bootloader.bin 0x10000 hello_world.bin 0x8000 partition_table/partition-table.bin
esptool.py v4.5.dev2
Serial port rfc2217://host.docker.internal:4000?ign_set_control
Connecting...
Device PID identification is only supported on COM and /dev/ serial ports.
.
Chip is ESP32-S3 (revision v0.1)
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e8:5d:68
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00000000 to 0x00005fff...
Flash will be erased from 0x00010000 to 0x00040fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Compressed 20736 bytes to 13161...
Writing at 0x00000000... (100 %)
Wrote 20736 bytes (13161 compressed) at 0x00000000 in 1.1 seconds (effective 149.8 kbit/s)...
Hash of data verified.
Compressed 200400 bytes to 109258...
Writing at 0x00010000... (14 %)
Writing at 0x0001c99d... (28 %)
Writing at 0x000225b9... (42 %)
Writing at 0x00028aa4... (57 %)
Writing at 0x0002ef6c... (71 %)
Writing at 0x00036ea2... (85 %)
Writing at 0x0003ca31... (100 %)
Wrote 200400 bytes (109258 compressed) at 0x00010000 in 5.7 seconds (effective 281.5 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 103...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.8 seconds (effective 30.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
Done

Without the -v and -w options I got the same output as yours.

Thank you

TheElectronicMusicProject · 2023-02-16T11:24:12Z

Nice explanation @fhrbata !

I used:
docker run --rm -v %cd%:/esp1 -w /esp1 espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

But it gave me the following error on the ESP-IDF terminal:

docker: Error response from daemon: create %cd%: "%cd%" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'

So I switched to the Command Prompt and I used docker run --rm -v "%cd%":/esp1 -w /esp1 espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control build flash instad... And it worked!

Now, I didn't tried this inside the Docker container of VSCode, however I think it is sufficient to close this.

Thank you to all!

P.S. First of all I used fullclean because of an error during compiling.

fhrbata · 2023-02-16T11:53:11Z

@TheElectronicMusicProject Yeah, that confusion is on me. The %cd% is environment variable in cmd.exe shell. It might not be available elsewhere. You can always replace it with a full path to be sure it's working everywhere as expected. The example
C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v %cd%:/hello_world -w /hello_world espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

should be the same as this with spelling the host directory explicitly

C:\Users\FrantisekHrbata\work\hello_world>docker run --rm -v C:\Users\FrantisekHrbata\work\hello_world:/hello_world -w /hello_world espressif/idf idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

It's great to hear that you got it working!

fhrbata · 2023-02-17T09:50:58Z

@dobairoland Hi Roland, I believe, based on @TheElectronicMusicProject feedback, that this can be closed. Thank you all

On Windows/Mac the serial port cannot be access directly inside docker container. This is already mentioned in the documentation. This expands the documentation for steps which can be used to overcome this limitation by using remote serial port access via telnet protocol. Closes #10617 Signed-off-by: Frantisek Hrbata <[email protected]>

mvcorrea-ufc · 2023-12-26T03:58:25Z

This is cool, but how could run this commands from within a working container? like this gives me port closed/service not working!
The port is opened (can telnet:4000 from host)!

podman run -v $PWD:/project -p 4000:4000 -it --rm esp-idf
on host> python esp_rfc2217_server.py -v -p 4000 /dev/cu.usbmodemFA1301
on container> idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

fhrbata · 2023-12-28T11:36:12Z

Hello @mvcorrea-ufc ,

This is cool, but how could run this commands from within a working container? like this gives me port closed/service not working! The port is opened (can telnet:4000 from host)!

podman run -v $PWD:/project -p 4000:4000 -it --rm esp-idf

why is the container port 4000 published to host? It doesn't seem you are running anything on 4000 within the container. With this, podman(rootlessport) opens the port for forwarding between host and container. Meaning you can probably telnet to it from host, but I guess that the connection will be closed asap, because there is nothing listening on 4000 in the container.

on host> python esp_rfc2217_server.py -v -p 4000 /dev/cu.usbmodemFA1301

Don't you get Address already in use error, because the port is already open?

on container> idf.py --port rfc2217://host.docker.internal:4000?ign_set_control flash

I'm not sure I understand what you are trying to achieve, but using the -p 4000:4000 for the podman-run doesn't seem correct.

HTH

Thank you

TheElectronicMusicProject added the Type: Bug bugs in IDF label Jan 25, 2023

espressif-bot added the Status: Opened Issue is new label Jan 25, 2023

github-actions bot changed the title ~~I can't flash my device when I use a Docker container~~ I can't flash my device when I use a Docker container (IDFGH-9230) Jan 25, 2023

TheElectronicMusicProject closed this as completed Feb 16, 2023

espressif-bot added Resolution: NA Issue resolution is unavailable Status: Done Issue is done internally Resolution: Done Issue is done internally and removed Status: Reviewing Issue is being reviewed Resolution: NA Issue resolution is unavailable labels Feb 21, 2023

luarss mentioned this issue Aug 11, 2023

Flashing issues on docker using Mac M2 chip (IDFGH-10856) #12060

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I can't flash my device when I use a Docker container (IDFGH-9230) #10617

I can't flash my device when I use a Docker container (IDFGH-9230) #10617

TheElectronicMusicProject commented Jan 25, 2023

KaeLL commented Jan 25, 2023

TheElectronicMusicProject commented Jan 26, 2023

KaeLL commented Jan 26, 2023

fhrbata commented Feb 6, 2023

TheElectronicMusicProject commented Feb 7, 2023

fhrbata commented Feb 8, 2023

igrr commented Feb 8, 2023 •

edited

Loading

fhrbata commented Feb 8, 2023

TheElectronicMusicProject commented Feb 9, 2023

dobairoland commented Feb 13, 2023

dobairoland commented Feb 13, 2023

fhrbata commented Feb 13, 2023 •

edited

Loading

TheElectronicMusicProject commented Feb 13, 2023

fhrbata commented Feb 14, 2023 •

edited

Loading

TheElectronicMusicProject commented Feb 15, 2023

dobairoland commented Feb 16, 2023

fhrbata commented Feb 16, 2023

TheElectronicMusicProject commented Feb 16, 2023 •

edited

Loading

fhrbata commented Feb 16, 2023

fhrbata commented Feb 17, 2023

mvcorrea-ufc commented Dec 26, 2023 •

edited

Loading

fhrbata commented Dec 28, 2023 •

edited

Loading

I can't flash my device when I use a Docker container (IDFGH-9230) #10617

I can't flash my device when I use a Docker container (IDFGH-9230) #10617

Comments

TheElectronicMusicProject commented Jan 25, 2023

Answers checklist.

IDF version.

Operating System used.

How did you build your project?

If you are using Windows, please specify command line type.

Development Kit.

Power Supply used.

What is the expected behavior?

What is the actual behavior?

Steps to reproduce.

Debug Logs.

More Information.

KaeLL commented Jan 25, 2023

TheElectronicMusicProject commented Jan 26, 2023

KaeLL commented Jan 26, 2023

fhrbata commented Feb 6, 2023

TheElectronicMusicProject commented Feb 7, 2023

fhrbata commented Feb 8, 2023

igrr commented Feb 8, 2023 • edited Loading

fhrbata commented Feb 8, 2023

TheElectronicMusicProject commented Feb 9, 2023

dobairoland commented Feb 13, 2023

dobairoland commented Feb 13, 2023

fhrbata commented Feb 13, 2023 • edited Loading

TheElectronicMusicProject commented Feb 13, 2023

fhrbata commented Feb 14, 2023 • edited Loading

TheElectronicMusicProject commented Feb 15, 2023

dobairoland commented Feb 16, 2023

fhrbata commented Feb 16, 2023

TheElectronicMusicProject commented Feb 16, 2023 • edited Loading

fhrbata commented Feb 16, 2023

fhrbata commented Feb 17, 2023

mvcorrea-ufc commented Dec 26, 2023 • edited Loading

fhrbata commented Dec 28, 2023 • edited Loading

igrr commented Feb 8, 2023 •

edited

Loading

fhrbata commented Feb 13, 2023 •

edited

Loading

fhrbata commented Feb 14, 2023 •

edited

Loading

TheElectronicMusicProject commented Feb 16, 2023 •

edited

Loading

mvcorrea-ufc commented Dec 26, 2023 •

edited

Loading

fhrbata commented Dec 28, 2023 •

edited

Loading