More user friendly flashing if the build results need to be moved to another location (IDFGH-9856)

[amrsoll]

Is your feature request related to a problem?

I build the project from a different machine than the one I am flashing the device(s) from.

This message shows up:

Executing action: flash
Build directory '/home/flash_user/my_project/build' configured for project '/home/build_user/my_project' not '/home/flash_user/my_project'. Run 'idf.py fullclean' to start again. 

I could use esptool as the output of the build suggest, but all the options will change from one project to another and I would need to create extra logic to parse the files or create the esptool command on the fly after each build.

I would prefer to use idf.py flash as it figures out what and where to flash the binaries from the current working directory.

Describe the solution you'd like.

idf.py flash --skip-build would ignore the build step and use the files in work_dir/build/ to figure out what to flash.

idf.py build flash --skip-build has conflicting options. I suggest the build command takes precedence over --skip-build. This allows the build step to ignore the --skip-build flag.

Describe alternatives you've considered.

Alternatively my problem could be resolved with:

Flash over SSH

idf.py flash being able to run a remote flash job (say through ssh) using local build files

Example: idf.py flash -p flash_user@address:/dev/ttyUSB1

usbipd

usbipd could be used to create a virtual usb forward to the build machine. Perhaps there could be some documentation on how to use it, or a built-in functionality that could be used transparently using the syntax described in "Flash over SSH"

Note The alternatives "Flash over SSH" and "usbipd" do not solve delayed flashing after build. For example, when there is a queue of tests to run on different firmwares using the same board. The flashing step should be possible to separate completely from the build step for that to work.

Relative paths

the project configuration would use relative paths instead of absolute paths. It would make any project much more portable. This way you can sync all the build files which would skip/shorten the build step on underpowered machines.

Additional context.

No response

[dobairoland]

Thank you for the suggestion. This is not as simple as it may seem. Underneath idf.py flash is the cmake flash target where this had to be addressed as well.

I would prefer to use idf.py flash as it figures out what and where to flash the binaries from the current working directory.

It figures out the flashing settings based on the project configuration - which is linked to at least basic cmake project configuration. I think the current error message requesting to run idf.py fullclean because the project was moved is appropriate and helps to avoid potential issues.

I think using esptool directly for flashing binaries should be the preferred way. Perhaps we need to think about how to make this more user friendly, e.g. create an archive with binaries, file with esptool arguments and README.txt of how to invoke esptool without bothering about the right arguments.

[amrsoll]

Thank you for your prompt answer.

It figures out the flashing settings based on the project configuration

That would actually also work for me. File transfer is no issue, I just don't have the CPU power to build the projects comfortably from the remote device meant for flashing.

Unless it is a CMake behaviour, I still think it is best to use relative paths for the project files; in the interest of portability

Wouldn't it be possible to recreate the esptool settings given the config files?

[Edit] I didn't realise you actually addressed that point already, thank you!

Perhaps we need to think about how to make this more user friendly, e.g. create an archive with binaries, file with esptool arguments and README.txt of how to invoke esptool without bothering about the right arguments

Example:

$ generate_esptool_flash_command -p /dev/ttyUSB1 --project_dir . > flash_target.sh
$ sh ./flash_target.sh
[ ... esptool output ... ]
$ idf.py monitor
[ ... monitor output ... ]

[amrsoll]

Should I rename/move this issue to focus on having a more user-friendly esptool?

[dobairoland]

I updated the title. This needs to be addressed here, not in esptool.

Thank you for the suggestion!

[ESP-Marius]

What I usually do is to cd into the build folder and do esptool.py write_flash @flash_args, which is pretty convenient.

Not sure if that helps in your case or not.

[amrsoll]

@ESP-Marius

esptool.py write_flash @flash_args

That is actually suuuper convenient and really helps, I was totally unaware! I wish that was the default behaviour of write_flash when you're inside the build folder?

Current behaviour:

.../firmware/build $ esptool.py write_flash
esptool write_flash: error: the following arguments are required: <address> <filename>

Perhaps this tip could be the basic showcase for write_flash in the documentation so it is more prominent?

Thank you again for the help 👍

[igrr]

Perhaps this tip could be the basic showcase for write_flash in the documentation so it is more prominent?

For a long time we have tried to keep esptool code and docs as free as possible from IDF-isms, which is the reason why write_flash @flash_args is not mentioned in that section.

However, we did mention @flash_args in the description of merge_bin command, so perhaps we could do the same for write_flash.

[dobairoland]

The issue with running just esptool.py write_flash @flash_args is that it is not a one-to-one replacement of idf.py flash. There are some project specific setups missing (baudrate, chip, before & after behavior).

Here is a work-in-progress output change printed out in the end of idf.py build:

Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 2MB --flash_freq 40m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/blink.bin
or from the "/home/dragon/esp/esp-idf/examples/get-started/blink/build" directory
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash @flash_args

You can see that the last command is based on the above suggestion to use flash_args from the build directory, and it is extended with the missing argument to behave the same as idf.py flash would.