Unit testing fails with Arduino Micro due to COM port change

[SebastianSteiner]

What kind of issue is this?

• Question.
  This issue tracker is not the place for questions. If you want to ask how to do something,
  or to understand why something isn't working the way you expect it to,
  use Community Forums or Premium Support

• PlatformIO IDE.
  All issues related to PlatformIO IDE should be reported to appropriate repository:
  PlatformIO IDE for Atom or
  PlatformIO IDE for VSCode

• Development Platform or Board.
  All issues (building, uploading, adding new boards, etc.) related to PlatformIO development platforms
  should be reported to appropriate repository related to your hardware
  https://github.com/topics/platformio-platform

• Feature Request.
  Start by telling us what problem you’re trying to solve. Often a solution
  already exists! Don’t send pull requests to implement new features without first getting our
  support. Sometimes we leave features out on purpose to keep the project small.

• [ X] PlatformIO Core.
  If you’ve found a bug, please provide an information below.

You can erase any parts of this template not applicable to your Issue.

---

Configuration

Operating system: Windows 10 64 bit

PlatformIO Version (platformio --version): 4.3.4

Description of problem

When running unit tests on an Arduino Micro using the atmelavr platform and arduino framework, the test runner fails. The reason is that the Micro uses the native USB port for serial communication, and the bootloader and the application code appear as different COM ports. There is a slight delay after reset, before the COM port becomes available. Therefore, PIO cannot immediately connect and throws an error.

Steps to Reproduce

1. Create a project for an Arduino Micro

[env:micro]
platform = atmelavr
board = micro
framework = arduino

2. Create a testable library and a unit test for it
3. Run the Test command for that env

Actual Results

The firmware upload succeeds, but right after, I get the following error:

Error: Traceback (most recent call last):
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\command.py", line 170, in cli
    "succeeded": tp.process(),
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\embedded.py", line 52, in process
    return self.run()
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\platformio\commands\test\embedded.py", line 83, in run
    line = ser.readline().strip()
  File "c:\users\sebastian\.platformio\penv\lib\site-packages\serial\serialwin32.py", line 273, in read
    raise SerialException("ClearCommError failed ({!r})".format(ctypes.WinError()))
serial.serialutil.SerialException: ClearCommError failed (PermissionError(13, 'The device does not recognize the command.', None, 22))

============================================================

An unexpected error occurred. Further steps:

* Verify that you have the latest version of PlatformIO using
  `pip install -U platformio` command

* Try to find answer in FAQ Troubleshooting section
  https://docs.platformio.org/page/faq.html

* Report this problem to the developers
  https://github.com/platformio/platformio-core/issues

============================================================

This is likely because once the bootloader exits, the COM port disappears. If I explicitly specify a test_port in my ini, I get:

could not open port 'COM5': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)

Expected Results

The test is expected to just run and report success or failure.

Additional info

I suspected that this issue was caused by the delay between the bootloader exiting and the user application creating a VCP. Thus, in line 62 of embedded.py I added a sleep(2). With that, the test runs as expected, regardless of whether or not I specify a test_port.

I propose a solution along the lines of:

retries = 0
while True:
    try:
        ser = serial.Serial(
            baudrate=self.get_baudrate(), timeout=self.SERIAL_TIMEOUT
        )
        ser.port = self.get_test_port()
        ser.rts = self.options["monitor_rts"]
        ser.dtr = self.options["monitor_dtr"]
        ser.open()
    except serial.SerialException as e:
        if retries < max_retries:
            retries += 1
            sleep(a_little_while)
            continue
        else:
            click.secho(str(e), fg="red", err=True)
            return False

This would give a seamless experience for other boards that connect on the first try, but allow laggard Micros to still connect.

PS: This seems to be related to an issue in platform-atmelavr: platformio/platform-atmelavr#55

[SebastianSteiner]

For everyone who's having trouble with that, until a fix is available, here's a workaround using an extra script:

[env:micro]
platform = atmelavr
board = micro
framework = arduino
extra_scripts = post:test/extra_script.py

The extra_script.py:

Import("env")

def after_upload(source, target, env):
    import time
    time.sleep(2)

env.AddPostAction("upload", after_upload)

[ivankravets]

Duplicate of #3742