test_cmd_install test cases failing on macOS in CI

[akihironitta]

Bug description

These app tests are failing in master and blocking my PR.
https://github.com/Lightning-AI/lightning/actions/runs/3636672987/jobs/6136867843

Error messages and logs

=========================== short test summary info ============================
FAILED tests_app/cli/test_cmd_install.py::test_valid_org_app_name - assert "app: 'fakeuser/ALKKLJAUHREKJ21234KLAKJDLF' is not available on ⚡ Lightning AI ⚡" in ''
 +  where '' = <Result JSONDecodeError('Expecting value: line 1 column 1 (char 0)')>.output
FAILED tests_app/cli/test_cmd_install.py::test_valid_org_component_name - assert "component: 'fakeuser/ALKKLJAUHREKJ21234KLAKJDLF' is not available on ⚡ Lightning AI ⚡" in ''
 +  where '' = <Result JSONDecodeError('Expecting value: line 1 column 1 (char 0)')>.output
FAILED tests_app/cli/test_cmd_install.py::test_prompt_actions - AssertionError: assert 'Installation aborted!' in ''
 +  where '' = <Result JSONDecodeError('Expecting value: line 1 column 1 (char 0)')>.output
FAILED tests_app/cli/test_cmd_install.py::test_version_arg_component - assert "component: 'Version NOT-EXIST for lightning/lit-slack-messenger' is not" in 'Expecting value: line 1 column 1 (char 0)'
 +  where 'Expecting value: line 1 column 1 (char 0)' = str(JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))
 +    where JSONDecodeError('Expecting value: line 1 column 1 (char 0)') = <Result JSONDecodeError('Expecting value: line 1 column 1 (char 0)')>.exception
FAILED tests_app/cli/test_cmd_install.py::test_version_arg_app - assert "app: 'Version NOT-EXIST for lightning/invideo' is not" in 'Expecting value: line 1 column 1 (char 0)'
 +  where 'Expecting value: line 1 column 1 (char 0)' = str(JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))
 +    where JSONDecodeError('Expecting value: line 1 column 1 (char 0)') = <Result JSONDecodeError('Expecting value: line 1 column 1 (char 0)')>.exception
FAILED tests_app/cli/test_cmd_install.py::test_proper_url_parsing - requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
===== 6 failed, 582 passed, 52 skipped, 1978 warnings in 382.68s (0:06:22) =====
Error: Process completed with exit code 1.

Environment

https://github.com/Lightning-AI/lightning/actions/runs/3638338139/jobs/6140786023

More info

No response

cc @carmocca @akihironitta @Borda

[Borda]

ok, I saw it a few times too, it is very random and not sure what is happening...
cc: @tchaton @ethanwharris @awaelchli

[tchaton]

I am not 100 % but I think it could be rate limiting from the platform side.

[akihironitta]

I was wondering the same. I have seen this error only on macOS runners, which also makes me think that it's something related to the framework since I assume the behaviour from the platform side doesn't change depending on clients' OS.

I tried to reproduce the error and debug it by ssh into the runner in #15937, but I no longer see any errors in the last few CI runs. I'll just leave this issue open for now until we become confident that this is not an issue anymore.

[akihironitta]

@tchaton @ethanwharris Would you mind closing this issue if it's resolved?

[ethanwharris]

I think this was due to some issue on the platform that is now resolved, probably safe to close it 😃