Autorest failing with ModuleNotFoundError: No module named 'jsonrpc'

[forteddyt]

Swagger spec being used: https://github.com/zhaomuzhi/azure-rest-api-specs/tree/8d9b6a0d3040feaf8ccaf18e5cabceb0820767bd/specification/machinelearningservices/resource-manager

Repro steps:

npm uninstall -g autorest
npm install -g autorest@latest

autorest --reset

autorest --az --clear-output-folder=true --azure-cli-extension-folder=cli --multiapi readme.md

Output:

SourceMapConsumer.initialize is a no-op when running in node.js
AutoRest code generation utility [cli version: 3.5.1; node: v16.13.1]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
installing core... [========================================] 100% | 2/2
SourceMapConsumer.initialize is a no-op when running in node.js
info    |    Loading AutoRest core      'C:\Users\<USER>\.autorest\@[email protected]\nodemodules\@autorest\core\dist' (3.7.2)
info    |    Installing AutoRest extension '@autorest/az' (latest -> 1.8.0)
installing... [========================================] 100% | 2264/2264
info    |    Installed AutoRest extension '@autorest/az' (latest->1.8.0)
info    |    Installing AutoRest extension '@autorest/python' (5.4.0 -> 5.4.0)
installing... [========================================] 100% | 916/916
info    |    Installed AutoRest extension '@autorest/python' (5.4.0->5.4.0)
info    |    Installing AutoRest extension '@autorest/clicommon' (0.6.2 -> 0.6.2)
installing... [========================================] 100% | 51/51
info    |    Installed AutoRest extension '@autorest/clicommon' (0.6.2->0.6.2)
info    |    Installing AutoRest extension '@autorest/modelerfour' (4.15.421 -> 4.15.421)
installing... [========================================] 100% | 2/2
info    |    Installed AutoRest extension '@autorest/modelerfour' (4.15.421->4.15.421)
Executing: C:\Users\<USER>\.autorest\@[email protected]\node_modules\@autorest\python\venv\Scripts\python.exe -m autorest.jsonrpc.server
Traceback (most recent call last):
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\<USER>\.autorest\@[email protected]\node_modules\@autorest\python\autorest\jsonrpc\server.py", line 11, in <module>
    from jsonrpc import dispatcher, JSONRPCResponseManager
ModuleNotFoundError: No module named 'jsonrpc'
Command '['C:\\Users\\<USER>\\.autorest\\@[email protected]\\node_modules\\@autorest\\python\\venv\\Scripts\\python.exe', '-m', 'autorest.jsonrpc.server']' returned non-zero exit status 1.
fatal   | Process() cancelled due to failure
error   | [Exception] AutoRest extension '@autorest/python' terminated.
error   |   Error: [Exception] AutoRest extension '@autorest/python' terminated.

We're seeing this issue in an internal Azure DevOps ubuntu 18 pipeline, but I am able to repro locally on Windows 10.

[lynshi]

I ran into the same problem today on Ubuntu 20.04. I've had issues in the past with autorest not installing its Python modules correctly and resolved it by manually installing dependencies; see a similar workaround below. The command I run for the CodeGen is autorest --version=3.0.6370 --az <...> (from within a virtualenv containing azure-cli, azdev) as in the sample command from the docs.

1. cd to ~/.autorest/@[email protected]/node_modules/@autorest/python to find the dependencies listed in setup.py. It looks like

install_requires=[
        "json-rpc",
        "Jinja2 >= 2.11", # I need "include" and auto-context + blank line are not indented by default
        "pyyaml",
        "m2r",
    ],

Note there are later versions of autorest_python available, but this version of autorest will try to use this one. I think there might be a flag for picking the autorest_python version but I don't remember it and it didn't show under --help; using a different version could also resolve the dependency installation issue.

2. We'll need to install the above dependencies into the venv found in the @autorest/python folder.
   a. venv/bin/python3 -m pip install json-rpc
   b. venv/bin/python3 -m pip install Jinja2
   c. venv/bin/python3 -m pip install pyyaml (this might already be installed)
   d. m2r is trickier because of a breaking API change in the mistune dependency that will cause the m2r installation to fail. As described in this GitHub issue, run venv/bin/python3 -m pip install mistune==0.8.4 to revert mistune to a compatible version. Then, venv/bin/python3 -m pip install m2r to install m2r.