autorest --postprocess script raises FileNotFoundError

[kristapratico]

I need to use the postprocessing script on the Question Answering client library for Python to avoid a linting error. Issue describing that here: Azure/azure-sdk-for-python#25557

When I run the postprocessing script, I'm getting a FileNotFoundError:

(py311) C:\Users\krpratic\azure-sdk-for-python\sdk\cognitivelanguage\azure-ai-language-questionanswering>autorest --postprocess --output-folder=C:\Users\krpratic\azure-sdk-for-python\sdk\cognitivelanguage\azure-ai-language-questionanswering --perform-load=false --python
AutoRest code generation utility [cli version: 3.6.1; node: v16.17.0]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    | AutoRest core version selected from configuration: ^3.7.0.

There is a new version of AutoRest available (3.6.2).
 > You can install the newer version with with npm install -g autorest@latest

info    |    Loading AutoRest core      'C:\Users\krpratic\.autorest\@[email protected]\nodemodules\@autorest\core\dist' (3.9.2)
info    |    Installing AutoRest extension '@autorest/python' (latest -> 6.1.7)
installing... [========================================] 100% | 16/16
info    |    Installed AutoRest extension '@autorest/python' (latest->6.1.7)
info    |    Loading AutoRest extension '@autorest/modelerfour' (~4.23.5->4.23.7)
info    | [blib2to3.pgen2.driver.loadgrammar:266] Generating grammar tables from C:\Users\krpratic\.autorest\@[email protected]\nodemodules\@autorest\python\venv\Lib\site-packages\blib2to3\Grammar.txt
info    | [blib2to3.pgen2.driver.load_grammar:269] Writing grammar tables to C:\Users\krpratic\AppData\Local\black\black\Cache\22.6.0\Grammar3.11.0.candidate.1.pickle
info    | [blib2to3.pgen2.driver.load_grammar:273] Writing failed: [Errno 2] No such file or directory: 'C:\\Users\\krpratic\\AppData\\Local\\black\\black\\Cache\\22.6.0\\tmptruvd7e7'
info    | [blib2to3.pgen2.driver.loadgrammar:266] Generating grammar tables from C:\Users\krpratic\.autorest\@[email protected]\nodemodules\@autorest\python\venv\Lib\site-packages\blib2to3\PatternGrammar.txt
info    | [blib2to3.pgen2.driver.load_grammar:269] Writing grammar tables to C:\Users\krpratic\AppData\Local\black\black\Cache\22.6.0\PatternGrammar3.11.0.candidate.1.pickle
info    | [blib2to3.pgen2.driver.loadgrammar:273] Writing failed: [Errno 2] No such file or directory: 'C:\\Users\\krpratic\\AppData\\Local\\black\\black\\Cache\\22.6.0\\tmp04ozn2a'
error   | [_main_.Process:75] Python generator raised an exception
Traceback (most recent call last):
  File "C:\Users\krpratic\.autorest\@[email protected]\nodemodules\@autorest\python\autorest\jsonrpc\server.py", line 73, in Process
    return plugin.process()
           ^^^^^^^^^^^^^^^^
  File "C:\Users\krpratic\.autorest\@[email protected]\node_modules\@autorest\python\autorest\postprocess\__init_.py", line 106, in process  
    self.fiximports_ininit(
  File "C:\Users\krpratic\.autorest\@[email protected]\node_modules\@autorest\python\autorest\postprocess\__init__.py", line 146, in fix_imports_ininit
    customizedobjects_str = pythonrun(
                             ^^^^^^^^^^^
  File "C:\Users\krpratic\.autorest\@[email protected]\node_modules\@autorest\python\autorest\postprocess\venvtools.py", line 60, in pythonrun
    subprocess.run(
  File "C:\Python311rc\Lib\subprocess.py", line 546, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311rc\Lib\subprocess.py", line 1022, in _init_
    self.execute_child(args, executable, preexec_fn, closefds,
  File "C:\Python311rc\Lib\subprocess.py", line 1491, in executechild
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
fatal   | Process() cancelled due to failure 
error   |   Error: Plugin postprocess reported failure.
error   | Autorest completed with an error. If you think the error message is unclear, or is a bug, please declare an issues at https://github.com/Azure/autorest/issues with the error message you are seeing.

swagger README is here (if you need it): https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cognitivelanguage/azure-ai-language-questionanswering/swagger/README.md

[iscai-msft]

Here is the postprocessing: https://github.com/Azure/autorest.python/blob/autorestv3/docs/customizations.md#postprocessing information. @msyyc @changlong-liu can you guys take a look? I did a quick hacky way of getting a postprocessing step, looks like it can be improved.

[changlong-liu]

Sure @iscai-msft .

Hi @kristapratico ,
I can't reproduce this problem in my two windows machine yet.
Want to confirm does the problem exists on the latest code here? https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cognitivelanguage/azure-ai-language-questionanswering

And will this problem still occur after doing below local cleaning steps?

1. > autorest --reset
2. delete the folder "sdk\cognitivelanguage\azure-ai-language-questionanswering.temp_folder"

[kristapratico]

Thanks @changlong-liu

Want to confirm does the problem exists on the latest code here? https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cognitivelanguage/azure-ai-language-questionanswering

Yes. Upon doing a reset and deleting the .temp folder I was able to run the script.

However, this seems to have regenerated in a way that breaks the code. I've pushed the output from script here: https://github.com/Azure/azure-sdk-for-python/compare/main...kristapratico:azure-sdk-for-python:qa-fix-lint?expand=1

[changlong-liu]

Thanks @changlong-liu

Want to confirm does the problem exists on the latest code here? https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cognitivelanguage/azure-ai-language-questionanswering

Yes. Upon doing a reset and deleting the .temp folder I was able to run the script.

However, this seems to have regenerated in a way that breaks the code. I've pushed the output from script here: https://github.com/Azure/azure-sdk-for-python/compare/main...kristapratico:azure-sdk-for-python:qa-fix-lint?expand=1

Thanks @kristapratico! I have the same situation now.

Looks like the problem of the generation is that it falsely modified the below two file:

• azure/ai/language/questionanswering/authoring/_operations/init.py
• azure/ai/language/questionanswering/authoring/aio/_operations/init.py

I guess the "authoring" is manually copied to the generated "questionanswering"? I think currently the --postprocess has problem with this embedded-packages situation.

As a temporary solution, will it be non breaking if you revert these two files?
If we also want the "authoring" get postprocessed, we may need to do "--postprocess" in the authoring's original generation folder first and then copy them to "questionanswering".

[kristapratico]

I can revert the authoring changes for now. I think there still might be a bug in the script for the files which should be edited:

• azure/ai/language/questionanswering/_operations/init.py
• azure/ai/language/questionanswering/aio/_operations/init.py
• azure/ai/language/questionanswering/models/init.py

My understanding is that the script should generate something like this in the init file: https://github.com/Azure/autorest.python/blob/autorestv3/packages/autorest.python/test/dpg/version-tolerant/Expected/AcceptanceTests/DPGCustomizationCustomizedVersionTolerant/dpgcustomizationcustomizedversiontolerant/_operations/__init__.py

The files I listed above, still have imports from the non-patch code: https://github.com/kristapratico/azure-sdk-for-python/blob/80afbdd98e096f2b1e253dfc64c0b6f79e42034a/sdk/cognitivelanguage/azure-ai-language-questionanswering/azure/ai/language/questionanswering/__init__.py#L9 and also have duplicate types defined in the all: https://github.com/kristapratico/azure-sdk-for-python/blob/80afbdd98e096f2b1e253dfc64c0b6f79e42034a/sdk/cognitivelanguage/azure-ai-language-questionanswering/azure/ai/language/questionanswering/__init__.py#L22