from vmware.vapi.vsphere.client import create_vsphere_client

[packet3]

Describe the bug

I am getting the following message when running my python script:

c:\SourceControl\dev\VspeherePython>python vspehere.py
<unknown>:1: SyntaxWarning: invalid escape sequence '\['
<unknown>:1: SyntaxWarning: invalid escape sequence '\['

the issue seems to be when importing the following:
from vmware.vapi.vsphere.client import create_vsphere_client

Reproduction steps

1. from vmware.vapi.vsphere.client import create_vsphere_client
2. execute the script
3. :1: SyntaxWarning: invalid escape sequence '['
   :1: SyntaxWarning: invalid escape sequence '['
   ...

Expected behavior

I should not expect to see :1: SyntaxWarning: invalid escape sequence '['
:1: SyntaxWarning: invalid escape sequence '['

Additional context

No response

[kunal-pmj]

can you share the vsphere.py?

[Zer0x00]

You need to use Python 3.12 to get a SyntaxWarning or run it with warnings enabled on Python 3.11 (then it will be a DeprecationWarning)

Code to reproduce:

from com.vmware.vapi.std.errors_client import Unauthorized

# or

from vmware.vapi.vsphere.client import create_vsphere_client

I've narrowed the issue down to the class URIValidator in vmware.vapi.bindings.uri_helper.
In the _rules variable the regexes for IRI_reference and IRI are causing the issue here.

[kunal-pmj]

Thanks @Zer0x00

This is bug reproducible with python 3.12. This is due to handling of bug http://bugs.python.org/issue3665
in the SDK.

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence, use raw strings for regular expression: re.compile(r"\d+.\d+")). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning. (Contributed by Victor Stinner in gh-98401.)

We will address this in the future release of the SDK

[kunal-pmj]

The issue will be addressed in the next SDK release

[sodul]

We are getting a lot of complaints about these warnings polluting our console outputs. I was also able to track at least one of the sources to vmware.vapi.bindings.uri_helper.

The URIValidator class is generating a bunch of regexes by combining strings and then passing them to ast.literal_eval(). This feels like a very inneficient way of generating the regexes at import time. The line in question is regex_str = ast.literal_eval(unicode_wrap.format(regex_str)) and this is why we get <unknown>:1: on the error making it very hard to track down.

I urge the maintainers of this code to avoid using eval and related methods as much as possible as it is removing the advantage of the cached pyc files and masks the origin of warnings.

Note that this has been raising DeprecationWarning since Python 3.6, and was made a SyntaxWarning in 3.12 to try to get it addressed in code such as here. It will raise a SyntaxError in future versions of python.

I did try to use -Werror to help track the source but there are other deprecation warnings that also need to be addressed:

> python -We -m vmware.vapi.bindings.uri_helper
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/Users/stephane/.pyenv/versions/3.12.1/lib/python3.12/site-packages/vmware/__init__.py", line 4, in <module>
    import pkg_resources
  File "/Users/stephane/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pkg_resources/__init__.py", line 118, in <module>
    warnings.warn(
DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

Links:
https://bugs.python.org/issue27364
https://docs.python.org/3/whatsnew/3.12.html#other-language-changes

[kunal-pmj]

The issue will be fixed in the next SDK release.

[mariolenz]

The issue will be fixed in the next SDK release.

Any ETA for this? Or will you wait until vSphere 8.0 U3 is released where AFAIR there's no official ETA?

[sodul]

@mariolenz as a temporary workaround we automatically patch the file to silence the issue:

Shell snippet:

    vcenter_location="$(pip show vcenter_bindings | grep Location: | sed -e 's/Location: //')"
    uri_helper_py="${vcenter_location}/vmware/vapi/bindings/uri_helper.py"
    if (( $(grep -c -m 1 'warnings.catch_warnings' "${uri_helper_py}") == 0 )); then
        echo "Patching VMWare's uri_helper.py to silence SyntaxWarning:"
        echo "  https://github.com/vmware/vsphere-automation-sdk-python/issues/400"
        patch -N -u "${uri_helper_py}" -i "${install_dir}/uri_helper.patch"
    else
        echo "VMWare's uri_helper.py already patched."
    fi

And the uri_helper.patch file:

--- uri_helper.py.orig	2024-01-17 12:53:21
+++ uri_helper.py	2024-01-17 12:53:58
@@ -6,6 +6,7 @@
 
 import re
 import ast
+import warnings
 
 from vmware.vapi.exception import CoreException
 from vmware.vapi.l10n.runtime import message_factory
@@ -95,7 +96,8 @@
             # ``\u`` and ``\U`` escapes must be preprocessed
             # http://bugs.python.org/issue3665
             unicode_wrap = 'u"""{0}"""'
-            regex_str = ast.literal_eval(unicode_wrap.format(regex_str))
+            with warnings.catch_warnings(action='ignore'):
+                regex_str = ast.literal_eval(unicode_wrap.format(regex_str))
 
             regex = re.compile(regex_str)
             compiled_regex[rule_type] = regex