From 70b5e98cd35acdb83dea75cc553378fcd4d61946 Mon Sep 17 00:00:00 2001 From: Wolfgang Ziegler Date: Tue, 12 Nov 2019 14:51:00 +0100 Subject: [PATCH 1/2] Prepare release v1.3.0 --- README.md | 87 ++++++++----- docs/conf.py | 2 +- pylintrc | 4 +- samples/fork-sdk-sample/README.md | 4 + samples/fork-sdk-sample/fork_sdk_sample.py | 128 ++++++++++++++++++++ samples/fork-sdk-sample/setup.py | 55 +++++++++ setup.py | 115 +++++++++++++++--- src/oneagent/__init__.py | 52 ++++++-- src/oneagent/_impl/native/sdkctypesiface.py | 21 ++-- src/oneagent/_impl/native/sdknulliface.py | 2 +- src/oneagent/_impl/native/sdkversion.py | 4 +- src/oneagent/common.py | 18 ++- src/oneagent/sdk/__init__.py | 7 +- src/oneagent/sdk/tracers.py | 5 - src/oneagent/version.py | 32 +++++ test-util-src/sdkmockiface.py | 2 +- test/test_load_old_agent.py | 11 +- test/test_public_sdk.py | 115 +++++++++--------- tox.ini | 14 ++- 19 files changed, 532 insertions(+), 146 deletions(-) create mode 100644 samples/fork-sdk-sample/README.md create mode 100644 samples/fork-sdk-sample/fork_sdk_sample.py create mode 100644 samples/fork-sdk-sample/setup.py create mode 100644 src/oneagent/version.py diff --git a/README.md b/README.md index a0cc751..b154d5e 100644 --- a/README.md +++ b/README.md @@ -19,14 +19,14 @@ This SDK enables Dynatrace customers to extend request level visibility into Pyt * [Incoming web requests](#incoming-web-requests) * [Outgoing web requests](#outgoing-web-requests) * [Trace in-process asynchronous execution](#trace-in-process-asynchronous-execution) - * [Custom request attributes](#custom-request-attributes) + * [Custom Request Attributes](#custom-request-attributes) * [Custom services](#custom-services) * [Messaging](#messaging) - * [Outgoing Messages](#outgoing-messaging) - * [Incoming Messages](#incoming-messaging) + + [Outgoing Messages](#outgoing-messages) + + [Incoming Messages](#incoming-messages) +- [Using the OneAgent SDK for Python with forked child processes (only available on Linux)](#using-the-oneagent-sdk-for-python-with-forked-child-processes-only-available-on-linux) - [Troubleshooting](#troubleshooting) - * [Installation issues](#installation-issues) - * [Post-installation issues](#post-installation-issues) + * [Extended SDK State](#extended-sdk-state) - [Repository contents](#repository-contents) - [Help & Support](#help--support) * [Read the manual](#read-the-manual) @@ -40,12 +40,14 @@ This SDK enables Dynatrace customers to extend request level visibility into Pyt ## Requirements The SDK supports Python 2 ≥ 2.7 and Python 3 ≥ 3.4. Only the official CPython (that is, the "normal" Python, i.e. the Python implementation -from ) is supported and only on Linux (musl libc is currently not supported) and Windows with the x86 (including -x86-64) architecture. Additionally, `pip` ≥ 8.1.0 (2016-03-05) is required for installation. +from ) is supported and only on Linux (musl libc which is used, e.g., on Alpine Linux, is currently not supported) +and Windows with the x86 (including x86-64) architecture. +Additionally, `pip` ≥ 8.1.0 (2016-03-05) is required for installation, and on Linux, the system should be +[`manylinux1`-compatible](https://www.python.org/dev/peps/pep-0513/) to ensure a smooth installation via `pip`. The Dynatrace OneAgent SDK for Python is a wrapper of the [Dynatrace OneAgent SDK for C/C++](https://github.com/Dynatrace/OneAgent-SDK-for-C) and therefore the SDK for C/C++ is required and delivered with the Python SDK. See -[here](https://github.com/Dynatrace/OneAgent-SDK-for-C#dynatrace-oneagent-sdk-for-cc-requirements) +[here](https://github.com/Dynatrace/OneAgent-SDK-for-C#requirements) for its requirements, which also apply to the SDK for Python. The version of the SDK for C/C++ that is included in each version of the SDK for Python is shown in the following table along with the required @@ -56,6 +58,7 @@ Dynatrace OneAgent version (it is the same as |OneAgent SDK for Python|OneAgent SDK for C/C++|Dynatrace OneAgent|Support status | |:----------------------|:---------------------|:-----------------|:------------------| +|1.3 |1.5.1 |≥1.179 |Supported | |1.2 |1.4.1 |≥1.161 |Supported | |1.1 |1.3.1 |≥1.151 |Supported | |1.0 |1.1.0 |≥1.141 |EAP (not supported)| @@ -429,7 +432,6 @@ the callback of a periodic timer. ```python with sdk.trace_custom_service('onTimer', 'CleanupTask'): # Do the cleanup task - : ``` Check out the documentation at: @@ -535,31 +537,34 @@ See the documentation for more information: * [General information on tagging](https://dynatrace.github.io/OneAgent-SDK-for-Python/docs/tagging.html) * [Messaging tracers in the specification repository](https://github.com/Dynatrace/OneAgent-SDK#messaging) - -## Troubleshooting + + +## Using the OneAgent SDK for Python with forked child processes (only available on Linux) - -### Installation issues +Some applications, especially web servers, use a concurrency model that is based on forked child processes. +Typically a master process is started which is responsible only for creating and managing child processes by means of forking. +The child processes do the real work, for example handling web requests. -* `ValueError` when installing, complaining about missing `DT_PYSDK_CSDK_PATH`. +The recommended way to use the Python SDK in such a scenario is as follows: You initialize the SDK in the master process setting +the `forkable` argument to `True`. - Make sure you are using pip to install a prebuilt package wheel for your system from PyPI, as described in [Using the OneAgent SDK for - Python in your application](#installation). Also make sure you are using an up-to date version of `pip`, `setuptools` and `wheel`. You can - try upgrading them with `python -m pip install --upgrade pip setuptools wheel` (make sure to use the same `python` that you use to install - the `oneagent-sdk` package). ATTENTION: If you use the system-provided pip (e.g. installed via `apt-get` on Ubuntu) you should instead use - a `pip` inside a `virtualenv` (the same as your project), as uprading system-provided packages via `pip` may cause issues. +```python +oneagent.initialize(sdk_options, forkable=True) +``` - If this does not resolve the issue, make sure you are using a supported platform, as listed in [Requirements](#requirements). If you *are* - using a supported system, you can try downloading the [OneAgent SDK for C/C++](https://github.com/Dynatrace/OneAgent-SDK-for-C) in the - version corresponding to your OneAgent SDK for Python as listed in [the table in Requirements](#requirements). Then set the - `DT_PYSDK_CSDK_PATH` environment variable to the `.so`/`.dll` file corresponding to your platform in the `lib` subdirectory of the C SDK - and retry the installation (e.g. in a bash shell, use `export DT_PYSDK_CSDK_PATH=path/to/onesdk_shared.so`). If there is no corresponding - directory, your platfom is not supported. Otherwise, regardless if it works with that method or not, please report an issue as desribed in - [Let us help you](#let-us-help-you). +This way you will not be able to use the SDK in the master process (attempts to do so will be ignored, if applicable with +an error code), but all forked child processes will share the same agent. This has a lower overhead, for example the +startup of worker processes is not slowed down, and the per-worker memory overhead is reduced. +For more information on forked child processes, take a look at those resources: +* [Documentation on forking for the Dynatrace OneAgent SDK for C/C++](https://github.com/Dynatrace/OneAgent-SDK-for-C/blob/master/README.md#forking) +* [Forking sample application](./samples/fork-sdk-sample/fork_sdk_sample.py) + + +## Troubleshooting + -### Post-installation issues To debug your OneAgent SDK for Python installation, execute the following Python code: @@ -580,6 +585,34 @@ Known gotchas: Make sure that the `pip install` or equivalent succeeded (see [here](#installation)). Also make sure you use the `pip` corresponding to your `python` (if in doubt, use `python -m pip` instead of `pip` for installing). +* Output ending in a message similar to `InitResult=InitResult(status=-2, error=SDKError(-1342308345, 'Failed loading SDK stub from .../site-packages/oneagent/_impl/native/libonesdk_shared.so: "/.../libonesdk_shared.so: cannot open shared object file: No such file or directory". Check your installation of the oneagent-sdk Python package, e.g., try running `pip install --verbose --force-reinstall oneagent-sdk`.'))`. + + Follow the advice of the message and run `python -m pip install --verbose --force-reinstall oneagent-sdk` + (or the equivalent pip invocation with the `--verbose` and `--force-reinstall` flags). + It is likely that you will now see another message similar to + + ****************************************************************************** + *** You are trying to build the Python SDK from source. *** + *** This could mean that you are using an outdated version of pip (older *** + *** than 8.1.0) or you are attempting to install the SDK on an *** + *** unsupported platform. Please check the requirements at *** + *** https://github.com/Dynatrace/OneAgent-SDK-for-Python#requirements *** + ****************************************************************************** + + Make sure you are using pip to install a prebuilt package wheel for your system from PyPI, as described in [Using the OneAgent SDK for + Python in your application](#installation). Also make sure you are using an up-to date version of `pip`, `setuptools` and `wheel`. You can + try upgrading them with `python -m pip install --upgrade pip setuptools wheel` (make sure to use the same `python` that you use to install + the `oneagent-sdk` package). ATTENTION: If you use the system-provided pip (e.g. installed via `apt-get` on Ubuntu) you should instead use + a `pip` inside a `virtualenv` (the same as your project), as upgrading system-provided packages via `pip` may cause issues. + + If this does not resolve the issue, make sure you are using a supported platform, as listed in [Requirements](#requirements). If you *are* + using a supported system, you can try downloading the [OneAgent SDK for C/C++](https://github.com/Dynatrace/OneAgent-SDK-for-C) in the + version corresponding to your OneAgent SDK for Python as listed in [the table in Requirements](#requirements). Then set the + `DT_PYSDK_CSDK_PATH` environment variable to the `.so`/`.dll` file corresponding to your platform in the `lib` subdirectory of the C SDK + and retry the installation (e.g. in a bash shell, use `export DT_PYSDK_CSDK_PATH=path/to/onesdk_shared.so`). If there is no corresponding + directory, your platfom is not supported. Otherwise, regardless if it works with that method or not, please report an issue as described + in [Let us help you](#let-us-help-you). + ### Extended SDK State diff --git a/docs/conf.py b/docs/conf.py index 994a700..1b1b512 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -37,7 +37,7 @@ import datetime -from oneagent import __version__ as version +from oneagent.version import __version__ as version # Fix spurious "= None" for instance attributes # https://github.com/sphinx-doc/sphinx/issues/2044 diff --git a/pylintrc b/pylintrc index 92a2f7c..4ed6b6d 100644 --- a/pylintrc +++ b/pylintrc @@ -12,6 +12,6 @@ good-names=i,j,e,_ max-line-length=100 [TYPECHECK] -ignored-classes=oneagent._impl.native.sdkctypesiface.SDKDllInterface +ignored-classes=oneagent._impl.native.sdkctypesiface.SDKDllInterface,oneagent.sdk.tracers.Tracer redefining-builtins-modules=oneagent._impl.six.moves -disable=missing-docstring,fixme,too-few-public-methods,missing-return-doc,useless-object-inheritance +disable=missing-docstring,fixme,too-few-public-methods,missing-return-doc,useless-object-inheritance,cyclic-import,import-outside-toplevel diff --git a/samples/fork-sdk-sample/README.md b/samples/fork-sdk-sample/README.md new file mode 100644 index 0000000..e75d905 --- /dev/null +++ b/samples/fork-sdk-sample/README.md @@ -0,0 +1,4 @@ +# OneAgent SDK for Python forking sample + +This example demonstrates how to use the [OneAgent SDK for +Python](https://github.com/Dynatrace/OneAgent-SDK-for-Python) with forked child processes. diff --git a/samples/fork-sdk-sample/fork_sdk_sample.py b/samples/fork-sdk-sample/fork_sdk_sample.py new file mode 100644 index 0000000..b804828 --- /dev/null +++ b/samples/fork-sdk-sample/fork_sdk_sample.py @@ -0,0 +1,128 @@ +# -*- coding: utf-8 -*- +# +# Copyright 2019 Dynatrace LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +'''This example demonstrates how to use the OneAgent SDK for Python in a +parent/child process environment. The OneAgent SDK for Python will be initialized +in the parent process and the child processes can then use the SDK. + +Note: this example will only work on Linux. There's no Windows support available. +''' + +import os +import sys + +import oneagent # SDK initialization functions +import oneagent.sdk as onesdk # All other SDK functions. + +try: # Python 2 compatibility. + input = raw_input #pylint:disable=redefined-builtin +except NameError: + pass + + +getsdk = oneagent.get_sdk # Just to make the code shorter. + + +def do_some_fancy_stuff(proc_number): + sdk = getsdk() + + # The agent state in the child process should be ACTIVE (0). + print('Agent state (child process #{}): {}'.format(proc_number, sdk.agent_state), flush=True) + + print('Agent found:', sdk.agent_found) + print('Agent is compatible:', sdk.agent_is_compatible) + print('Agent version:', sdk.agent_version_string) + + # This call below will complete the OneAgent for Python SDK initialization and then it + # will start the tracer for tracing the custom service + with sdk.trace_custom_service('my_fancy_transaction', 'MyFancyService #{}'.format(proc_number)): + print('do some fancy stuff') + +def create_child_process(proc_number): + pid = os.fork() + if pid == 0: + print('child #{} is running ...'.format(proc_number)) + do_some_fancy_stuff(proc_number) + print('child #{} is exiting ...'.format(proc_number)) + sys.exit(0) + + return pid + +def fork_children(): + print('now starting children ...', flush=True) + pid_1 = create_child_process(1) + pid_2 = create_child_process(2) + + print('waiting for child #1 ...', flush=True) + os.waitpid(pid_1, 0) + print('child #1 exited', flush=True) + + print('waiting for child #2 ...', flush=True) + os.waitpid(pid_2, 0) + print('child #2 exited', flush=True) + + print('all children exited', flush=True) + +def main(): + # This gathers arguments prefixed with '--dt_' from sys.argv into the + # returned list. See also the basic-sdk-sample. + sdk_options = oneagent.sdkopts_from_commandline(remove=True) + + # Before using the SDK you have to initialize the OneAgent. In this scenario, we + # initialize the SDK and prepare it for forking. + # + # Passing in the sdk_options is entirely optional and usually not required + # as all settings will be automatically provided by the Dynatrace OneAgent + # that is installed on the host. + # + # To activate the forking support add the optional 'forkable' parameter and set it to True. + # + # If you run this example on Windows then you'll get an "Invalid Argument" error back + # because there's no forking support for Windows available. + init_result = oneagent.initialize(sdk_options, forkable=True) + try: + if init_result.error is not None: + print('Error during SDK initialization:', init_result.error) + + # While not by much, it is a bit faster to cache the result of + # oneagent.get_sdk() instead of calling the function multiple times. + sdk = getsdk() + + # The agent state is one of the integers in oneagent.sdk.AgentState. + # Since we're using the 'forkable' mode the state will be TEMPORARILY_INACTIVE (1) on Linux. + print('Agent state (parent process):', sdk.agent_state) + + # The instance attribute 'agent_found' indicates whether an agent could be found or not. + print('Agent found:', sdk.agent_found) + + # If an agent was found but it is incompatible with this version of the SDK for Python + # then 'agent_is_compatible' would be set to false. + print('Agent is compatible:', sdk.agent_is_compatible) + + # The agent version is a string holding both the OneAgent version and the + # OneAgent SDK for C/C++ version separated by a '/'. + print('Agent version:', sdk.agent_version_string) + + if init_result.error is None: + fork_children() + input('Now wait until the path appears in the UI ...') + finally: + shutdown_error = oneagent.shutdown() + if shutdown_error: + print('Error shutting down SDK:', shutdown_error) + +if __name__ == '__main__': + main() diff --git a/samples/fork-sdk-sample/setup.py b/samples/fork-sdk-sample/setup.py new file mode 100644 index 0000000..6932a28 --- /dev/null +++ b/samples/fork-sdk-sample/setup.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +# +# Copyright 2018 Dynatrace LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import io + +from setuptools import setup + +with io.open('README.md', encoding='utf-8') as readmefile: + long_description = readmefile.read() +del readmefile + +setup( + py_modules=['fork_sdk_sample'], + zip_safe=True, + name='oneagent-sdk-fork-sample', + version='0.0', # This sample is not separately versioned + + install_requires=['oneagent-sdk==1.*,>=1.3'], + + description='OneAgent SDK for Python: Fork sample application', + long_description=long_description, + long_description_content_type='text/markdown', + url='https://github.com/Dynatrace/OneAgent-SDK-for-Python', + maintainer='Dynatrace LLC', + maintainer_email='dynatrace.oneagent.sdk@dynatrace.com', + license='Apache License 2.0', + entry_points={ + 'console_scripts': ['oneagent-sdk-basic-sample=fork_sdk_sample:main'], + }, + classifiers=[ + 'Intended Audience :: Developers', + 'License :: OSI Approved', + 'License :: OSI Approved :: Apache Software License', # 2.0 + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: Implementation :: CPython', + 'Operating System :: POSIX :: Linux', + 'Operating System :: Microsoft :: Windows', + 'Topic :: System :: Monitoring' + ]) diff --git a/setup.py b/setup.py index df3cdb2..3a50e73 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,6 @@ #! /usr/bin/env python3 # -# Copyright 2018 Dynatrace LLC +# Copyright 2019 Dynatrace LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -55,24 +55,76 @@ long_description = readmefile.read() del readmefile -VER_RE = re.compile(r"^__version__ = '([^']+)'$") -verfilepath = path.join(_THIS_DIR, 'src/oneagent/__init__.py') -with io.open(verfilepath, encoding='utf-8') as verfile: - for line in verfile: - match = VER_RE.match(line) - if match: - __version__ = match.group(1) - break - else: - raise AssertionError('Version not found in src/oneagent/__init__.py') +def get_version_from_pkg_info(): + pkg_info_path = path.join(_THIS_DIR, 'PKG-INFO') + + if not path.isfile(pkg_info_path): + return None + + ver_re = re.compile(r"^Version: (.+)$") + + with io.open(pkg_info_path, encoding='utf-8') as pinfofile: + for line in pinfofile: + match = ver_re.match(line) + if match: + return match.group(1) + + return None + +def get_version_from_version_py(): + ver_re = re.compile(r"^__version__ = '([^']+)'$") + + verfilepath = path.join(_THIS_DIR, 'src/oneagent/version.py') + with io.open(verfilepath, encoding='utf-8') as verfile: + for line in verfile: + match = ver_re.match(line) + if match: + version = match.group(1) + break + else: + raise AssertionError('Version not found in src/oneagent/version.py') + + build_timestamp = os.environ.get('BUILD_TIMESTAMP') + if build_timestamp: + version = '{}.{}'.format(version, re.sub(r'-0*', '.', build_timestamp)) + + return version + + +__version__ = get_version_from_pkg_info() +if not __version__: + __version__ = get_version_from_version_py() -del match, verfile, verfilepath, VER_RE if __version__ != str(parse_version(__version__)): raise AssertionError( 'Version {} normalizes to {}'.format( __version__, parse_version(__version__))) +# This function was adapted from https://www.python.org/dev/peps/pep-0513/ +# (public domain) +def get_glibc_version_string(): + import ctypes + + try: + process_namespace = ctypes.CDLL(None) + gnu_get_libc_version = process_namespace.gnu_get_libc_version + + # Call gnu_get_libc_version, which returns a string like "2.5". + gnu_get_libc_version.restype = ctypes.c_char_p + version_str = gnu_get_libc_version() + + # py2 / py3 compatibility: + if not isinstance(version_str, str): + version_str = version_str.decode("ascii") + return version_str + except Exception: #pylint:disable=broad-except + # Symbol doesn't exist -> therefore, we are not linked to + # glibc. + return None + + + def unsupported_msg(plat_name): try: import pip @@ -80,6 +132,14 @@ def unsupported_msg(plat_name): except Exception: #pylint:disable=broad-except pipver = 'Unknown or not using pip' + glibc_ver = get_glibc_version_string() + if glibc_ver: + glibc = '\nGNU libc version: ' + glibc_ver + '\n' + elif os.name != 'nt': + glibc = '\nNot using GNU libc. Note that musl libc is not supported.\n' + else: + glibc = '' + return ''' ****************************************************************************** @@ -90,7 +150,7 @@ def unsupported_msg(plat_name): *** https://github.com/Dynatrace/OneAgent-SDK-for-Python#requirements *** ****************************************************************************** Your pip version: {pipver} -Your target platform: {plat} +Your target platform: {plat}{glibc} If you are intentionally building from source, download the OneAgent SDK for C/C++ that corresponds to this Python SDK (v{v}; see table at @@ -98,7 +158,7 @@ def unsupported_msg(plat_name): https://github.com/Dynatrace/OneAgent-SDK-for-C and set the environment variable {env} to the path to the shared library/DLL correponding to the platform you are building for.'''.format( - v=__version__, plat=plat_name, env=CSDK_ENV_NAME, pipver=pipver) + v=__version__, plat=plat_name, env=CSDK_ENV_NAME, pipver=pipver, glibc=glibc) def compilefile(fname, mode='exec'): @@ -151,13 +211,22 @@ def get_dll_input_path(plat_name): if not sdkpath: warn_msg = unsupported_msg(plat_name) distlog.error(warn_msg) - raise ValueError(warn_msg) + distlog.warn( + 'Continuning installation, but resulting package will always be in' + ' no-op mode (no connection to Dynatrace will be possible)') + return None + if path.isfile(sdkpath): return sdkpath + if not path.exists(sdkpath): raise ValueError( '****** Path "{}" in ${} does not exist. ******'.format( sdkpath, CSDK_ENV_NAME)) + + # A folder is specified in skdpath. We can try to find some well-known + # binaries in a folder with one of two well-known structures. + if not dll_info['WIN32'] and 'linux' not in plat_name: raise ValueError( '****** Your platform ({}) is not supported by the ' @@ -226,19 +295,24 @@ def get_outputs(self): return self.__base.get_outputs(self) + [self.get_dll_output_path()] def get_inputs(self): + extra_inputs = [] try: dll_input = get_dll_input_path(self.plat_name) except ValueError: - return self.__base.get_inputs(self) + pass else: - return self.__base.get_inputs(self) + [dll_input] + if dll_input: + extra_inputs.append(dll_input) + + return self.__base.get_inputs(self) + extra_inputs def run(self): src = get_dll_input_path(self.plat_name) dst = self.get_dll_output_path() self.__base.run(self) self.mkpath(path.dirname(dst)) - self.copy_file(src, dst) + if src: + self.copy_file(src, dst) def copy_extensions_to_source(self): self.__base.copy_extensions_to_source(self) @@ -247,8 +321,9 @@ def copy_extensions_to_source(self): src_filename = get_dll_input_path(self.plat_name) package = 'oneagent._impl.native' package_dir = build_py.get_package_dir(package) - dest_filename = path.join(package_dir, path.basename(src_filename)) - self.copy_file(src_filename, dest_filename) + if src_filename: + dest_filename = path.join(package_dir, path.basename(src_filename)) + self.copy_file(src_filename, dest_filename) diff --git a/src/oneagent/__init__.py b/src/oneagent/__init__.py index 0b5a0a3..9c36a68 100644 --- a/src/oneagent/__init__.py +++ b/src/oneagent/__init__.py @@ -82,15 +82,13 @@ from threading import Lock from oneagent._impl.six.moves import range #pylint:disable=import-error +from oneagent.version import __version__ -from .common import SDKError, SDKInitializationError, ErrorCode +from .common import SDKError, SDKInitializationError, ErrorCode, _ONESDK_INIT_FLAG_FORKABLE from ._impl.native import nativeagent from ._impl.native.nativeagent import try_get_sdk from ._impl.native.sdknulliface import SDKNullInterface - -# See https://www.python.org/dev/peps/pep-0440/ "Version Identification and -# Dependency Specification" -__version__ = '1.2.1' +from ._impl.native.sdkdllinfo import WIN32 logger = logging.getLogger('py_sdk') logger.setLevel(logging.CRITICAL + 1) # Disabled by default @@ -169,7 +167,7 @@ def get_sdk(): return _sdk_instance -def initialize(sdkopts=(), sdklibname=None): +def initialize(sdkopts=(), sdklibname=None, forkable=False): '''Attempts to initialize the SDK with the specified options. Even if initialization fails, a dummy SDK will be available so that SDK @@ -180,8 +178,36 @@ def initialize(sdkopts=(), sdklibname=None): will be ignored (the return value will have the :data:`InitResult.STATUS_ALREADY_INITIALIZED` status code in that case). + When setting the ``forkable`` flag the OneAgent SDK for Python will only be partly + initialized. In this special **parent-initialized** initialization state, only the following + functions can be called: + + * All functions that are valid to call before calling initialize remain valid. + * :meth:`oneagent.sdk.SDK.agent_version_string` works as expected. + * :meth:`oneagent.sdk.SDK.agent_state` will return + :data:`oneagent.common.AgentState.TEMPORARILY_INACTIVE` - but see the note below. + * :meth:`oneagent.sdk.SDK.set_diagnostic_callback` works as expected, the callback will be + carried over to forked child processes. + * It is recommended you call :func:`shutdown` when the original process will not fork any more + children that want to use the SDK. + + After you fork, the child becomes **pre-initialized**: the first call to an SDK function that + needs a **fully initialized** agent will automatically complete the initialization. + + You can still fork another child (e.g. in a double-fork scenario) in the **pre-initialized** + state. However if you fork another child in the **fully initialized** state, it will not be + able to use the SDK - not even if it tries to shut down the SDK and initialize it again. + + .. note:: Calling :meth:`oneagent.sdk.SDK.agent_state` in the **pre-initialized** state will + cause the agent to become **fully initialized**. + + All children forked from a **parent-initialized** process will use the same agent. That agent + will shut down when all child processes and the original **parent-initialized** process have + terminated or called shutdown. Calling :func:`shutdown` in a **pre-initialized** process is + not required otherwise. + :param sdkopts: A sequence of strings of the form - :samp:`{NAME}={VALUE}` that set the given SDK options. Igored in all but + :samp:`{NAME}={VALUE}` that set the given SDK options. Ignored in all but the first :code:`initialize` call. :type sdkopts: ~typing.Iterable[str] :param str sdklibname: The file or directory name of the native C SDK @@ -189,6 +215,7 @@ def initialize(sdkopts=(), sdklibname=None): used. Using a value other than None is only acceptable for debugging. You are responsible for providing a native SDK version that matches the Python SDK version. + :param bool forkable: Use the SDK in 'forkable' mode. :rtype: .InitResult ''' @@ -198,14 +225,14 @@ def initialize(sdkopts=(), sdklibname=None): with _sdk_ref_lk: logger.debug("initialize: ref count = %d", _sdk_ref_count) - result = _try_init_noref(sdkopts, sdklibname) + result = _try_init_noref(sdkopts, sdklibname, forkable) if _sdk_instance is None: _sdk_instance = SDK(try_get_sdk()) _sdk_ref_count += 1 return result -def _try_init_noref(sdkopts=(), sdklibname=None): +def _try_init_noref(sdkopts=(), sdklibname=None, forkable=False): global _should_shutdown #pylint:disable=global-statement sdk = nativeagent.try_get_sdk() @@ -236,7 +263,12 @@ def _try_init_noref(sdkopts=(), sdklibname=None): err, sdk.strerror(err)) - nativeagent.checkresult(sdk, sdk.initialize(), 'onesdk_initialize') + if WIN32 and forkable: + logger.warning('SDK can''t be initialized in forkable mode on Windows and Solaris') + + flags = _ONESDK_INIT_FLAG_FORKABLE if forkable else 0 + + nativeagent.checkresult(sdk, sdk.initialize(flags), 'onesdk_initialize_2') _should_shutdown = True logger.debug("initialize successful") return InitResult( diff --git a/src/oneagent/_impl/native/sdkctypesiface.py b/src/oneagent/_impl/native/sdkctypesiface.py index 31a39c5..7cd96ab 100644 --- a/src/oneagent/_impl/native/sdkctypesiface.py +++ b/src/oneagent/_impl/native/sdkctypesiface.py @@ -27,6 +27,7 @@ from functools import wraps from oneagent import logger +from oneagent.version import min_stub_version, max_stub_version from oneagent._impl import six from oneagent.common import SDKError, SDKInitializationError, ErrorCode @@ -40,9 +41,6 @@ CCSID_UTF16_BE = 1201 CCSID_UTF16_LE = 1203 -min_stub_version = OnesdkStubVersion(1, 4, 1) -max_stub_version = OnesdkStubVersion(2, 0, 0) - bool_t = ctypes.c_int32 result_t = ctypes.c_uint32 if WIN32 else ctypes.c_int32 xchar_p = ctypes.c_wchar_p if WIN32 else ctypes.c_char_p @@ -268,8 +266,8 @@ def __init__(self, libname): # Init/Shutdown initfn( - 'initialize', - (), + 'initialize_2', + (ctypes.c_uint32,), result_t, public=False) initfn( @@ -549,8 +547,8 @@ def _init_messaging(self): None).__doc__ = '(tracer_handle, correlation_id)' - def initialize(self): - result = self._initialize() + def initialize(self, init_flags=0): + result = self._initialize_2(init_flags) self._agent_version = ufromxstr(self._agent_get_version_string()) \ + '/' + self._agent_sdk_version @@ -736,9 +734,9 @@ def loadsdk(libname=None): try: import pkg_resources libname = pkg_resources.resource_filename(__name__, dll_name()) - except ImportError: + except Exception: #pylint:disable=broad-except logger.warning( - 'Could not import pkg_resources module:' + 'Could not get native SDK path via pkg_resources:' ' loading native SDK library might fail', exc_info=sys.exc_info()) thisdir = path.dirname(path.abspath(__file__)) @@ -748,5 +746,8 @@ def loadsdk(libname=None): logger.info('Loading native SDK library "%s".', libname) return SDKDllInterface(libname) except OSError as e: - msg = 'Failed loading SDK stub from ' + libname + ': ' + str(e) + msg = 'Failed loading SDK stub from ' + libname + ': "' + str(e) + \ + '". Check your installation of the oneagent-sdk Python package,' + \ + ' e.g., try running ' + \ + '`pip install --verbose --force-reinstall oneagent-sdk`.' six.raise_from(SDKError(ErrorCode.LOAD_AGENT, msg), e) diff --git a/src/oneagent/_impl/native/sdknulliface.py b/src/oneagent/_impl/native/sdknulliface.py index de87238..8b1898f 100644 --- a/src/oneagent/_impl/native/sdknulliface.py +++ b/src/oneagent/_impl/native/sdknulliface.py @@ -64,7 +64,7 @@ def agent_found(self): def agent_is_compatible(self): return False - def initialize(self): + def initialize(self, init_flags=0): return ErrorCode.AGENT_NOT_ACTIVE def shutdown(self): diff --git a/src/oneagent/_impl/native/sdkversion.py b/src/oneagent/_impl/native/sdkversion.py index f6816e3..db1270b 100644 --- a/src/oneagent/_impl/native/sdkversion.py +++ b/src/oneagent/_impl/native/sdkversion.py @@ -29,10 +29,10 @@ def __gt__(self, version): if self.major > version.major: return True - elif self.major == version.major: + if self.major == version.major: if self.minor > version.minor: return True - elif self.minor == version.minor: + if self.minor == version.minor: return self.patch > version.patch return False diff --git a/src/oneagent/common.py b/src/oneagent/common.py index f13cb50..5527611 100644 --- a/src/oneagent/common.py +++ b/src/oneagent/common.py @@ -26,9 +26,25 @@ if _DEBUG_LEAKS: import traceback -#: The Dynatrace Tag request header name which is used to transport the tag between agents. +#: The Dynatrace Tag request header name which is used to transport the tag between agents +#: (as a string tag). DYNATRACE_HTTP_HEADER_NAME = 'X-dynaTrace' +#: The Dynatrace Tag messaging property name which is is used to transport the tag between agents +#: (as a byte tag). +#: +#: .. versionadded:: 1.3 +DYNATRACE_MESSAGE_PROPERTY_NAME = "dtdTraceTagInfo" + +#: DEPRECATED alias for :data:`DYNATRACE_MESSAGE_PROPERTY_NAME` +#: +#: .. deprecated:: 1.3 +DYNATRACE_MESSAGE_PROPERTYNAME = DYNATRACE_MESSAGE_PROPERTY_NAME + + +#: Allow SDK to be used in forked child processes. +_ONESDK_INIT_FLAG_FORKABLE = 1 + class _Uninstantiable(object): '''Classes deriving from this class cannot be instantiated.''' diff --git a/src/oneagent/sdk/__init__.py b/src/oneagent/sdk/__init__.py index 9328ae2..b617da9 100644 --- a/src/oneagent/sdk/__init__.py +++ b/src/oneagent/sdk/__init__.py @@ -34,7 +34,12 @@ each channel type. ''' -from collections import namedtuple, Mapping +from collections import namedtuple + +try: + from collections.abc import Mapping +except (ImportError, NameError): + from collections import Mapping from oneagent._impl import six from oneagent._impl.native.nativeagent import try_get_sdk as _try_get_nsdk diff --git a/src/oneagent/sdk/tracers.py b/src/oneagent/sdk/tracers.py index 2f63678..3f49381 100644 --- a/src/oneagent/sdk/tracers.py +++ b/src/oneagent/sdk/tracers.py @@ -225,12 +225,10 @@ def set_round_trip_count(self, round_trip_count): class IncomingRemoteCallTracer(Tracer): '''Traces an incoming remote call. See :meth:`oneagent.sdk.SDK.trace_incoming_remote_call`.''' - pass class OutgoingRemoteCallTracer(Tracer, OutgoingTaggable): '''Traces an outgoing remote call. See :meth:`oneagent.sdk.SDK.trace_outgoing_remote_call`.''' - pass def _make_add_kvs_fn(fnname): add_kv_name = fnname @@ -367,7 +365,6 @@ class InProcessLinkTracer(Tracer): .. versionadded:: 1.1.0 ''' - pass class OutgoingMessageTracer(Tracer, OutgoingTaggable): '''Tracer for outgoing messages. @@ -408,7 +405,6 @@ class IncomingMessageReceiveTracer(Tracer): .. versionadded:: 1.2.0 ''' - pass class IncomingMessageProcessTracer(Tracer): '''Tracer for processing incoming messages. @@ -442,4 +438,3 @@ class CustomServiceTracer(Tracer): .. versionadded:: 1.2.0 ''' - pass diff --git a/src/oneagent/version.py b/src/oneagent/version.py new file mode 100644 index 0000000..62738fd --- /dev/null +++ b/src/oneagent/version.py @@ -0,0 +1,32 @@ +# +# Copyright 2019 Dynatrace LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +'''Version defines related to the OneAgent SDK for Python. +''' + +from oneagent._impl.native.sdkversion import OnesdkStubVersion + +# That's the OneAgent SDK for Python version. +# See https://www.python.org/dev/peps/pep-0440/ "Version Identification and +# Dependency Specification" +__version__ = '1.3.0' + +# Define the OneAgent SDK for C/C++ version which should be shipped with this +# Python SDK version. +shipped_c_stub_version = '1.5.1' + +# Below are the minimum and maximum required/supported OneAgent SDK for C/C++ versions. +min_stub_version = OnesdkStubVersion(1, 5, 1) +max_stub_version = OnesdkStubVersion(2, 0, 0) diff --git a/test-util-src/sdkmockiface.py b/test-util-src/sdkmockiface.py index 665e6e7..9752e13 100644 --- a/test-util-src/sdkmockiface.py +++ b/test-util-src/sdkmockiface.py @@ -403,7 +403,7 @@ def agent_is_compatible(self): return True @_checkstate(AgentState.NOT_INITIALIZED) - def initialize(self): + def initialize(self, init_flags=0): self._state = AgentState.ACTIVE return ErrorCode.SUCCESS diff --git a/test/test_load_old_agent.py b/test/test_load_old_agent.py index a44c9d5..d5194b8 100644 --- a/test/test_load_old_agent.py +++ b/test/test_load_old_agent.py @@ -15,16 +15,20 @@ import os +import pytest + import oneagent +from oneagent.version import shipped_c_stub_version from oneagent import InitResult from oneagent.common import AgentState +@pytest.mark.dependsnative def test_load_old_agent(): saved_path = os.environ.get('DT_AGENTLIBRARY', '') try: + assert os.environ['DT_OLDAGENTLIBRARY'] is not None + assert os.environ['DT_OLDAGENTLIBRARY'] != '' os.environ['DT_AGENTLIBRARY'] = os.environ.get('DT_OLDAGENTLIBRARY', '') - assert os.environ['DT_AGENTLIBRARY'] is not None - assert os.environ['DT_AGENTLIBRARY'] != '' sdk_options = oneagent.sdkopts_from_commandline(remove=True) init_result = oneagent.initialize(sdk_options) @@ -36,7 +40,8 @@ def test_load_old_agent(): assert sdk.agent_state == AgentState.NOT_INITIALIZED assert sdk.agent_found assert not sdk.agent_is_compatible - assert sdk.agent_version_string == '1.141.112.20180322-095721/1.4.1' + + assert sdk.agent_version_string == '1.141.246.20180604-140607/' + shipped_c_stub_version finally: oneagent.shutdown() os.environ['DT_AGENTLIBRARY'] = saved_path diff --git a/test/test_public_sdk.py b/test/test_public_sdk.py index f74c281..cf783ac 100644 --- a/test/test_public_sdk.py +++ b/test/test_public_sdk.py @@ -195,69 +195,70 @@ def chk_all(vals, chk): for val in vals: exec_chk(chk, val) +def check_remote_node(node): + assert node.vals[:3] == ( + 'dummyPyMethod', 'DummyPyService', + 'dupypr://localhost/dummyEndpoint') + assert node.protocol_name == 'DUMMY_PY_PROTOCOL' + +def check_root(root): + assert type(root) is sdkmockiface.InRemoteCallHandle + assert root.vals == ('main', 'main', 'main') + + chk_seq( + root.children, + [check_remote_child_ok] * 2 + [check_remote_child_err]) + +def check_remote_child(child): + link, node = child + assert link == sdkmockiface.TracerHandle.LINK_CHILD + assert type(node) is sdkmockiface.OutRemoteCallHandle + check_remote_node(node) + assert node.vals[3:] == ( + onesdk.ChannelType.IN_PROCESS, 'localhost') + +def check_remote_child_err(child): + check_remote_child(child) + node = child[1] + assert node.err_info == ( + RTERR_QNAME, 'Remote call failed on the server side.') + + def check_linked_remote_thread_err(rmchild): + rmlink, rmnode = rmchild + assert rmlink == sdkmockiface.TracerHandle.LINK_TAG + assert type(rmnode) is sdkmockiface.InRemoteCallHandle + check_remote_node(rmnode) + assert not rmnode.children + + chk_seq(node.children, [check_linked_remote_thread_err]) + +def check_remote_child_ok(child): + check_remote_child(child) + node = child[1] + assert node.err_info is None + + def check_linked_remote_thread(rmchild): + rmlink, rmnode = rmchild + assert rmlink == sdkmockiface.TracerHandle.LINK_TAG + assert type(rmnode) is sdkmockiface.InRemoteCallHandle + check_remote_node(rmnode) + assert rmnode.children + + def chk_dbcall(dbchild): + dblnk, dbnode = dbchild + assert dblnk == sdkmockiface.TracerHandle.LINK_CHILD + assert type(dbnode) is sdkmockiface.DbRequestHandle + + chk_all(rmnode.children, chk_dbcall) + + chk_seq(node.children, [check_linked_remote_thread]) + def test_public_sdk_sample(native_sdk): nativeagent._force_initialize(native_sdk) #pylint:disable=protected-access from . import onesdksamplepy onesdksamplepy.main() assert_resolve_all(native_sdk) - def check_remote_node(node): - assert node.vals[:3] == ( - 'dummyPyMethod', 'DummyPyService', - 'dupypr://localhost/dummyEndpoint') - assert node.protocol_name == 'DUMMY_PY_PROTOCOL' - - def check_root(root): - assert type(root) is sdkmockiface.InRemoteCallHandle - assert root.vals == ('main', 'main', 'main') - - def check_remote_child(child): - link, node = child - assert link == sdkmockiface.TracerHandle.LINK_CHILD - assert type(node) is sdkmockiface.OutRemoteCallHandle - check_remote_node(node) - assert node.vals[3:] == ( - onesdk.ChannelType.IN_PROCESS, 'localhost') - - def check_remote_child_err(child): - check_remote_child(child) - node = child[1] - assert node.err_info == ( - RTERR_QNAME, 'Remote call failed on the server side.') - - def check_linked_remote_thread_err(rmchild): - rmlink, rmnode = rmchild - assert rmlink == sdkmockiface.TracerHandle.LINK_TAG - assert type(rmnode) is sdkmockiface.InRemoteCallHandle - check_remote_node(rmnode) - assert not rmnode.children - - chk_seq(node.children, [check_linked_remote_thread_err]) - - def check_remote_child_ok(child): - check_remote_child(child) - node = child[1] - assert node.err_info is None - - def check_linked_remote_thread(rmchild): - rmlink, rmnode = rmchild - assert rmlink == sdkmockiface.TracerHandle.LINK_TAG - assert type(rmnode) is sdkmockiface.InRemoteCallHandle - check_remote_node(rmnode) - assert rmnode.children - - def chk_dbcall(dbchild): - dblnk, dbnode = dbchild - assert dblnk == sdkmockiface.TracerHandle.LINK_CHILD - assert type(dbnode) is sdkmockiface.DbRequestHandle - - chk_all(rmnode.children, chk_dbcall) - - chk_seq(node.children, [check_linked_remote_thread]) - - chk_seq( - root.children, - [check_remote_child_ok] * 2 + [check_remote_child_err]) def check_is_linked(root): assert root.linked_parent diff --git a/tox.ini b/tox.ini index 2384fa1..a5d47f0 100644 --- a/tox.ini +++ b/tox.ini @@ -1,20 +1,24 @@ [tox] -envlist = {py27,py36}-{lint,test}, docs +envlist = lint, py27-lint, py{27,34,35,36,37,37,39}-test, docs +skip_missing_interpreters=true [testenv] setenv = test: PYTHONDONTWRITEBYTECODE = 1 - PYTHONPATH = test-util-src{:}samples/basic-sdk-sample + PYTHONPATH = {toxinidir}/test-util-src{:}{toxinidir}/samples/basic-sdk-sample passenv = DT_AGENTLIBRARY DT_OLDAGENTLIBRARY +changedir = + test: test commands = - test: pytest test {posargs} + test: pytest . -p testconfig {posargs} lint: pylint oneagent lint: pylint ./setup.py lint: pylint samples/basic-sdk-sample/setup.py samples/basic-sdk-sample/basic_sdk_sample.py - lint-py36: pylint test --disable=redefined-outer-name,unidiomatic-typecheck + lint-!py27-!pypy: pylint test --disable=redefined-outer-name,unidiomatic-typecheck lint-py27,lint-pypy: pylint test --disable=redefined-outer-name,unidiomatic-typecheck --ignore-patterns=.*py3.* deps = - lint: pylint==1.9.* + py27-lint: pylint~=1.9 + lint-!py27-!pypy: pylint~=2.4 pytest mock From 44c2748564a679a9a5f6a6d409ca52e5caaedcd4 Mon Sep 17 00:00:00 2001 From: Wolfgang Ziegler Date: Wed, 13 Nov 2019 16:22:30 +0100 Subject: [PATCH 2/2] Add suggested changes and fix typo in README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b154d5e..3a869c0 100644 --- a/README.md +++ b/README.md @@ -585,11 +585,11 @@ Known gotchas: Make sure that the `pip install` or equivalent succeeded (see [here](#installation)). Also make sure you use the `pip` corresponding to your `python` (if in doubt, use `python -m pip` instead of `pip` for installing). -* Output ending in a message similar to `InitResult=InitResult(status=-2, error=SDKError(-1342308345, 'Failed loading SDK stub from .../site-packages/oneagent/_impl/native/libonesdk_shared.so: "/.../libonesdk_shared.so: cannot open shared object file: No such file or directory". Check your installation of the oneagent-sdk Python package, e.g., try running `pip install --verbose --force-reinstall oneagent-sdk`.'))`. +* Output ending in a message like `InitResult=InitResult(status=-2, error=SDKError(-1342308345, 'Failed loading SDK stub from .../site-packages/oneagent/_impl/native/libonesdk_shared.so: "/.../libonesdk_shared.so: cannot open shared object file: No such file or directory". Check your installation of the oneagent-sdk Python package, e.g., try running `pip install --verbose --force-reinstall oneagent-sdk`.'))`. Follow the advice of the message and run `python -m pip install --verbose --force-reinstall oneagent-sdk` (or the equivalent pip invocation with the `--verbose` and `--force-reinstall` flags). - It is likely that you will now see another message similar to + It is likely that you will now see another message like ****************************************************************************** *** You are trying to build the Python SDK from source. *** @@ -610,7 +610,7 @@ Known gotchas: version corresponding to your OneAgent SDK for Python as listed in [the table in Requirements](#requirements). Then set the `DT_PYSDK_CSDK_PATH` environment variable to the `.so`/`.dll` file corresponding to your platform in the `lib` subdirectory of the C SDK and retry the installation (e.g. in a bash shell, use `export DT_PYSDK_CSDK_PATH=path/to/onesdk_shared.so`). If there is no corresponding - directory, your platfom is not supported. Otherwise, regardless if it works with that method or not, please report an issue as described + directory, your platform is not supported. Otherwise, regardless if it works with that method or not, please report an issue as described in [Let us help you](#let-us-help-you).