Merge pull request #5281 from nicoddemus/merge-master-into-features

Merge master into features (make CI green again)

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,12 +5,12 @@ Here's a quick checklist that should be present in PRs.
 (please delete this text from the final description, this is just a guideline)
 -->
 
-- [ ] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/master/changelog/README.rst) for details.
 - [ ] Target the `master` branch for bug fixes, documentation updates and trivial changes.
 - [ ] Target the `features` branch for new features and removals/deprecations.
 - [ ] Include documentation when adding new features.
 - [ ] Include new tests or update existing tests when applicable.
 
 Unless your change is trivial or a small documentation fix (e.g.,  a typo or reword of a small section) please:
 
+- [ ] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/master/changelog/README.rst) for details.
 - [ ] Add yourself to `AUTHORS` in alphabetical order;

AUTHORS

@@ -86,6 +86,7 @@ Endre Galaczi
 Eric Hunsberger
 Eric Siegerman
 Erik M. Bray
+Evan Kepner
 Fabien Zarifian
 Fabio Zadrozny
 Feng Ma

azure-pipelines.yml

@@ -5,7 +5,6 @@ trigger:
 variables:
   PYTEST_ADDOPTS: "--junitxml=build/test-results/$(tox.env).xml -vv"
   python.needs_vc: False
-  python.exe: "python"
   COVERAGE_FILE: "$(Build.Repository.LocalPath)/.coverage"
   COVERAGE_PROCESS_START: "$(Build.Repository.LocalPath)/.coveragerc"
   PYTEST_COVERAGE: '0'
@@ -42,15 +41,13 @@ jobs:
         #   Also seen with py27-nobyte (using xdist), and py27-xdist.
         #   But no exception with py27-pexpect,py27-twisted,py27-numpy.
         PYTEST_COVERAGE: '1'
-      pypy:
-        python.version: 'pypy2'
-        tox.env: 'pypy'
-        python.exe: 'pypy'
-      # NOTE: pypy3 fails to install pip currently due to an internal error.
+      # -- pypy2 and pypy3 are disabled for now: #5279 --
+      # pypy:
+      #   python.version: 'pypy2'
+      #   tox.env: 'pypy'
       # pypy3:
       #   python.version: 'pypy3'
       #   tox.env: 'pypy3'
-      #   python.exe: 'pypy3'
       py34-xdist:
         python.version: '3.4'
         tox.env: 'py34-xdist'
@@ -94,12 +91,12 @@ jobs:
     condition: eq(variables['python.needs_vc'], True)
     displayName: 'Install VC for py27'
 
-  - script: $(python.exe) -m pip install --upgrade pip && $(python.exe) -m pip install tox
+  - script: python -m pip install --upgrade pip && python -m pip install tox
     displayName: 'Install tox'
 
   - script: |
       call scripts/setup-coverage-vars.bat || goto :eof
-      $(python.exe) -m tox -e $(tox.env)
+      python -m tox -e $(tox.env)
     displayName: 'Run tests'
 
   - task: PublishTestResults@2
@@ -112,6 +109,5 @@ jobs:
     displayName: 'Report and upload coverage'
     condition: eq(variables['PYTEST_COVERAGE'], '1')
     env:
-      PYTHON: $(python.exe)
       CODECOV_TOKEN: $(CODECOV_TOKEN)
       PYTEST_CODECOV_NAME: $(tox.env)

changelog/5250.doc.rst

@@ -0,0 +1 @@
+Expand docs on use of ``setenv`` and ``delenv`` with ``monkeypatch``.

changelog/5256.bugfix.rst

@@ -0,0 +1 @@
+Handle internal error due to a lone surrogate unicode character not being representable in Jython.

changelog/5257.bugfix.rst

@@ -0,0 +1 @@
+Ensure that ``sys.stdout.mode`` does not include ``'b'`` as it is a text stream.

doc/en/example/markers.rst

@@ -4,7 +4,9 @@
 Working with custom markers
 =================================================
 
-Here are some example using the :ref:`mark` mechanism.
+Here are some examples using the :ref:`mark` mechanism.
+
+.. _`mark run`:
 
 Marking test functions and selecting them for a run
 ----------------------------------------------------

doc/en/historical-notes.rst

@@ -4,6 +4,117 @@ Historical Notes
 This page lists features or behavior from previous versions of pytest which have changed over the years. They are
 kept here as a historical note so users looking at old code can find documentation related to them.
 
+
+.. _marker-revamp:
+
+Marker revamp and iteration
+---------------------------
+
+.. versionchanged:: 3.6
+
+pytest's marker implementation traditionally worked by simply updating the ``__dict__`` attribute of functions to cumulatively add markers. As a result, markers would unintentionally be passed along class hierarchies in surprising ways. Further, the API for retrieving them was inconsistent, as markers from parameterization would be stored differently than markers applied using the ``@pytest.mark`` decorator and markers added via ``node.add_marker``.
+
+This state of things made it technically next to impossible to use data from markers correctly without having a deep understanding of the internals, leading to subtle and hard to understand bugs in more advanced usages.
+
+Depending on how a marker got declared/changed one would get either a ``MarkerInfo`` which might contain markers from sibling classes,
+``MarkDecorators`` when marks came from parameterization or from a ``node.add_marker`` call, discarding prior marks. Also ``MarkerInfo`` acts like a single mark, when it in fact represents a merged view on multiple marks with the same name.
+
+On top of that markers were not accessible in the same way for modules, classes, and functions/methods.
+In fact, markers were only accessible in functions, even if they were declared on classes/modules.
+
+A new API to access markers has been introduced in pytest 3.6 in order to solve the problems with
+the initial design, providing the :func:`_pytest.nodes.Node.iter_markers` method to iterate over
+markers in a consistent manner and reworking the internals, which solved a great deal of problems
+with the initial design.
+
+
+.. _update marker code:
+
+Updating code
+~~~~~~~~~~~~~
+
+The old ``Node.get_marker(name)`` function is considered deprecated because it returns an internal ``MarkerInfo`` object
+which contains the merged name, ``*args`` and ``**kwargs`` of all the markers which apply to that node.
+
+In general there are two scenarios on how markers should be handled:
+
+1. Marks overwrite each other. Order matters but you only want to think of your mark as a single item. E.g.
+``log_level('info')`` at a module level can be overwritten by ``log_level('debug')`` for a specific test.
+
+    In this case, use ``Node.get_closest_marker(name)``:
+
+    .. code-block:: python
+
+        # replace this:
+        marker = item.get_marker("log_level")
+        if marker:
+            level = marker.args[0]
+
+        # by this:
+        marker = item.get_closest_marker("log_level")
+        if marker:
+            level = marker.args[0]
+
+2. Marks compose in an additive manner. E.g. ``skipif(condition)`` marks mean you just want to evaluate all of them,
+order doesn't even matter. You probably want to think of your marks as a set here.
+
+   In this case iterate over each mark and handle their ``*args`` and ``**kwargs`` individually.
+
+   .. code-block:: python
+
+        # replace this
+        skipif = item.get_marker("skipif")
+        if skipif:
+            for condition in skipif.args:
+                # eval condition
+                ...
+
+        # by this:
+        for skipif in item.iter_markers("skipif"):
+            condition = skipif.args[0]
+            # eval condition
+
+
+If you are unsure or have any questions, please consider opening
+`an issue <https://github.com/pytest-dev/pytest/issues>`_.
+
+Related issues
+~~~~~~~~~~~~~~
+
+Here is a non-exhaustive list of issues fixed by the new implementation:
+
+* Marks don't pick up nested classes (`#199 <https://github.com/pytest-dev/pytest/issues/199>`_).
+
+* Markers stain on all related classes (`#568 <https://github.com/pytest-dev/pytest/issues/568>`_).
+
+* Combining marks - args and kwargs calculation (`#2897 <https://github.com/pytest-dev/pytest/issues/2897>`_).
+
+* ``request.node.get_marker('name')`` returns ``None`` for markers applied in classes (`#902 <https://github.com/pytest-dev/pytest/issues/902>`_).
+
+* Marks applied in parametrize are stored as markdecorator (`#2400 <https://github.com/pytest-dev/pytest/issues/2400>`_).
+
+* Fix marker interaction in a backward incompatible way (`#1670 <https://github.com/pytest-dev/pytest/issues/1670>`_).
+
+* Refactor marks to get rid of the current "marks transfer" mechanism (`#2363 <https://github.com/pytest-dev/pytest/issues/2363>`_).
+
+* Introduce FunctionDefinition node, use it in generate_tests (`#2522 <https://github.com/pytest-dev/pytest/issues/2522>`_).
+
+* Remove named marker attributes and collect markers in items (`#891 <https://github.com/pytest-dev/pytest/issues/891>`_).
+
+* skipif mark from parametrize hides module level skipif mark (`#1540 <https://github.com/pytest-dev/pytest/issues/1540>`_).
+
+* skipif + parametrize not skipping tests (`#1296 <https://github.com/pytest-dev/pytest/issues/1296>`_).
+
+* Marker transfer incompatible with inheritance (`#535 <https://github.com/pytest-dev/pytest/issues/535>`_).
+
+More details can be found in the `original PR <https://github.com/pytest-dev/pytest/pull/3317>`_.
+
+.. note::
+
+    in a future major relase of pytest we will introduce class based markers,
+    at which point markers will no longer be limited to instances of :py:class:`Mark`.
+
+
 cache plugin integrated into the core
 -------------------------------------
 

doc/en/mark.rst

@@ -15,151 +15,64 @@ some builtin markers, for example:
   to the same test function.
 
 It's easy to create custom markers or to apply markers
-to whole test classes or modules. See :ref:`mark examples` for examples
-which also serve as documentation.
+to whole test classes or modules. Those markers can be used by plugins, and also
+are commonly used to :ref:`select tests <mark run>` on the command-line with the ``-m`` option.
+
+See :ref:`mark examples` for examples which also serve as documentation.
 
 .. note::
 
     Marks can only be applied to tests, having no effect on
     :ref:`fixtures <fixtures>`.
 
 
-.. _unknown-marks:
-
-Raising errors on unknown marks
--------------------------------
+Registering marks
+-----------------
 
-Unknown marks applied with the ``@pytest.mark.name_of_the_mark`` decorator
-will always emit a warning, in order to avoid silently doing something
-surprising due to mis-typed names.  You can disable the warning for custom
-marks by registering them in ``pytest.ini`` like this:
+You can register custom marks in your ``pytest.ini`` file like this:
 
 .. code-block:: ini
 
     [pytest]
     markers =
-        slow
+        slow: marks tests as slow (deselect with '-m "not slow"')
         serial
 
-When the ``--strict-markers`` command-line flag is passed, any unknown marks applied
-with the ``@pytest.mark.name_of_the_mark`` decorator will trigger an error.
-Marks added by pytest or by a plugin instead of the decorator will not trigger
-this error.  To enforce validation of markers, add ``--strict-markers`` to ``addopts``:
-
-.. code-block:: ini
-
-    [pytest]
-    addopts = --strict-markers
-    markers =
-        slow
-        serial
-
-Third-party plugins should always :ref:`register their markers <registering-markers>`
-so that they appear in pytest's help text and do not emit warnings.
-
-
-.. _marker-revamp:
-
-Marker revamp and iteration
----------------------------
-
-
-
-pytest's marker implementation traditionally worked by simply updating the ``__dict__`` attribute of functions to cumulatively add markers. As a result, markers would unintentionally be passed along class hierarchies in surprising ways. Further, the API for retrieving them was inconsistent, as markers from parameterization would be stored differently than markers applied using the ``@pytest.mark`` decorator and markers added via ``node.add_marker``.
-
-This state of things made it technically next to impossible to use data from markers correctly without having a deep understanding of the internals, leading to subtle and hard to understand bugs in more advanced usages.
-
-Depending on how a marker got declared/changed one would get either a ``MarkerInfo`` which might contain markers from sibling classes,
-``MarkDecorators`` when marks came from parameterization or from a ``node.add_marker`` call, discarding prior marks. Also ``MarkerInfo`` acts like a single mark, when it in fact represents a merged view on multiple marks with the same name.
-
-On top of that markers were not accessible the same way for modules, classes, and functions/methods.
-In fact, markers were only accessible in functions, even if they were declared on classes/modules.
-
-A new API to access markers has been introduced in pytest 3.6 in order to solve the problems with the initial design, providing :func:`_pytest.nodes.Node.iter_markers` method to iterate over markers in a consistent manner and reworking the internals, which solved great deal of problems with the initial design.
-
-
-.. _update marker code:
-
-Updating code
-~~~~~~~~~~~~~
-
-The old ``Node.get_marker(name)`` function is considered deprecated because it returns an internal ``MarkerInfo`` object
-which contains the merged name, ``*args`` and ``**kwargs`` of all the markers which apply to that node.
-
-In general there are two scenarios on how markers should be handled:
+Note that everything after the ``:`` is an optional description.
 
-1. Marks overwrite each other. Order matters but you only want to think of your mark as a single item. E.g.
-``log_level('info')`` at a module level can be overwritten by ``log_level('debug')`` for a specific test.
+Alternatively, you can register new markers programatically in a
+:ref:`pytest_configure <initialization-hooks>` hook:
 
-    In this case, use ``Node.get_closest_marker(name)``:
+.. code-block:: python
 
-    .. code-block:: python
+    def pytest_configure(config):
+        config.addinivalue_line(
+            "markers", "env(name): mark test to run only on named environment"
+        )
 
-        # replace this:
-        marker = item.get_marker("log_level")
-        if marker:
-            level = marker.args[0]
 
-        # by this:
-        marker = item.get_closest_marker("log_level")
-        if marker:
-            level = marker.args[0]
+Registered marks appear in pytest's help text and do not emit warnings (see the next section). It
+is recommended that third-party plugins always :ref:`register their markers <registering-markers>`.
 
-2. Marks compose in an additive manner. E.g. ``skipif(condition)`` marks mean you just want to evaluate all of them,
-order doesn't even matter. You probably want to think of your marks as a set here.
-
-   In this case iterate over each mark and handle their ``*args`` and ``**kwargs`` individually.
-
-   .. code-block:: python
-
-        # replace this
-        skipif = item.get_marker("skipif")
-        if skipif:
-            for condition in skipif.args:
-                # eval condition
-                ...
-
-        # by this:
-        for skipif in item.iter_markers("skipif"):
-            condition = skipif.args[0]
-            # eval condition
-
-
-If you are unsure or have any questions, please consider opening
-`an issue <https://github.com/pytest-dev/pytest/issues>`_.
-
-Related issues
-~~~~~~~~~~~~~~
-
-Here is a non-exhaustive list of issues fixed by the new implementation:
-
-* Marks don't pick up nested classes (`#199 <https://github.com/pytest-dev/pytest/issues/199>`_).
-
-* Markers stain on all related classes (`#568 <https://github.com/pytest-dev/pytest/issues/568>`_).
-
-* Combining marks - args and kwargs calculation (`#2897 <https://github.com/pytest-dev/pytest/issues/2897>`_).
-
-* ``request.node.get_marker('name')`` returns ``None`` for markers applied in classes (`#902 <https://github.com/pytest-dev/pytest/issues/902>`_).
-
-* Marks applied in parametrize are stored as markdecorator (`#2400 <https://github.com/pytest-dev/pytest/issues/2400>`_).
-
-* Fix marker interaction in a backward incompatible way (`#1670 <https://github.com/pytest-dev/pytest/issues/1670>`_).
-
-* Refactor marks to get rid of the current "marks transfer" mechanism (`#2363 <https://github.com/pytest-dev/pytest/issues/2363>`_).
-
-* Introduce FunctionDefinition node, use it in generate_tests (`#2522 <https://github.com/pytest-dev/pytest/issues/2522>`_).
-
-* Remove named marker attributes and collect markers in items (`#891 <https://github.com/pytest-dev/pytest/issues/891>`_).
-
-* skipif mark from parametrize hides module level skipif mark (`#1540 <https://github.com/pytest-dev/pytest/issues/1540>`_).
+.. _unknown-marks:
 
-* skipif + parametrize not skipping tests (`#1296 <https://github.com/pytest-dev/pytest/issues/1296>`_).
+Raising errors on unknown marks
+-------------------------------
 
-* Marker transfer incompatible with inheritance (`#535 <https://github.com/pytest-dev/pytest/issues/535>`_).
+Unregistered marks applied with the ``@pytest.mark.name_of_the_mark`` decorator
+will always emit a warning in order to avoid silently doing something
+surprising due to mis-typed names. As described in the previous section, you can disable
+the warning for custom marks by registering them in your ``pytest.ini`` file or
+using a custom ``pytest_configure`` hook.
 
-More details can be found in the `original PR <https://github.com/pytest-dev/pytest/pull/3317>`_.
+When the ``--strict-markers`` command-line flag is passed, any unknown marks applied
+with the ``@pytest.mark.name_of_the_mark`` decorator will trigger an error. You can
+enforce this validation in your project by adding ``--strict-markers`` to ``addopts``:
 
-.. note::
+.. code-block:: ini
 
-    in a future major relase of pytest we will introduce class based markers,
-    at which point markers will no longer be limited to instances of :py:class:`Mark`.
+    [pytest]
+    addopts = --strict-markers
+    markers =
+        slow: marks tests as slow (deselect with '-m "not slow"')
+        serial