Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the pytest-randomly plugin to run tests in random order #2936

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

seisman
Copy link
Member

@seisman seisman commented Jan 1, 2024

Description of proposed changes

Fixes #

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
  • If adding new functionality, add an example to docstrings or tutorials.
  • Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

  • /format: automatically format and lint the code
  • /test-gmt-dev: run full tests on the latest GMT development version

@seisman seisman changed the title Run tests randomly Use the pytest-randomly plugin to run tests in randonly order Jan 1, 2024
@seisman seisman changed the title Use the pytest-randomly plugin to run tests in randonly order Use the pytest-randomly plugin to run tests in randomly order Jan 1, 2024
@seisman seisman changed the title Use the pytest-randomly plugin to run tests in randomly order Use the pytest-randomly plugin to run tests in random order Jan 1, 2024
@seisman
Copy link
Member Author

seisman commented Jan 1, 2024

@weiji14 Randomly running the tests finds a minor bug in PR #2774. Some tests like below fail:

______________________ test_virtualfile_from_matrix_slice ______________________

dtypes = ['int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', ...]

    def test_virtualfile_from_matrix_slice(dtypes):
        """
        Test transforming a slice of a larger array to virtual file dataset.
        """
        shape = (10, 6)
        for dtype in dtypes:
>           full_data = np.arange(shape[0] * shape[1], dtype=dtype).reshape(shape)
E           TypeError: data type 'int8[pyarrow]' not understood

The failures are because dtypes is a list and is mutable, so changes in dtypes affect all later tests:

if find_spec("pyarrow") is not None:
dtypes.extend([f"{dtype}[pyarrow]" for dtype in dtypes])

@weiji14
Copy link
Member

weiji14 commented Jan 2, 2024

The failures are because dtypes is a list and is mutable, so changes in dtypes affect all later tests:

Good spotting, I opened a PR at #2941 to fix this.

@weiji14 weiji14 closed this in #2941 Jan 2, 2024
@seisman seisman reopened this Jan 2, 2024
@seisman
Copy link
Member Author

seisman commented Oct 10, 2024

Note to myself:

This is the command to reproduce the errors stably

$ pytest --randomly-seed=2418297293 pygmt/tests/test_grdimage_image.py pygmt/tests/test_grdclip.py
pygmt/tests/test_grdclip.py::test_grdclip_no_outgrid PASSED                                                                                                                                 [  9%]
pygmt/tests/test_grdclip.py::test_grdclip_outgrid PASSED                                                                                                                                    [ 18%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[uint32] PASSED                                                                                          [ 27%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int16] PASSED                                                                                           [ 36%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[float64] PASSED                                                                                         [ 45%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[uint16] PASSED                                                                                          [ 54%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image FAILED                                                                                                                              [ 63%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[float32] PASSED                                                                                         [ 72%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int8] PASSED                                                                                            [ 81%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray_unsupported_dtype[int32] PASSED                                                                                           [ 90%]
pygmt/tests/test_grdimage_image.py::test_grdimage_image_dataarray FAILED

If adding verbose="d" to the test_grdimage_image test, we will see debugging messages like:

...
pygmt-session [DEBUG]: Input remote grid modified to have registration: @earth_day_01d_p
pygmt-session [DEBUG]: Revised remote file name to @earth_day_01d_p
pygmt-session [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
pygmt-session [DEBUG]: Use PS filename /Users/seisman/.gmt/sessions/gmt_session.6186/gmt_5.ps-
pygmt-session [DEBUG]: gmtinit_get_current_panel: No current panel selected so not in subplot mode
pygmt-session [DEBUG]: Modern mode: Added -R to options since history is available [-53/-49/-19/-16].
pygmt-session [DEBUG]: Modern: Adding -JQ15c+du to options since there is no history available.
pygmt-session [DEBUG]: Revised options: @earth_day_01d_p -Vd -R -JQ15c+du
grdimage [DEBUG]: History: Process -R
grdimage [DEBUG]: History: Process -JQ15c+du
grdimage [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
grdimage [DEBUG]: Got regular w/e/s/n for region (-53/-49/-19/-16)
...

The region (-53/-49/-19/-16) comes from history and is leaked from test_grdclip_no_outgrid.

@seisman
Copy link
Member Author

seisman commented Oct 10, 2024

import pytest
from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief


def test_grdclip_no_outgrid():
    """
    Test the below and above parameters for grdclip with no set outgrid.
    """
    grid = load_static_earth_relief()
    grdclip(
        grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
    )


@pytest.mark.mpl_image_compare
def test_grdimage_image():
    """
    Plot a 3-band RGB image using file input.
    """
    fig = Figure()
    fig.grdimage(grid="@earth_day_01d", verbose="d")
    return fig

Save the content above into a test file pygmt/test/test_random.py and run pytest pygmt/test/test_random.py to reproduce the failures. The reason for the failures is because we're using a single GMT CLI session, so the region setting from test_grdclip_no_outgrid is saved in GMT session history and is further used in test_grdimage_image. If we swap the order of the two tests, then they pass.

Here are two normal scripts to understand the differences.

from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief

grid = load_static_earth_relief()
grdclip(
    grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
)

fig = Figure()
fig.grdimage(grid="@earth_day_01d", verbose="d")
fig.show()
from pygmt import Figure, grdclip
from pygmt.helpers.testing import load_static_earth_relief

fig = Figure()
fig.grdimage(grid="@earth_day_01d", verbose="d")
fig.show()

grid = load_static_earth_relief()
grdclip(
    grid=grid, below=[550, -1000], above=[700, 1000], region=[-53, -49, -19, -16]
)

Related to #1242 but is more complicated than that one.

@seisman seisman self-assigned this Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants