Setup Continuous Benchmarking workflow with pytest-codspeed

[weiji14]

Description of proposed changes

Measuring the execution speed of tests to track performance of PyGMT functions over time.

Using pytest-codspeed to do the benchmarking, with the help of https://github.com/CodSpeedHQ/action. Decorated test_basemap with @pytest.mark.benchmark to see if the benchmarking works.

Note: Running the benchmarks on Python 3.12 to enable flame graph generation, available with pytest-codspeed>=2.0.0 - see https://docs.codspeed.io/features/trace-generation.

References:

• https://codspeed.io/blog/introducing-codspeed
• https://docs.codspeed.io/benchmarks/python#running-the-benchmarks-in-your-ci

Relates to #2730 (comment), addresses #2910.

Reminders

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst.
• Write detailed docstrings for all functions/methods.
• If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
• If adding new functionality, add an example to docstrings or tutorials.
• Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

• /format: automatically format and lint the code
• /test-gmt-dev: run full tests on the latest GMT development version

[seisman]

So which tests should we benchmark (and should we mark all those tests with @pytest.mark.benchmark in this PR or a follow-up one)?

I think we should focus on benchmarking low-level functions rather than modules' wrappers, since the low-level functions are heavily used in everywhere and most wrappers have very simple and similar code structures.

For example, most plotting modules have very simple code like the one in Figure.basemap

    kwargs = self._preprocess(**kwargs)                                         
    with Session() as lib:                                                      
        lib.call_module(module="basemap", args=build_arg_string(kwargs))

so we just need to benchmark one basemap test (e.g., test_basemap()) and don't need to benchmark other basemap's tests and other plotting methods (e.g., Figure.coast). Of course, there are few exceptions, for example, Figure.meca, Figure.plot, Figure.plot3d, Figure.text are among the complicated wrappers and should be benchmarked.

Similarly, for table-processing and grid-processing functions, benchmarking pygmt.select and pygmt.grdfill should be enough.

[weiji14]

Need to mention the new workflow in:

* `doc/maintenance.md`
* `.github/ISSUE_TEMPLATE/bump_gmt_checklist.md`

Ok, done at c92fcb2 and 7bf09b9

* `.github/ISSUE_TEMPLATE/release_checklist.md` (maybe unnecessary in this file)

Probably not needed in the checklist, but I added a trigger at c8d1965 so that the benchmarks will be run when a release is published.

So which tests should we benchmark (and should we mark all those tests with @pytest.mark.benchmark in this PR or a follow-up one)?

I think we should focus on benchmarking low-level functions rather than modules' wrappers, since the low-level functions are heavily used in everywhere and most wrappers have very simple and similar code structures.

Let me open up a separate issue to discuss this, and also to track which unit tests we should benchmark. Edit: see #2910.

[seisman]

Looks good to me.

[weiji14]

Thanks for reviewing @seisman! I'll merge this now and open a follow-up PR for marking more unit tests with @pytest.mark.benchmark. We can discuss more in #2910 too.

[seisman]

The Tests workflow (https://github.com/GenericMappingTools/pygmt/actions/runs/7319763038/job/19937998567) now has the following warnings because pytest-benchmark is not installed in these workflows:

  /home/runner/work/pygmt/pygmt/pygmt/tests/test_basemap.py:8: PytestUnknownMarkWarning: Unknown pytest.mark.benchmark - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.benchmark

I believe we need to silence these warnings.

[weiji14]

The Tests workflow (https://github.com/GenericMappingTools/pygmt/actions/runs/7319763038/job/19937998567) now has the following warnings because pytest-benchmark is not installed in these workflows:

Ok, opened #2912 for this.

[weiji14]

By the way, can you change the CodSpeed settings to only make PR comments when there is a performance improvement or regression (still don't have permissions)? This should reduce noise like #2907 (comment) when there's no noticeable change (we can still see the report in the 'Checks' tab if needed).

Also, maybe reduce the regression threshold to 5%?

[seisman]

By the way, can you change the CodSpeed settings to only make PR comments when there is a performance improvement or regression (still don't have permissions)? This should reduce noise like #2907 (comment) when there's no noticeable change (we can still see the report in the 'Checks' tab if needed).

Done.