From bf61fd651cdbc373899871945e8615002e0d45c8 Mon Sep 17 00:00:00 2001 From: Tomer Nosrati Date: Tue, 5 Mar 2024 14:29:36 +0200 Subject: [PATCH 1/4] Fixed English typo --- docs/includes/installation.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/includes/installation.txt b/docs/includes/installation.txt index 3f5e5e61..c2385cc9 100644 --- a/docs/includes/installation.txt +++ b/docs/includes/installation.txt @@ -23,7 +23,7 @@ Installing pytest-celery vendors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The plugin detects which vendor dependencies are installed in the test environment to configure -the default configurations automatically. This means that just by installing the matching dependencies, +the default configurations automatically. This means that by just installing the matching dependencies, the plugin will allow extending the default configurations, up to the supported built-in :ref:`vendors`. .. warning:: From abd7bf6a17eb967b1419eff1e01102cc4baf9da3 Mon Sep 17 00:00:00 2001 From: Tomer Nosrati Date: Wed, 6 Mar 2024 13:56:05 +0200 Subject: [PATCH 2/4] Renamed _utils_module -> _utils-module --- docs/userguide/examples/myutils.rst | 2 +- docs/userguide/utils-module.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/userguide/examples/myutils.rst b/docs/userguide/examples/myutils.rst index ae588eb6..77178dbe 100644 --- a/docs/userguide/examples/myutils.rst +++ b/docs/userguide/examples/myutils.rst @@ -14,7 +14,7 @@ Description =========== -This example project shows :ref:`utils_module` to add custom APIs to the ``utils.py`` module +This example project shows :ref:`utils-module` to add custom APIs to the ``utils.py`` module that is being injected into the worker container. Breakdown diff --git a/docs/userguide/utils-module.rst b/docs/userguide/utils-module.rst index b450b276..3791114d 100644 --- a/docs/userguide/utils-module.rst +++ b/docs/userguide/utils-module.rst @@ -1,4 +1,4 @@ -.. _utils_module: +.. _utils-module: ========================================== How to inject your own utility functions From 527af48db2509b74b9a0831739f0c459246a8e12 Mon Sep 17 00:00:00 2001 From: Tomer Nosrati Date: Wed, 6 Mar 2024 13:56:36 +0200 Subject: [PATCH 3/4] New Doc: docs/userguide/advanced-installation.rst --- docs/getting-started/introduction.rst | 4 ++ docs/userguide/advanced-installation.rst | 79 ++++++++++++++++++++++++ docs/userguide/index.rst | 1 + src/pytest_celery/defaults.py | 10 ++- 4 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 docs/userguide/advanced-installation.rst diff --git a/docs/getting-started/introduction.rst b/docs/getting-started/introduction.rst index 0731dfc7..03d10059 100644 --- a/docs/getting-started/introduction.rst +++ b/docs/getting-started/introduction.rst @@ -257,3 +257,7 @@ Quick Jump .. _installation: .. include:: ../includes/installation.txt + +.. tip:: + + See :ref:`advanced-installation` for more advanced installation options. diff --git a/docs/userguide/advanced-installation.rst b/docs/userguide/advanced-installation.rst new file mode 100644 index 00000000..15232149 --- /dev/null +++ b/docs/userguide/advanced-installation.rst @@ -0,0 +1,79 @@ +.. _advanced-installation: + +============================= + Advanced Installation Guide +============================= + +:Release: |version| +:Date: |today| + +The pytest-celery plugin uses the environment dependencies to install and configure the default setup matrix. +The available dependencies are the feature flags for the available :ref:`vendors`. + +This guide will explain how to install the plugin, how to use the dependencies feature flags and how to +fit the configurations to your needs. + +We will start by reviewing the standard installation instructions. + +.. contents:: + :local: + :depth: 2 + +.. include:: ../includes/installation.txt + +Advanced Installation +===================== + +.. versionadded:: 1.0.0 + +In this section, we'll improve our understanding of the plugin installation process and how to customize it. + +Feature Flags +~~~~~~~~~~~~~ + +The installed dependencies dynamically configure which vendors are available for the plugin to use. +Each vendor will provide its set of components, such as the broker, backend, and the worker, and the plugin will +automatically add the matching components to the default setup matrix. + +For example, let's assume you want to use the RabbitMQ/Redis combination. + +1. For RabbitMQ we will need :pypi:`kombu`. +2. For Redis we will need :pypi:`redis`. + +To install the plugin with the RabbitMQ/Redis combination, you will need to install the following dependencies: + +.. code-block:: bash + + pip install "pytest-celery[redis]" + +Let's break down the command: + +- The ``pytest-celery`` is the plugin package, it will install the plugin alongside Celery and its dependencies,including **Kombu** (if not installed already). +- The ``[redis]`` is the feature flag for the Redis vendor, it will install the :pypi:`redis` package and configure the plugin to use it which will add the Redis backend and Redis broker components to the default setup matrix. + +Experimental Vendors +~~~~~~~~~~~~~~~~~~~~ + +:ref:`vendors` that are in not stable, will not be added to the default setup matrix. +To use the experimental vendors, you will need to configure the setup matrix manually. + +.. tip:: + + The automatic vendors detection is implemented in :mod:`defaults.py `. + +The ``all`` Extra +~~~~~~~~~~~~~~~~~ + +The ``all`` extra is a special feature flag that will install all available vendors and their dependencies. + +.. code-block:: bash + + pip install "pytest-celery[all]" + +This command will install the plugin and configure it to use all available **stable** vendors in a setup matrix for each test case +that uses the :ref:`test-setup`. + +.. warning:: + + The ``all`` extra will install **all** of the vendors dependencies, including the experimental ones, + to allow manual configuration of the setup matrix. diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index 670c4786..7b0d859b 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -10,6 +10,7 @@ .. toctree:: :maxdepth: 1 + advanced-installation app-conf utils-module tasks diff --git a/src/pytest_celery/defaults.py b/src/pytest_celery/defaults.py index b29cb19a..98f05198 100644 --- a/src/pytest_celery/defaults.py +++ b/src/pytest_celery/defaults.py @@ -1,3 +1,11 @@ +"""Default plugin configurations. + +This module contains the default configurations for the pytest-celery +plugin. It also contains the automatic collections that are used to +parametrize the user's tests according to the components they need to +run against. By default, all possible combinations are used. +""" + # flake8: noqa from __future__ import annotations @@ -41,7 +49,7 @@ # Uses Kombu ALL_CELERY_BROKERS.append(CELERY_RABBITMQ_BROKER) -# Memcached is disabled by default regardless of its availability. +# Memcached is disabled by default regardless of its availability due to its experimental status. if _is_vendor_installed("memcached") and False: ALL_CELERY_BACKENDS.append(CELERY_MEMCACHED_BACKEND) From 58087819eb6506a3bf8480ca8268acd77cca8509 Mon Sep 17 00:00:00 2001 From: Tomer Nosrati Date: Wed, 6 Mar 2024 14:57:12 +0200 Subject: [PATCH 4/4] New Doc: docs/userguide/setup-matrix.rst --- docs/getting-started/first-steps.rst | 2 +- docs/userguide/advanced-installation.rst | 2 +- docs/userguide/index.rst | 5 + docs/userguide/setup-matrix.rst | 130 +++++++++++++++++++++++ 4 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 docs/userguide/setup-matrix.rst diff --git a/docs/getting-started/first-steps.rst b/docs/getting-started/first-steps.rst index 641d82b2..5241ba83 100644 --- a/docs/getting-started/first-steps.rst +++ b/docs/getting-started/first-steps.rst @@ -537,7 +537,7 @@ Let's have a quick recap over what we just learned in this section then. * - **Configurable Components** - Each default component has its ``default_`` fixtures list, which can be used to control or extend the component's functionality. * - **Setup Matrix** - - The :func:`celery_setup ` will generate a matrix of isolated environments for each test case, based on the enabled components and their configurations. + - The :func:`celery_setup ` will generate a :ref:`setup-matrix` of isolated environments for each test case, based on the enabled components and their configurations. .. _built-in-components: diff --git a/docs/userguide/advanced-installation.rst b/docs/userguide/advanced-installation.rst index 15232149..2d2009d9 100644 --- a/docs/userguide/advanced-installation.rst +++ b/docs/userguide/advanced-installation.rst @@ -75,5 +75,5 @@ that uses the :ref:`test-setup`. .. warning:: - The ``all`` extra will install **all** of the vendors dependencies, including the experimental ones, + The ``all`` extra will install **all** of the vendors dependencies, including the experimental vendor's dependencies, to allow manual configuration of the setup matrix. diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index 7b0d859b..12a2a7f2 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -7,10 +7,15 @@ :Release: |version| :Date: |today| +The following sections will enhance your understanding of the pytest-celery plugin and help you to use it effectively. +It is recommended to first review the :ref:`first-steps` and :ref:`next-steps` sections to get comfortable with the basics +of the plugin before diving into the advanced features. + .. toctree:: :maxdepth: 1 advanced-installation + setup-matrix app-conf utils-module tasks diff --git a/docs/userguide/setup-matrix.rst b/docs/userguide/setup-matrix.rst new file mode 100644 index 00000000..685cff5e --- /dev/null +++ b/docs/userguide/setup-matrix.rst @@ -0,0 +1,130 @@ +.. _setup-matrix: + +=================== + Test Setup Matrix +=================== + +:Release: |version| +:Date: |today| + +The plugin simplifies the setup of test environments by utilizing the +`pytest fixtures mechanism `_ to configure +each component of the test setup independently which allows for a high degree of flexibility in matching +specific Celery architectures to each test case. + +In this guide we'll discuss how does the setup matrix work and how to use it. + +.. contents:: + :local: + :depth: 2 + +What is the setup matrix? +========================= + +.. versionadded:: 1.0.0 + +The setup matrix is a combination of all of the possible Celery architectures that can be used +with the available :ref:`vendors` for each test. It is automatically generated by the plugin +to match the installed dependencies when using the default configuration, or it can be manually +configured to set each test case separately. + +The matrix is configured using the :ref:`default-fixtures` of each component and is available +to the test case using the :ref:`test-setup`. + +Common Setups +============= + +.. versionadded:: 1.0.0 + +The following snippets show how to use the setup matrix to configure manual setups, overriding +the default configuration using the built-in :ref:`vendors`. + +It may be used in conftest.py or in the test file itself. + +RabbitMQ Broker and No Backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Use only RabbitMQ as the broker. + +.. code-block:: python + + @pytest.fixture + def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster: + cluster = CeleryBrokerCluster(celery_rabbitmq_broker) + yield cluster + cluster.teardown() + +2. :ref:`Disable the backend `. + +.. code-block:: python + + @pytest.fixture + def celery_backend_cluster(): + return None + +RabbitMQ Broker and Redis Backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Use only RabbitMQ as the broker. + +.. code-block:: python + + @pytest.fixture + def celery_broker_cluster(celery_rabbitmq_broker: RabbitMQTestBroker) -> CeleryBrokerCluster: + cluster = CeleryBrokerCluster(celery_rabbitmq_broker) + yield cluster + cluster.teardown() + +2. Use Redis as the backend. + +.. code-block:: python + + @pytest.fixture + def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster: + cluster = CeleryBackendCluster(celery_redis_backend) + yield cluster + cluster.teardown() + +Redis Broker and No Backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Use only Redis as the broker. + +.. code-block:: python + + @pytest.fixture + def celery_broker_cluster(celery_redis_broker: RedisTestBroker) -> CeleryBrokerCluster: + cluster = CeleryBrokerCluster(celery_redis_broker) + yield cluster + cluster.teardown() + +2. :ref:`Disable the backend `. + +.. code-block:: python + + @pytest.fixture + def celery_backend_cluster(): + return None + +Redis Broker and Redis Backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Use only Redis as the broker. + +.. code-block:: python + + @pytest.fixture + def celery_broker_cluster(celery_redis_broker: RedisTestBroker) -> CeleryBrokerCluster: + cluster = CeleryBrokerCluster(celery_redis_broker) + yield cluster + cluster.teardown() + +2. Use Redis as the backend. + +.. code-block:: python + + @pytest.fixture + def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster: + cluster = CeleryBackendCluster(celery_redis_backend) + yield cluster + cluster.teardown()