feat!: allow specifying the dependency depth resolution through CLI and make dependency resolution off by default

docs/source/pages/cli_usage/command_analyze.rst

@@ -22,7 +22,8 @@ Usage
     usage: ./run_macaron.sh analyze
         [-h] [-sbom SBOM_PATH] [-purl PURL] [-rp REPO_PATH] [-b BRANCH]
         [-d DIGEST] [-pe PROVENANCE_EXPECTATION]
-        [--skip-deps] [-g TEMPLATE_PATH]
+        [--skip-deps] [--deps-depth DEPS_DEPTH] [-g TEMPLATE_PATH]
+        [--python-venv PYTHON_VENV]
 
 -------
 Options
@@ -64,12 +65,20 @@ Options
 
 .. option:: --skip-deps
 
-    Skip automatic dependency analysis.
+    DEPRECATED. Dependency resolution is off by default. This flag does nothing and will be removed in the next release.
+
+.. option:: --deps-depth DEPS_DEPTH
+
+    The depth of the dependency resolution. 0: disable, 1: direct dependencies, inf: all transitive dependencies. (Default: 0)
 
 .. option:: -g TEMPLATE_PATH, --template-path TEMPLATE_PATH
 
     The path to the Jinja2 html template (please make sure to use .html or .j2 extensions).
 
+.. option::  --python-venv PYTHON_VENV
+
+    The path to the Python virtual environment of the target software component.
+
 -----------
 Environment
 -----------

docs/source/pages/supported_technologies/index.rst

@@ -107,6 +107,18 @@ Provenances
        * The provenance should be published on JFrog Artifactory
      - :doc:`page </pages/supported_technologies/jfrog>`
 
+.. _supported_automatic_deps_resolution:
+
+-------------------------------
+Automatic dependency resolution
+-------------------------------
+
+Currently, we support the following type of project for automatic dependency resolution.
+
+* Java Maven
+* Java Gradle
+* Python (with a Python virtual environment created and packages installed using Python3.11, see :ref:`providing Python virtual environment <python-venv-deps>`.)
+
 --------
 See also
 --------

docs/source/pages/tutorials/commit_finder.rst

@@ -50,7 +50,7 @@ To perform an analysis on Arrow, Macaron can be run with the following command:
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow --skip-deps
+    ./run_macaron.sh analyze -rp https://github.com/arrow-py/arrow
 
 However, this will return results based only on the current state of the repository, which as described above, is not what we want to achieve in this tutorial. To perform analyses on other repository states, we need to provide Macaron with the target artifact versions in the form of `PURLs <https://github.com/package-url/purl-spec>`_, or Package URLs, which is a convenient way to encode packages from different ecosystems into the same format.
 
@@ -67,7 +67,7 @@ We will start by running the analysis on the latest version, ``1.3.0``, with the
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --skip-deps
+    ./run_macaron.sh analyze -purl pkg:pypi/[email protected]
 
 The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. Results from the analysis, including checks, are stored in the database found at ``output/macaron.db``  (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will also produce a report in the form of a HTML file.
 
@@ -101,7 +101,7 @@ Now we should run the next analysis, and then open the new report.
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --skip-deps
+    ./run_macaron.sh analyze -purl pkg:pypi/[email protected]
     open output/reports/pypi/arrow/arrow.html
 
 .. _fig_arrow_0.15.0_top:

docs/source/pages/tutorials/detect_malicious_package.rst

@@ -65,7 +65,7 @@ First, we need to run the ``analyze`` command of Macaron to run a number of :ref
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --skip-deps
+  ./run_macaron.sh analyze -purl pkg:pypi/[email protected]
 
 .. note:: By default, Macaron clones the repositories and creates output files under the ``output`` directory. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
 
@@ -172,27 +172,17 @@ Let's assume ``/tmp/.django_venv`` is the virtual environment where ``[email protected]
 
 .. note:: If you want Macaron to analyze the virtual environment directly to identify the dependencies, we require Python 3.11 to be used to install the package. Alternatively, you can generate the SBOM as instructed :ref:`here <python-sbom>` and pass it to Macaron as input.
 
-Run Macaron as follows to analyze ``django`` and its dependencies.
+Run Macaron as follows to analyze ``django`` and its direct dependencies.
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv"
+  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv" --deps-depth=1
 
-
-By default Macaron only checks the direct dependencies. To turn on recursive dependency analysis, add the following to the ``configurations.ini`` file:
-
-.. code-block:: ini
-
-  [dependency.resolver]
-  recursive = True
-
-And pass that to the ``analyze`` command:
+Or alternatively, run Macaron as follows to analyze ``django`` and all its transitive dependencies.
 
 .. code-block:: shell
 
-  ./run_macaron.sh --defaults-path configurations.ini analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv"
-
-To learn more about changing configurations see :ref:`here <change-config>`.
+  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv" --deps-depth=inf
 
 Now we can enforce the policy below to ensure that the ``mcn_detect_malicious_metadata_1`` check always passes on ``django`` and its dependencies, indicating that none of the dependencies have malicious behavior.
 

docs/source/pages/tutorials/exclude_include_checks.rst

@@ -37,7 +37,7 @@ Normally, this is how you would run Macaron:
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze --package-url pkg:maven/io.micronaut/[email protected] --skip-deps
+  ./run_macaron.sh analyze --package-url pkg:maven/io.micronaut/[email protected]
 
 However, there can be checks in Macaron that are not relevant for the ``io.micronaut/micronaut-core`` artifact.
 For example, the ``mcn_provenance_witness_level_one_1`` check (defined in :class:`ProvenanceWitnessL1Check <macaron.slsa_analyzer.checks.provenance_witness_l1_check.ProvenanceWitnessL1Check>`) is not relevant because ``micronaut-projects/micronaut-core`` generates and publishes :term:`SLSA` provenances and no :term:`Witness` provenances. Therefore, we could exclude this check from running by performing the following steps.
@@ -61,7 +61,7 @@ With these two configuration options, all checks except for the excluded ``mcn_p
 
 .. code-block:: shell
 
-  ./run_macaron.sh --defaults-path ./defaults.ini analyze --package-url pkg:maven/io.micronaut/[email protected] --skip-deps
+  ./run_macaron.sh --defaults-path ./defaults.ini analyze --package-url pkg:maven/io.micronaut/[email protected]
 
 This time, the check ``mcn_provenance_witness_level_one_1`` doesn't run. After the ``analyze`` command finishes, we can view the data that Macaron has gathered about the ``micronaut-projects/micronaut-core`` repository at ``v4.3.10`` in an HTML report. Note that the result of the excluded check is not recorded in the Macaron HTML reports, JSON reports, or the database).
 

docs/source/pages/tutorials/generate_verification_summary_attestation.rst

@@ -47,8 +47,7 @@ In order to verify the artifact with Macaron, you can follow the following steps
   ./run_macaron.sh analyze \
         --package-url pkg:maven/io.micronaut.openapi/[email protected]?type=jar \
         --provenance-file multiple.intoto.jsonl \
-        --provenance-expectation expectation.cue \
-        --skip-deps
+        --provenance-expectation expectation.cue
 
 .. note::
 

docs/source/pages/tutorials/npm_provenance.rst

@@ -42,7 +42,7 @@ To perform an analysis on the latest version of semver (when this tutorial was w
 
 .. code-block:: shell
 
-    ./run_macaron.sh analyze -purl pkg:npm/[email protected] --skip-deps
+    ./run_macaron.sh analyze -purl pkg:npm/[email protected]
 
 The analysis involves Macaron downloading the contents of the target repository to the configured, or default, ``output`` folder. Results from the analysis, including checks, are stored in the database found at ``output/macaron.db``  (See :ref:`Output Files Guide <output_files_guide>`). Once the analysis is complete, Macaron will also produce a report in the form of a HTML file.
 

docs/source/pages/using.rst

@@ -17,7 +17,7 @@ Using Macaron
 Analyzing an artifact with a PURL string
 ----------------------------------------
 
-Macaron can analyze an artifact (and its dependencies) to determine its supply chain security posture. To analyze an artifact, you need to provide the PURL identifier of the artifact:
+Macaron can analyze an artifact to determine its supply chain security posture. To analyze an artifact, you need to provide the PURL identifier of the artifact:
 
  .. code-block::
 
@@ -51,6 +51,7 @@ To run Macaron on an artifact, we use the following command:
 
   ./run_macaron.sh analyze -purl <artifact-purl>
 
+Macaron can also analyze the package's dependencies. Please see :ref:`automate-deps-resolution`.
 
 ''''''''''''''''''''''''''''''''''''''
 Automated repository and commit finder
@@ -79,6 +80,7 @@ Within the configuration file under the ``repofinder.java`` header, three option
 - ``repo_pom_paths`` (Values: List of POM tags) - Determines where to search for repository information in the POM files. E.g. scm.url.
 - ``find_parents`` (Values: True or False) - When enabled, the Repository Finding feature will also search for repository URLs in parents POM files of the current dependency.
 
+.. note:: Dependency related configurations like ``artifact_repositories`` or ``find_parents`` can affect  :ref:`Macaron automatic dependency resolution <automate-deps-resolution>`.
 
 .. note:: Finding repositories requires at least one remote call, adding some additional overhead to an analysis run.
 
@@ -113,7 +115,7 @@ Analyzing a source code repository
 Analyzing a public GitHub repository
 ''''''''''''''''''''''''''''''''''''
 
-Macaron can also analyze a public GitHub repository (and potentially the repositories of its dependencies).
+Macaron can also analyze a public GitHub repository.
 
 To run Macaron on a GitHub public repository, we use the following command:
 
@@ -135,14 +137,6 @@ For example, to analyze the SLSA posture of `micronaut-core <https://github.com/
 
   ./run_macaron.sh analyze -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603
 
-.. note:: Macaron automatically detects and analyzes **direct** dependencies for Java Maven and Gradle projects. This process might take a while and can be skipped by using the ``--skip-deps`` option.
-
-Take the same example as above, to disable analyzing `micronaut-core <https://github.com/micronaut-projects/micronaut-core>`_ direct dependencies, we could use the following command:
-
-.. code-block:: shell
-
-  ./run_macaron.sh analyze -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603 --skip-deps
-
 .. note:: By default, Macaron would generate report files into the ``output`` directory in the current working directory. To understand the structure of this directory please see :ref:`Output Files Guide <output_files_guide>`.
 
 With the example above, the generated output reports can be seen here:
@@ -242,7 +236,7 @@ workflows.
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -pe micronaut-core.cue -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603 --skip-deps
+  ./run_macaron.sh analyze -pe micronaut-core.cue -rp https://github.com/micronaut-projects/micronaut-core -b 4.0.x -d 82d115b4901d10226552ac67b0a10978cd5bc603
 
 where ``micronaut-core.cue`` file can contain:
 
@@ -263,6 +257,32 @@ where ``micronaut-core.cue`` file can contain:
 .. note::
   The provenance expectation is verified via the ``provenance_expectation`` check in Macaron. You can see the result of this check in the HTML or JSON report and see if the provenance found by Macaron meets the expectation CUE file.
 
+.. _automate-deps-resolution:
+
+------------------------------------
+Analyzing dependencies automatically
+------------------------------------
+
+Macaron supports automatically detecting and analyzing dependencies for certain types of projects (:ref:`supported_automatic_deps_resolution`). This feature is disabled by default and can be enabled with the CLI flag ``--deps-depth``.
+
+The ``--deps-depth`` flag currently accepts these values:
+
+* ``0``: Disable dependency resolution (Default).
+* ``1``: Resolve and analyze direct dependencies.
+* ``inf``: Resolve and analyze all transitive dependencies.
+
+For example, to analyze `micronaut-core <https://github.com/micronaut-projects/micronaut-core>`_ and its **direct** dependencies, we could use the following command:
+
+.. code-block:: shell
+
+  ./run_macaron.sh analyze \
+    -rp https://github.com/micronaut-projects/micronaut-core \
+    -b 4.0.x \
+    -d 82d115b4901d10226552ac67b0a10978cd5bc603 \
+    --deps-depth=1
+
+.. note:: This process might take a while. Alternatively, you can help Macaron by providing the dependencies information through : :ref:`an sbom <with-sbom>` or :ref:`a Python virtual environment <python-venv-deps>` (for Python packages only).
+
 .. _with-sbom:
 
 ----------------------
@@ -283,10 +303,12 @@ To run the analysis against that SBOM, run this command:
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:maven/org.apache.maven/[email protected]?type=pom -sbom <path_to_sbom>
+  ./run_macaron.sh analyze -purl pkg:maven/org.apache.maven/[email protected]?type=pom -sbom <path_to_sbom> --deps-depth=inf
 
 Where ``path_to_sbom`` is the path to the SBOM you want to use.
 
+.. note:: Make sure to enable dependency resolution with ``--deps-depth``.
+
 .. _python-sbom:
 
 ''''''''''''''''''''''''
@@ -305,7 +327,7 @@ Then run Macaron and pass the SBOM file as input:
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] -sbom <path_to_django_sbom.json>
+  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] -sbom <path_to_django_sbom.json> --deps-depth=inf
 
 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
 Analyzing dependencies in the SBOM without the main software component
@@ -320,7 +342,7 @@ Then the analysis can be run as follows:
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:maven/private.apache.maven/[email protected]?type=pom -sbom <path_to_sbom>
+  ./run_macaron.sh analyze -purl pkg:maven/private.apache.maven/[email protected]?type=pom -sbom <path_to_sbom> --deps-depth=inf
 
 Where ``path_to_sbom`` is the path to the SBOM you want to use.
 
@@ -344,10 +366,12 @@ Then run Macaron as follows:
 
 .. code-block:: shell
 
-  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv"
+  ./run_macaron.sh analyze -purl pkg:pypi/[email protected] --python-venv "/tmp/.django_venv" --deps-depth=1
 
 Where ``--python-venv`` is the path to virtual environment.
 
+.. note:: Make sure to enable dependency resolution with ``--deps-depth``.
+
 Alternatively, you can create an SBOM for the python package and provide it to Macaron as input as explained :ref:`here <with-sbom>`.
 
 .. note:: We only support Python 3.11 for this feature of Macaron. Please make sure to install the package using this version of Python.
@@ -396,7 +420,7 @@ We can run Macaron against the local repository at ``target`` by using this comm
 
   ./run_macaron.sh --local-repos-path ./boo/foo --defaults-path ./defaults.ini analyze --repo-path target <rest_of_args>
 
-With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
+With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` similar to two previous examples).
 
 The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
 
@@ -422,7 +446,7 @@ We can run Macaron against the local repository at ``target`` by using this comm
 
   ./run_macaron.sh --local-repos-path ./boo/foo analyze --repo-path target <rest_of_args>
 
-With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` or ``--skip-deps`` similar to two previous examples).
+With ``rest_of_args`` being the arguments to the ``analyze`` command (e.g. ``--branch/-b``, ``--digest/-d`` similar to two previous examples).
 
 The ``--local-repos-path/-lr`` flag tells Macaron to look into ``./boo/foo`` for local repositories. For more information, please see :ref:`Command Line Usage <cli-usage>`.
 

src/macaron/__main__.py

@@ -31,7 +31,23 @@
 
 def analyze_slsa_levels_single(analyzer_single_args: argparse.Namespace) -> None:
     """Run the SLSA checks against a single target repository."""
+    deps_depth = None
+    if analyzer_single_args.deps_depth == "inf":
+        deps_depth = -1
+    else:
+        try:
+            deps_depth = int(analyzer_single_args.deps_depth)
+        except ValueError:
+            logger.error("Please provide '1', '0' or 'inf' to `--deps-depth`")
+            sys.exit(os.EX_USAGE)
+
+    if deps_depth not in [-1, 0, 1]:
+        logger.error("Please provide '1', '0' or 'inf' to `--deps-depth`")
+        sys.exit(os.EX_USAGE)
+
     if not (analyzer_single_args.repo_path or analyzer_single_args.package_url):
+        # We don't mention --config-path as a possible option in this log message as it going to be move soon.
+        # See: https://github.com/oracle/macaron/issues/417
         logger.error(
             """Analysis target missing. Please provide a package url (PURL) and/or repo path.
             Examples of a PURL can be seen at https://github.com/package-url/purl-spec:
@@ -128,10 +144,16 @@ def analyze_slsa_levels_single(analyzer_single_args: argparse.Namespace) -> None
             logger.error("Error while loading the input provenance file: %s", error)
             sys.exit(os.EX_DATAERR)
 
+    if analyzer_single_args.skip_deps:
+        logger.warning(
+            "The --skip-deps flag has been deprecated and WILL NOT do anything. "
+            + "Dependency resolution is off by default. This flag does nothing and will be removed in the next release."
+        )
+
     status_code = analyzer.run(
         run_config,
         analyzer_single_args.sbom_path,
-        analyzer_single_args.skip_deps,
+        deps_depth,
         provenance_payload=prov_payload,
     )
     sys.exit(status_code)
@@ -312,7 +334,10 @@ def main(argv: list[str] | None = None) -> None:
         required=False,
         type=str,
         default="",
-        help=("The path to the SBOM of the analysis target."),
+        help=(
+            "The path to the SBOM of the analysis target. If this is set, "
+            + "dependency resolution must be enabled with '--deps-depth'."
+        ),
     )
 
     single_analyze_parser.add_argument(
@@ -375,7 +400,19 @@ def main(argv: list[str] | None = None) -> None:
         required=False,
         action="store_true",
         default=False,
-        help=("Skip automatic dependency analysis."),
+        help=(
+            "DEPRECATED. Dependency resolution is off by default. This flag does nothing and will be removed in the next release."
+        ),
+    )
+
+    single_analyze_parser.add_argument(
+        "--deps-depth",
+        required=False,
+        default="0",
+        help=(
+            "The depth of the dependency resolution. 0: disable, 1: direct dependencies, "
+            + "inf: all transitive dependencies. (Default: 0)"
+        ),
     )
 
     single_analyze_parser.add_argument(
@@ -390,7 +427,10 @@ def main(argv: list[str] | None = None) -> None:
     single_analyze_parser.add_argument(
         "--python-venv",
         required=False,
-        help=("The path to the Python virtual environment of the target software component."),
+        help=(
+            "The path to the Python virtual environment of the target software component. "
+            + "If this is set, dependency resolution must be enabled with '--deps-depth'."
+        ),
     )
 
     # Dump the default values.