Add new Mend Platform API 3.0 file types to existing Mend parser

[testaccount90009]

Existing Mend parser is for 1.0 or 2.0 and Unified Agent based 'legacy' vulnerabilities schema, whereas a new Mend Platform and API 3.0 has changed this structure, requiring a need for a new parser. This is for SCA only at the moment.

⚠️ Note on feature completeness ⚠️

We are narrowing the scope of acceptable enhancements to DefectDojo in preparation for v3. Learn more here:
https://github.com/DefectDojo/django-DefectDojo/blob/master/readme-docs/CONTRIBUTING.md

Description

Describe the feature / bug fix implemented by this PR.
If this is a new parser, the parser guide may be worth (re)reading.

Test results

Ideally you extend the test suite in tests/ and dojo/unittests to cover the changed in this PR.
Alternatively, describe what you have and haven't tested.

Documentation

Please update any documentation when needed in the documentation folder)

Checklist

This checklist is for your information.

• Make sure to rebase your PR against the very latest dev.
• Features/Changes should be submitted against the dev.
• Bugfixes should be submitted against the bugfix branch.
• Give a meaningful name to your PR, as it may end up being used in the release notes.
• Your code is flake8 compliant.
• Your code is python 3.11 compliant.
• If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
• Model changes must include the necessary migrations in the dojo/db_migrations folder.
• Add applicable tests to the unit tests.
• Add the proper label to categorize your PR.

Extra information

Please clear everything below when submitting your pull request, it's here purely for your information.

Moderators: Labels currently accepted for PRs:

• Import Scans (for new scanners/importers)
• enhancement
• performance
• feature
• bugfix
• maintenance (a.k.a chores)
• dependencies
• New Migration (when the PR introduces a DB migration)
• settings_changes (when the PR introduces changes or new settings in settings.dist.py)

Contributors: Git Tips

Rebase on dev branch

If the dev branch has changed since you started working on it, please rebase your work after the current dev.

On your working branch mybranch:

git rebase dev mybranch

In case of conflict:

 git mergetool
 git rebase --continue

When everything's fine on your local branch, force push to your myOrigin remote:

git push myOrigin --force-with-lease

To cancel everything:

git rebase --abort

Squashing commits

git rebase -i origin/dev

• Replace pick by fixup on the commits you want squashed out
• Replace pick by reword on the first commit if you want to change the commit message
• Save the file and quit your editor

Force push to your myOrigin remote:

git push myOrigin --force-with-lease

[dryrunsecurity]

DryRun Security Summary

The provided code changes focus on improving the parsing and processing of Mend (formerly WhiteSource) Software Composition Analysis (SCA) scan results, including handling different schema structures, enhancing vulnerability descriptions, and adding new unit tests to verify the correct handling of various Mend scan output scenarios, in order to provide more comprehensive and reliable vulnerability data for effective application security management.

Expand for full summary

Summary:

The provided code changes cover several updates to the application's security-related functionality, including improvements to the parsing and processing of Mend (formerly WhiteSource) Software Composition Analysis (SCA) scan results, as well as the addition of new unit tests to verify the correct handling of various Mend scan output scenarios.

The key changes and their security implications are as follows:

1. Mend SCA Parser Enhancements: The changes to the dojo/tools/mend/parser.py file focus on improving the parsing and processing of Mend scan results. This includes handling different schema structures, enhancing the vulnerability description, expanding the mitigation information, and improving the deduplication of findings. These enhancements help to provide more comprehensive and reliable vulnerability data, which is crucial for effective application security management.

2. Mend Scan Test Data Updates: The changes to the unittests/scans/mend/ files introduce new test scenarios, including cases with no findings, one finding, and multiple findings. These updates ensure that the application's security tooling can accurately process and interpret the various types of Mend scan outputs, improving the overall reliability and robustness of the security functionality.

3. Vulnerability Identification and Remediation: The changes to the unittests/scans/mend/mend-sca-platform-api3-eleven-findings.json file highlight several high-severity vulnerabilities in the application's dependencies, such as jackson-databind, Microsoft.Data.OData, and axis. Addressing these vulnerabilities by upgrading to the recommended versions is crucial to mitigate the associated security risks, which include resource exhaustion, denial of service, cross-site scripting, and remote code execution.

Files Changed:

1. unittests/scans/mend/mend-sca-platform-api3-no-findings.json: This file contains a test scenario with no security findings, which is an important case to validate the correct handling of Mend scan outputs.
2. unittests/scans/mend/mend-sca-platform-api3-one-finding.json: This file contains a test scenario with a single security vulnerability, which helps to ensure the parser can correctly process and report on individual findings.
3. dojo/tools/mend/parser.py: The changes to this file focus on improving the parsing and processing of Mend scan results, including handling different schema structures, enhancing vulnerability descriptions, and improving deduplication of findings.
4. unittests/scans/mend/mend-sca-platform-api3-eleven-findings.json: This file contains information about several high-severity vulnerabilities in the application's dependencies, which should be promptly addressed to mitigate the associated security risks.
5. unittests/tools/test_mend_parser.py: The changes to this file introduce new unit tests to cover the parsing of Mend SCA findings, including scenarios with no vulnerabilities, one vulnerability, and multiple vulnerabilities. This helps to ensure the reliability and robustness of the application's security functionality.

Code Analysis

We ran 9 analyzers against 5 files and 0 analyzers had findings. 9 analyzers had no findings.

Riskiness

🟢 Risk threshold not exceeded.

View PR in the DryRun Dashboard.

[testaccount90009]

@manuel-sommer I will try this tomorrow in my dev environment, along with an import of the new File Type to see how DefectDojo accepts it or not. Thank you again for the suggestion!

[testaccount90009]

I have tried this on my local dev stack and everything works, except for the Description field.. It is defaulting to the 'Unknown' value. Will look at fixing that and writing a unit test of some kind for it perhaps.

[Maffooch]

@testaccount90009 great job making sense of the combination of formats. Once you get the description issue figured out, I think this one will be good to go

[testaccount90009]

@Maffooch thank you, I appreciate it! Currently the description is returning 'Unknown' and I am going to shut down for the weekend -- but I will get back to this next week and eventually figure out why it's returning 'Unknown'. My guess is that it's having an issue with the schema of the new json structure... "response": ["vulnerability"{} node get "description"] is an example of the layout of the new schema, whereas the old schema is a bit simpler with just "vulnerabilities": {"description"}.

I have not yet figured it out, but I'll get it on my test stack and input those changes here, likely with a unit test to make sure.

[testaccount90009]

@Maffooch I believe this works now - Description is correctly populating with the newly added changes. I no longer get 'Unknown' on the new platform parser SCA json document and it correctly builds the description similarly to how the legacy json schema does.

[testaccount90009]

To confirm -- I have already built this in my dev stack and have imported a json findings file with multiple, one, etc.. and it populates description now correctly. I think what I set out to implement at this time has been accomplished. Please let me know if there's anything I overlooked or I may be missing.

[testaccount90009]

Ok, I believe I am done making changes. This is a good finishing point for the Mend Platform SCA specific parser.

ToDo in future PRs (that I, or someone else can take care of -- though I don't mind and will get to it soon anyways):
-Mend SAST
-Mend IaC
(both from Platform 3.0 API).

1 parser to rule them all :)

[mtesauro]

Approved