Feature: Dependency-Update-Tool should check whether tools are available

[artem-smotrakov]

Dependency-Update-Tool checks if a project uses Dependabot or Renovate bot. Although these tools support many languages and ecosystems, there are cases they don't support, for example, C/C++ projects. For such projects, it is often not feasible to set up a tool for automated dependency updates. This is related to #74.

For example, Dependency-Update-Tool reports 0 / 10 for CuraEngine that is writted in C++. This score doesn't look fair because it doesn't seem to be possible to configure automatic dependency updates. Neither Dependabot nor Renovatebot can help here.

I'd like to propose updating the check, so that it reports ? for projects that use languages/ecosystems that are not supported by Dependabot/Renovate bot. GitHub API provides a list of languates used in a project repository. The check could use this info to decide if automated dependency updates are possible.

Alternatively, the check could return a non-zero score (but less than 10) if no automated updates are possible.

I'd be happy to draft a pull request for that if the idea is accepted.

[artem-smotrakov]

These docs list languages/ecosystems that Dependabot and Renovatebot support:

• https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/about-dependabot-version-updates#supported-repositories-and-ecosystems
• https://docs.renovatebot.com/modules/manager/

[laurentsimon]

Thanks for the report, and sorry for the late reply.. I've missed your issue!

What you suggest seems like a reasonable thing to do. The only drawback is that the check won't be accessible for local repositories - since it now needs an API call; but I think that's fine. We could try to list files and look for extensions.. but that sounds like a a lot of work and prone to errors.

I think it's fine to return -1 if it's not possible with dependabot/renovatebot.

Feel free to send a PR and cc me to it for review. Thanks again for your help!

[agonzalez-plume]

Agree, C/C++ projects could use a software repo like Conan, but usage is sporadic. The auto dependency tools mentioned don't support it.

What would this check show for the linux kernel for instance, probably that it's not compliant.

[agonzalez-plume]

I'm assuming a change for this would be in the evaluation module using repo.languages. 'GET /repos/{owner}/{repo}/languages'

if repo_language == 'C' {
	dl.Warn(&checker.LogMessage{
		Text: "no dependency update tool available",
	})
	// Generic summary.
	return checker.CreateInconclusiveResult(name, "no tool available")
}

OR

if repo_language == 'C' {
	dl.Warn(&checker.LogMessage{
		Text: "unsupported language for dependency update",
	})
	// Generic summary.
	return checker.CreateInconclusiveResult(name, "unsupported language")
}

Right?

[laurentsimon]

@naveensrinivasan @azeemsgoogle @justaugustus any comment or thoughts on this?

[justaugustus]

It would be great to have something like a scorecard config file, where we could allow users to configure acceptable parameters like, "I am not using a dependency update tool/language that is supported by the supported dependency update tools".

The check could use this info to decide if automated dependency updates are possible.

Alternatively, the check could return a non-zero score (but less than 10) if no automated updates are possible.

I think the right approach is refuse to score a scenario that's impossible to satisfy.

[laurentsimon]

Would anyone object if we returned inconclusive results for ecosystems not supported by dependabot and other tools? I think this is what refuse to score a scenario that's impossible to satisfy means, correct?

Would that be a good first step towards improving this check?

[nealmcb]

If I'm evaluating the security of a tool, and the tool is written in a language doesn't allow effective dependency tracking, my guess is that in many if not most cases, the tool is more vulnerable than an otherwise similar tool written in a language with good dependency tracking support.

I.e. I'd hate to see what amounts to an undeserved bonus awarded for use of a problematic language / ecosystem.

And I'd love for us to find ways to incentivize projects and dependency-checking tools for languages that are currently unsupported. That is sort of what the current bad score does, but perhaps there is a better way to frame it.

If we provide a way for project admins to configure scorecard to explain such things away, it may be used appropriately at times, but I wonder if we'll need new tooling to detect unskillful or dubious configurations of that sort.

[laurentsimon]

to your last point: I would like to enable maintainers to explain their score/results, for example thru code annotation (or anything else really), so that we can provide that data along with the findings. We don't necessarily have to change the score, but at least the information would be available for a human to inspect.

Note that we're trying to move away from scores #1245 (WIP). Any ideas you have on results presentation are useful in this context. Please feel free to share

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[github-actions]

This issue has been marked stale because it has been open for 60 days with no activity.