Ingest Go dependencies using Semgrep API

[hanzo]

Summary

Motivation: The library dependency graph is already populated with PythonLibrary nodes via Cartography's Github module, but no other languages are supported. This PR adds GoLibrary nodes to the library dependency graph to bring support for Go up to parity with Python. I concur with the recommendation from @heryxpc that rather than writing code to manually parse go.mod files, we should instead use the dependency data returned by the Semgrep API.

Cartography's Semgrep module is already able to import supply chain vulnerability data from the Findings endpoint of the Semgrep API. Semgrep also provides a List Dependencies endpoint that returns a list of every known dependency for a given ecosystem (e.g. specifying the “gomod” ecosystem returns all dependencies found in go.mod files). The response contains useful information including the transitivity of the dependency and a link to where it’s defined in source code.

The dependency nodes imported from the Semgrep API will be labelled GoLibrary::SemgrepDependency::Dependency and will match the properties of existing PythonLibrary::Dependency nodes as closely as possible. This PR only imports Go dependencies from Semgrep, but I've structured the code to make it easy to import additional languages from Semgrep in the future.

Before these changes, a project with both Python and Go dependencies will only have PythonLibrary nodes in the dependency graph:

After these changes, for the same project the graph contains both PythonLibrary and GoLibrary nodes:

Logs from semgrep module before these changes

INFO:cartography.sync:Starting sync with update tag '1730497895'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'semgrep'
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA findings sync job.
INFO:cartography.intel.semgrep.findings:Loading Semgrep deployment info {'id': ...} into the graph...
INFO:cartography.intel.semgrep.findings:Retrieving Semgrep SCA vulns for deployment 'X'.
INFO:cartography.intel.semgrep.findings:Processed page 0 of Semgrep SCA vulnerabilities.
...
INFO:cartography.intel.semgrep.findings:Processed page X of Semgrep SCA vulnerabilities.
INFO:cartography.intel.semgrep.findings:Retrieved X Semgrep SCA vulns in X pages.
INFO:cartography.intel.semgrep.findings:Loading X Semgrep SCA vulns info into the graph.
INFO:cartography.intel.semgrep.findings:Loading X Semgrep SCA usages info into the graph.
INFO:cartography.graph.statement:Completed semgrep_sca_risk_analysis statement #1
...
INFO:cartography.graph.statement:Completed semgrep_sca_risk_analysis statement #X
INFO:cartography.graph.job:Finished job semgrep_sca_risk_analysis
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA findings cleanup job.
INFO:cartography.graph.statement:Completed SemgrepSCAFinding statement #1
...
INFO:cartography.graph.statement:Completed SemgrepSCAFinding statement #X
INFO:cartography.graph.job:Finished job SemgrepSCAFinding
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA Locations cleanup job.
INFO:cartography.graph.statement:Completed SemgrepSCALocation statement #1
...
INFO:cartography.graph.statement:Completed SemgrepSCALocation statement #X
INFO:cartography.graph.job:Finished job SemgrepSCALocation
INFO:cartography.sync:Finishing sync stage 'semgrep'
INFO:cartography.sync:Finishing sync with update tag '1730497895'

Logs from semgrep module after these changes

INFO:cartography.sync:Starting sync with update tag '1730505324'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'semgrep'
INFO:cartography.intel.semgrep.deployment:Loading Semgrep deployment info {'id': ...} into the graph...
INFO:cartography.intel.semgrep.dependencies:Running Semgrep dependencies sync job.
INFO:cartography.intel.semgrep.dependencies:Retrieving Semgrep dependencies for deployment 'X'.
INFO:cartography.intel.semgrep.dependencies:Processed page 0 of Semgrep dependencies.
...
INFO:cartography.intel.semgrep.dependencies:Processed page X of Semgrep dependencies.
INFO:cartography.intel.semgrep.dependencies:Retrieved X Semgrep dependencies in X pages.
INFO:cartography.intel.semgrep.dependencies:Loading X GoLibrary objects into the graph.
INFO:cartography.intel.semgrep.dependencies:Running Semgrep Go Library cleanup job.
INFO:cartography.graph.statement:Completed GoLibrary statement #1
...
INFO:cartography.graph.statement:Completed GoLibrary statement #X
INFO:cartography.graph.job:Finished job GoLibrary
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA findings sync job.
INFO:cartography.intel.semgrep.findings:Retrieving Semgrep SCA vulns for deployment 'lyft'.
INFO:cartography.intel.semgrep.findings:Processed page 0 of Semgrep SCA vulnerabilities.
...
INFO:cartography.intel.semgrep.findings:Processed page X of Semgrep SCA vulnerabilities.
INFO:cartography.intel.semgrep.findings:Retrieved X Semgrep SCA vulns in X pages.
INFO:cartography.intel.semgrep.findings:Loading X Semgrep SCA vulns info into the graph.
INFO:cartography.intel.semgrep.findings:Loading X Semgrep SCA usages info into the graph.
INFO:cartography.graph.statement:Completed semgrep_sca_risk_analysis statement #1
...
INFO:cartography.graph.statement:Completed semgrep_sca_risk_analysis statement #X
INFO:cartography.graph.job:Finished job semgrep_sca_risk_analysis
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA findings cleanup job.
INFO:cartography.graph.statement:Completed SemgrepSCAFinding statement #1
...
INFO:cartography.graph.statement:Completed SemgrepSCAFinding statement #X
INFO:cartography.graph.job:Finished job SemgrepSCAFinding
INFO:cartography.intel.semgrep.findings:Running Semgrep SCA Locations cleanup job.
INFO:cartography.graph.statement:Completed SemgrepSCALocation statement #1
...
INFO:cartography.graph.statement:Completed SemgrepSCALocation statement #X
INFO:cartography.graph.job:Finished job SemgrepSCALocation
INFO:cartography.sync:Finishing sync stage 'semgrep'
INFO:cartography.sync:Finishing sync with update tag '1730497895'

Checklist

Provide proof that this works (this makes reviews move faster). Please perform one or more of the following:

• Update/add unit or integration tests.
• Include a screenshot showing what the graph looked like before and after your changes.
• Include console log trace showing what happened before and after your changes.

If you are changing a node or relationship:

• Update the schema and readme.

If you are implementing a new intel module:

• Use the NodeSchema data model.

[hanzo]

Now that this sync() function is being used for more than just findings, should I move it to a separate file, e.g. common.py? Or I could create a separate sync() function in dependencies.py, but it would need to duplicate a lot of this function's code.

[heryxpc]

Good question. I would say an approach like the github ingestion is good, multiple resource syncs called from the start_semgrep_ingestion.

The only code you need to repeat is the cleanup and merge_module_sync_metadata process.

[hanzo]

The other thing is that the findings API requires the deployment ID, currently retrieved here:

cartography/cartography/intel/semgrep/findings.py

Lines 275 to 278 in 810e391

	semgrep_deployment = get_deployment(semgrep_app_token)
	deployment_id = semgrep_deployment["id"]
	deployment_slug = semgrep_deployment["slug"]
	load_semgrep_deployment(neo4j_sesion, semgrep_deployment, update_tag)

I could update start_semgrep_ingestion to fetch the deployment ID and pass it as an argument to the other sync functions?

[hanzo]

@heryxpc I've done some refactoring, what do you think? I don't love the way it's set up now but couldn't figure out a cleaner approach, open to suggestions

[hanzo]

I've taken another pass at organizing the code, this way is much cleaner but it feels a little brittle that sync_deployment sets values in common_job_parameters that are required by sync_findings and sync_dependencies, so it would break if you called the sync functions in a different order

[hanzo]

I named this deployment.py instead of deployments.py to match the existing file models/deployment.py.

The contents of this file have been moved here from intel/semgrep/findings.py without changes

[hanzo]

I considered what might cause this string split to give the wrong result, but I think it's very unlikely. Even if a repo stored its go.mod file inside a directory named /blob/ (which would be really strange), the url returned from semgrep would be something like https://github.com/org/repo/blob/sha/blob/go.mod#L112, so repo_url would still be set to https://github.com/org/repo as expected.

[hanzo]

I don't like this mechanism for getting the required parameters, would love to hear suggestions for improvement

[achantavy]

Left nits but nothing blocking. Thanks for doing this and for including the screenshots and tests!!

[achantavy]

Just wondering, why not just raise?

[hanzo]

Only because this is copied from

cartography/cartography/intel/semgrep/findings.py

Line 88 in 3e7941b

raise e

I'll update both lines to raise

[achantavy]

[non-block] i'm not a huge fan of metaprogramming here but I won't block on this.

That aside, if we decide to keep this bit of metaprogramming, it'd be good to log the label of the dependency_schema object so that the log message shows what asset is getting written to the graph.

[hanzo]

This is copied from

cartography/cartography/intel/semgrep/findings.py

Line 224 in 3e7941b

logger.info(f"Loading {len(vulns)} Semgrep SCA vulns info into the graph.")

I'll update all of these log lines to use the label of the schema object

[hanzo]

Updated logs:

INFO:cartography.intel.semgrep.dependencies:Retrieved X Semgrep dependencies in Y pages.
INFO:cartography.intel.semgrep.dependencies:Loading X GoLibrary objects into the graph.
INFO:cartography.intel.semgrep.dependencies:Running Semgrep Go Library cleanup job.

[achantavy]

[Non-block]
since this rel is specifically for a go-library to GitHub repo, I'd prefer to add some copypasta to make it separate. This is a style choice and I won't block here though since in future we can decouple that out if we decide that another semgrep dependency needs to have a different relationship definition than a go library.

[hanzo]

From what I've seen, all semgrep dependencies share the same relationship with the github repo where they were detected by semgrep. I'd be inclined to leave it as is for now and we can decouple it later if we discover that a different relationship is needed

[olivia-hong]

will defer approval to the cartography experts but overall lgtm!