Only Run Jobs for Repos which have Passed Checks

[ghickman]

Using the GitHub Checks API we can check the actions for a repo have run successfully. We should gate the creation of a JobRequest on those checks being successful so they can check incorrect config before we take up the job-runner's time/resources.

A check on page load with UI linking to where the User can see the source output (probably the Actions tab for the repo) can alert the User to why the form isn't submittable. Reloading the page should be ok as a way to pick up changes from GitHub.

[evansd]

This is a great idea, but I think we also need an escape hatch (possibly only available to users with sufficient privileges)

[bloodearnest]

I think we'd a need a warning deprecation period for legacy legendary users before enforcing. But we could enforce for new users, to get them in good habits early.

[ghickman]

I've done a quick spike on this.

Getting the list of checks for a given repo and branch (which we have easy access to in the WorkspaceDetail view) is nice and easy via this url /repos/{owner}/{repo}/commits/{ref}/check-runs, here ref can be a branch.

Where this gets trickier is how we check that research-action has run. Here's the output calling that URL with the master branch of this repo: https://github.com/opensafely/SGTF-CFR-research

	"total_count": 1,
	"check_runs": [{
		"id": 2693249580,
		"node_id": "MDg6Q2hlY2tSdW4yNjkzMjQ5NTgw",
		"head_sha": "5f3175af7acad93eb5e7fb88bb8913a1cef89b7d",
		"external_id": "5264e576-3c6f-51f6-f055-fab409685f20",
		"url": "https://api.github.com/repos/opensafely/SGTF-CFR-research/check-runs/2693249580",
		"html_url": "https://github.com/opensafely/SGTF-CFR-research/runs/2693249580",
		"details_url": "https://github.com/opensafely/SGTF-CFR-research/runs/2693249580",
		"status": "completed",
		"conclusion": "success",
		"started_at": "2021-05-28T10:09:29Z",
		"completed_at": "2021-05-28T10:22:41Z",
		"output": {
			"title": null,
			"summary": null,
			"text": null,
			"annotations_count": 0,
			"annotations_url": "https://api.github.com/repos/opensafely/SGTF-CFR-research/check-runs/2693249580/annotations"
		},
		"name": "Test the project can run, using dummy data",
		"check_suite": {
			"id": 2855068128
		},
		"app": {
			"id": 15368,
			"slug": "github-actions",
			"node_id": "MDM6QXBwMTUzNjg=",
			"owner": {
				"login": "github",
				"id": 9919,
				"node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
				"avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
				"gravatar_id": "",
				"url": "https://api.github.com/users/github",
				"html_url": "https://github.com/github",
				"followers_url": "https://api.github.com/users/github/followers",
				"following_url": "https://api.github.com/users/github/following{/other_user}",
				"gists_url": "https://api.github.com/users/github/gists{/gist_id}",
				"starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
				"subscriptions_url": "https://api.github.com/users/github/subscriptions",
				"organizations_url": "https://api.github.com/users/github/orgs",
				"repos_url": "https://api.github.com/users/github/repos",
				"events_url": "https://api.github.com/users/github/events{/privacy}",
				"received_events_url": "https://api.github.com/users/github/received_events",
				"type": "Organization",
				"site_admin": false
			},
			"name": "GitHub Actions",
			"description": "Automate your workflow from idea to production",
			"external_url": "https://help.github.com/en/actions",
			"html_url": "https://github.com/apps/github-actions",
			"created_at": "2018-07-30T09:30:17Z",
			"updated_at": "2019-12-10T19:04:12Z",
			"permissions": {
				"actions": "write",
				"checks": "write",
				"contents": "write",
				"deployments": "write",
				"discussions": "write",
				"issues": "write",
				"metadata": "read",
				"organization_packages": "write",
				"packages": "write",
				"pages": "write",
				"pull_requests": "write",
				"repository_hooks": "write",
				"repository_projects": "write",
				"security_events": "write",
				"statuses": "write",
				"vulnerability_alerts": "read"
			},
			"events": ["check_run", "check_suite", "create", "delete", "deployment", "deployment_status", "discussion", "discussion_comment", "fork", "gollum", "issues", "issue_comment", "label", "milestone", "page_build", "project", "project_card", "project_column", "public", "pull_request", "pull_request_review", "pull_request_review_comment", "push", "registry_package", "release", "repository", "repository_dispatch", "status", "watch", "workflow_dispatch", "workflow_run"]
		},
		"pull_requests": []
	}]
}

The only reference to the check for that repo is Test the project can run, using dummy data, which is the label we gave the action in the workflow file. This means anyone with write access to the repo can circumvent this check by changing that label.

I suspect that's not a huge problem as this check (geddit) will end up being a warning on the WorkspaceDetail page but wanted to flag it all the same.