github sync nonetype error when listing repo-teams

[achantavy]

Bug Template

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Cartography should not crash, the expected value isn't returned, the data schema is wrong, etc.

GitHub sync crashes when listing repo-teams with a Nonetype error;

To Reproduce:

Steps to reproduce the behavior. Provide all data and inputs required to reproduce the issue.

Not sure why this suddenly started failing.

Logs:

If applicable, copy and paste your console log with the failing stack trace.

Crash:

Traceback (most recent call last):
  File "PATH/cartography/intel/github/teams.py", line 174, in sync_github_teams
    team_repos = _get_team_repos_for_multiple_teams(teams_paginated.nodes, organization, github_url, github_api_key)
  File "PATH/cartography/util.py", line 192, in timed
    result = method(*args, **kwargs)
  File "PATH/cartography/intel/github/teams.py", line 63, in _get_team_repos_for_multiple_teams
    repo_urls = [t['url'] for t in team_repos.nodes] if team_repos.nodes else []
  File "PATH/cartography/intel/github/teams.py", line 63, in <listcomp>
    repo_urls = [t['url'] for t in team_repos.nodes] if team_repos.nodes else []
TypeError: 'NoneType' object is not subscriptable

Please complete the following information::

• Cartography release version or commit hash [e.g. 0.12.0 or 95e8e11]
  0.93.0

[achantavy]

We fail here
https://github.com/lyft/cartography/blob/e149967cd2750c614d0cb94fced6b1e60870a5cf/cartography/intel/github/teams.py#L63

which relies on data from this graphql object:
https://github.com/lyft/cartography/blob/e149967cd2750c614d0cb94fced6b1e60870a5cf/cartography/intel/github/teams.py#L85

and that is defined here:
https://docs.github.com/en/graphql/reference/objects#repository

The ! indicates the field is non-nullable, so since it has returned None that means something bad has happened on the GitHub server side and it is raising an exception to give us the option to stop execution.

We could add in logic to continue on despite this GitHub error, but this would cause us to lose team-to-repo mappings or team permission data in the graph. This is because something wrong happened on GitHub's end and if we don't include a team-to-repo mapping in this run, it will get cleaned up and disappear.

I think I'd prefer to continue the current behavior of failing the sync here rather than losing data when GitHub has problems. Still, there may be something wrong with our specific deployment if this problem is happening frequently enough.

cc: @heryxpc

[achantavy]

The change could look something like this:

but again, this would mean continuing on with data loss.

[achantavy]

Another approach: wrap it with retry + sleep. I started this in https://github.com/lyft/cartography/pull/1336/files. Will add tests when I have time but assuming my theory about GitHub API not behaving well is correct, this should help.