feat(taps): Add files and stats to PullRequestCommitsStream

[dlouseiro]

The purpose of this PR is to add the files and stats propertied to the PullRequestCommitsStream.

These properties contain information about the changes done on a given commit (additions, deletions and total). files has these detailed at the file level whereas stats has a summary of changes at the commit level.

[dlouseiro]

This PR does not seem to fix the issue. Need to investigate further!

[edgarrmondragon]

@dlouseiro Thanks for the PR!

This PR does not seem to fix the issue. Need to investigate further!

I wonder if you're running into meltano/meltano#6763. Can you try the recommended workaround of removing the cached catalog in .meltano/run/tap-github?

[dlouseiro]

@dlouseiro Thanks for the PR!

This PR does not seem to fix the issue. Need to investigate further!

I wonder if you're running into meltano/meltano#6763. Can you try the recommended workaround of removing the cached catalog in .meltano/run/tap-github?

Ah possibly! Thanks for the input. I'm already performing a test run on a clean meltano installation in our kubernetes cluster so should be able to test it soon!

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[dlouseiro]

@dlouseiro Thanks for the PR!

This PR does not seem to fix the issue. Need to investigate further!

I wonder if you're running into meltano/meltano#6763. Can you try the recommended workaround of removing the cached catalog in .meltano/run/tap-github?

Ah possibly! Thanks for the input. I'm already performing a test run on a clean meltano installation in our kubernetes cluster so should be able to test it soon!

@edgarrmondragon after some testing I actually noticed that, in order to get the files property at the commit level we'd need to do one REST call per commit as the files properties is not available in the list commits API.

Although, the same property as well as stats is available when listing Pull Request commits so I repurposed the PR to add these properties to the PullRequestCommitsStream instead!

[edgarrmondragon]

Thanks @dlouseiro!

[alexlevene]

I just started looking into using this tap in our Meltano setup and wanted to get a listing of files for a PR. This change seemed like it was great timing for what I needed, but I'm not actually seeing the files and stats keys in the output.

They are indeed listed in the "Response Schema" in the API docs but actual responses don't have them.

For example on this pull request:

$ curl -L -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28"   https://api.github.com/repos/MeltanoLabs/tap-github/pulls/241/commits | jq -n 'inputs[] | keys[]' | sort | uniq
"author"
"comments_url"
"commit"
"committer"
"html_url"
"node_id"
"parents"
"sha"
"url"

Just wondering if you you witnessed different responses? Just want to make sure I'm using the API correctly 😅

I've also started my own fork of this repo, building out a PullRequestFiles child stream which does reliably return the file information.

$ curl -L -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28"   https://api.github.com/repos/MeltanoLabs/tap-github/pulls/241/files
[
  {
    "sha": "16104839cb25e043016a9f00454e66e2c941778c",
    "filename": "tap_github/repository_streams.py",
    "status": "modified",
    "additions": 10,
    "deletions": 0,
    "changes": 10,
    ...
  },
  {
    "sha": "98381f16592975c5a1487871205a553b2ca6b85f",
    "filename": "tap_github/schema_objects.py",
    "status": "modified",
    "additions": 14,
    "deletions": 0,
    "changes": 14,
    ...
  }
]

Thanks!