Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: replace github search api with graphql in success lifecycle method #857

Merged
merged 37 commits into from
Jul 1, 2024

Conversation

babblebey
Copy link
Member

@babblebey babblebey commented Jun 13, 2024

This Pull Request refactors the success lifecycle function replacing the usage of Github "Search API" with "GraphQL API".

Changes Made

  • Added buildAssociatedPRsQuery helper function to which builds GraphQL query for fetching associated PRs to a list of commit hash (sha). It does this by collecting a list of shas which maps down to a list of strings that leverages the GraphQL fragment enabling ability to request data of multiple commits in one request using GitObject.

    /**
     * Builds GraphQL query for fetching associated PRs to a list of commit hash (sha)
     * @param {Array<string>} shas
     * @returns {string}
     */
    export function buildAssociatedPRsQuery(shas) {
      return `#graphql
        query getAssociatedPRs($owner: String!, $repo: String!) {
          repository(owner: $owner, name: $repo) {
            ${shas
              .map((sha) => {
                return `commit${sha.slice(0, 6)}: object(oid: "${sha}") {
                ...on Commit {
                  associatedPullRequests(first: 100) {
                    nodes {
                      url
                      number
                      body
                    }
                  }
                }
              }`;
              })
              .join("")}
          }
        }
      `;
    }
  • Replaced the instance of the search api consumption with the getSearchQueries method integration with the graphql api integration, which was used to retrieve all associated pull requests.

    const { repository } = await octokit.graphql(
      buildAssociatedPRsQuery(shas),
      { owner, repo },
    );
    const associatedPRs = Object.values(repository).map(
      (item) => item.associatedPullRequests.nodes,
    );
  • Modified test suites in acknowledgement of the changes..

Related Issue

Fixes #644

Screencast/Screenshot

screencast-bpconcjcammlapcogcnnelfmaeghhagj-2024.06.24-14_54_49.webm

lib/success.js Outdated Show resolved Hide resolved
lib/success.js Outdated Show resolved Hide resolved
Copy link
Member Author

@babblebey babblebey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure this works, but can't exactly test, hence I've got to figure how to update the test for a mock .... This is where i'm kinda BLOCKED..

See the comments associated to this review first.... then hear my BLOCKER 😉

I notice the convention with the usage of the fetch-mock's sandbox().getOnce() method which mocks a request and provides a response to the immediate request provided a mock endpoint like so...

fetchMock
    .sandbox()
    .getOnce(`https://api.github.local/repos/${owner}/${repo}`, {
      full_name: `${redirectedOwner}/${redirectedRepo}`,
    }); // getOnce("<mockRequestEnpoint>", "<mockResponse>")

but my question is...

HOW EXACTLY DO I MOCK THE GRAPHQL ENDPOINT??? using similar convention 🤔

  • I'll need the exact appropriate mock GraphQL endpoint????

I'm not sleeping on it too 😉


UPDATE:

I found https://github.com/octokit/graphql.js?tab=readme-ov-file#writing-tests 🤔, requires some new dependencies 😞 I think now... If there's a way to do this without adding any new dependency, I'll take that

@gr2m
Copy link
Member

gr2m commented Jun 14, 2024

requires some new dependencies

you don't need the @octokit/graphql dependency to mock graphql requests, you just mock the POST /graphql request like any other.

lib/success.js Outdated Show resolved Hide resolved
lib/success.js Outdated Show resolved Hide resolved
lib/success.js Outdated Show resolved Hide resolved
@babblebey
Copy link
Member Author

you don't need the @octokit/graphql dependency to mock graphql requests, you just mock the POST /graphql request like any other.

Oh wow, I'll give that a shot... I already done too much if this is the case in my initial attempt 🫣

@gr2m
Copy link
Member

gr2m commented Jun 14, 2024

Holler if you need help with this PR

@babblebey babblebey changed the title fix (refactor): replace github search api with graphql refactor: replace github search api with graphql in success lifecycle method Jun 14, 2024
@babblebey babblebey marked this pull request as ready for review June 14, 2024 20:47
Copy link
Member

@gr2m gr2m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yoooo!! Awesome PR, can't wait to give this a go! I've to run now but might get to it later tonight! Great work 💐

lib/get-search-queries.js Outdated Show resolved Hide resolved
@gr2m gr2m changed the title refactor: replace github search api with graphql in success lifecycle method fix: replace github search api with graphql in success lifecycle method Jun 15, 2024
@gr2m
Copy link
Member

gr2m commented Jun 15, 2024

I ran it locally and got the following error

[8:46:26 PM] [semantic-release] › ℹ  Start step "success" of plugin "@semantic-release/github"
[8:46:27 PM] [semantic-release] › ✘  Failed step "success" of plugin "@semantic-release/github"
[8:46:27 PM] [semantic-release] › ✘  An error occurred while running semantic-release: Error: Parse error on "commit338966" (STRING) at [4, 9]
    at file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:23:26
    at Array.map (<anonymous>)
    at new AggregateError (file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:16:19)
    at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
    at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
    at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
    at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
    at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
  locations: [ { line: 4, column: 9 } ],
  pluginName: '@semantic-release/github'
}
AggregateError: 
    Error: Parse error on "commit338966" (STRING) at [4, 9]
        at Array.map (<anonymous>)
        at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
        at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
        at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
        at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
        at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5)
    at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
    at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
    at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
    at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
    at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
  errors: [
    Error: Parse error on "commit338966" (STRING) at [4, 9]
        at file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:23:26
        at Array.map (<anonymous>)
        at new AggregateError (file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:16:19)
        at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
        at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
        at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
        at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
        at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
      locations: [Array],
      pluginName: '@semantic-release/github'
    }
  ]

In order to run it locally, checkout the semantic-release/semantic-release repo and the semantic-release/github repo. Run npm install in both. Then in the semantic-release folder, run npm install <relative path to the github folder> (e.g. in my case on Mac OS, the folders are next to each other, I ran npm install ../github.

The way I ran it locally is as follows

  1. Create a test repository with a package.json, set "version": "0.0.0-development"

  2. check it out locally

  3. Add a commit with feat: initial version or similar (git commit --allow-empty -m 'feat: initial version')

  4. Make sure you are signed out of npm locally (run npm logout)

  5. Create a classic automation token on npm (https://www.npmjs.com/settings//tokens/`)

  6. Create a personal access token with the repo scope

  7. Set the following environment variables

    CI=true
    GITHUB_ACTIONS=true
    GITHUB_REF=master
    GITHUB_SHA=<latest commit sha>
    GITHUB_RUN_ID=1
    GITHUB_REPOSITORY=<your full repository name (owner/repo)>
    GITHUB_WORKSPACE=<path to current folder>
    NPM_TOKEN=<your npm token>
    GITHUB_TOKEN=<your github token>
    
  8. Run the following command in your local clone of the test repository

    <path to semantic-release folder>/bin/semantic-release.js
    

Not sure if this could be streamlined ... but we should document it somewhere for future reference and improve if we can 😁

@gr2m
Copy link
Member

gr2m commented Jun 15, 2024

It's possible that this error is an edge case that only happens because I ran it locally. The commit only exist on my machine due to the local testing. When your query is sent, the commit does not exist and probably returns null or even an error. If that happens, I'd simply ignore the commit and move on to the next one

@babblebey
Copy link
Member Author

babblebey commented Jun 21, 2024

It's possible that this error is an edge case that only happens because I ran it locally. The commit only exist on my machine due to the local testing. When your query is sent, the commit does not exist and probably returns null or even an error. If that happens, I'd simply ignore the commit and move on to the next one

@gr2m, I figured it was a GraphQL error... turned out I had a some part of the query returned string (from the buildAssociatedPRsQuery) wrongly typed... i.e. the alias text and the oid.

I have recorded and added a screencast to description above.

@gr2m
Copy link
Member

gr2m commented Jun 21, 2024

Great work! I'd like to see that the success step actually posts a comment, we can pair on it during our checkin today

@babblebey
Copy link
Member Author

babblebey commented Jun 24, 2024

Hi @gr2m, turns out the reason why the comment wasn't posting to only merged PRs and not the associated closed/fixed issue is because I failed to retrieve the body property on the graphql fetched associatedPRs, hence no PR body to parse the associated closed/fixed issueNumber from.

Kindly checkout description for updated Demo Video, also PRs and Issue below...

🤦

Copy link
Member

@gr2m gr2m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's goo 🚀

Comment on lines +254 to +272
query getAssociatedPRs($owner: String!, $repo: String!) {
repository(owner: $owner, name: $repo) {
${shas
.map((sha) => {
return `commit${sha.slice(0, 6)}: object(oid: "${sha}") {
...on Commit {
associatedPullRequests(first: 100) {
nodes {
url
number
body
}
}
}
}`;
})
.join("")}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to address the possibility that there might be more than 100 commits. If that occurs, we need to send multiple requests as only 100 nodes can be requested with a single GraphQL request

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was this addressed in this PR? im working with a repo that has more than 100 commits and am experiencing this secondary rate error on v24.0.0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @brandon-kyle-bailey

The Edge Case isn't addressed in this PR nor yet.

But, could you kindly open an issue with details of this error you're getting in regard this 100+ commit, thanks.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately this is happening on an internal repo but ill do my best! @babblebey

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@babblebey im not able to reliable reproduce it at this point since re-running so will hold off on logging an issue. will revisit if it reappears!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm currently working on a fix in regard the edge case here #892

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just capturing the relation to #867 (comment) as well

@gr2m gr2m enabled auto-merge (squash) July 1, 2024 17:13
@gr2m gr2m merged commit be394cf into master Jul 1, 2024
6 checks passed
@gr2m gr2m deleted the refactor/replace-search-api branch July 1, 2024 17:15
@gr2m
Copy link
Member

gr2m commented Jul 1, 2024

How fitting that the release of this fix also ran into the secondary rate limit issue:
https://github.com/semantic-release/github/actions/runs/9748385825/job/26903265854
(but that was still using the old code though)

@babblebey
Copy link
Member Author

(but that was still using the old code though)

I sigh in relief**** 😮‍💨

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API rate limit exceeded for installation
4 participants