Add Score Review Feature #89
Conversation
This can help teams compare the review of the PR agent with that of a human reviewer, and fine-tune a score threshold for automatic approval where they decide the agent's review is satisfactory.
|
/describe |
|
/review |
github-actions
bot
changed the title
PR Analysis
PR Feedback
How to use
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
PR Analysis
PR Feedback
|
| "type": "float", | ||
| "description": "Rate this PR on a scale of 0-100 (inclusive), where 0 means the worst possible code, and 100 means code of the highest quality without any bugs or performace issues that is ready to be merged immediately and run in production at scale." |
There was a problem hiding this comment.
| "type": "float", | |
| "description": "Rate this PR on a scale of 0-100 (inclusive), where 0 means the worst possible code, and 100 means code of the highest quality without any bugs or performace issues that is ready to be merged immediately and run in production at scale." | |
| "type": "str", | |
| "description": "Rate this PR on a scale of 0-100 (inclusive), where 0 means the worst possible PR code, and 100 means PR code of the highest quality, without any bugs or performance issues, that is ready to be merged immediately and run in production at scale. Explain shortly the reason for the score you gave." |
|
Hi @zmeir. That's a nice idea for an additional sub-tool.
p.s. |
|
Hi @mrT23, thanks for the review. To answer your comments:
I was thinking since it's a small output and very similar to the theme of the other sections in the review analysis, that it should belong there. My thought was that separating it to a sub-tool is similar to separating the "Security concerns" or "Relevant tests added" parts of the PR Analysis, which felt wrong.
Strange, it actually worked for me with the code as I submitted it. For example:
I see what you mean, I just thought that this part should be very concise and only add a score that we can then use as a sort of reliability threshold for automatic approval. I thought it should behave in a similar way to the "Type of PR" or "Relevant tests added" fields. I like the idea that the "PR Analysis" part is a sort of "checklist" of simple measurements that you can get a feel of the quality of this PR at a glance, and if you want more details you look at the "PR Feedback" section.
Agree completely. I will set the default to false. |
PR Analysis
PR Feedback
|
@mrT23 I see what you mean, it really didn't output the Score when using my branch. When I tested it locally before opening the PR it did work, I'll check and see if there's something I missed... |
|
@mrT23 this is very strange - when I request review on this PR I don't get a score, but when I try it on a few of my PRs in another repo (private) it's working: Anyway, I added there relevant example output like you suggested, and now it works for all the PRs I've tested :) |
|
@zmeir Ok, if the default is off, maybe we can keep the output short, just as a number. Regarding example output, it makes outputting a json more stable, especially for complicated json like ours |
|
@mrT23 please see the latest changes and let me know what you think
Done. Although I'm usually a very precise person so I do appreciate the decimal precision, but I'm probably the odd one out so I see your point :)
Good to know, I'm very new to working with OpenAI and LLMs in general (relatively speaking, aren't we all?), so I'm always happy to learn new stuff :) |
|
Example for a truly horrible PR just to make sure the score isn't always high:
|
|
/review -i |
Incremental PR Review
PR Analysis
PR Feedback
How to use
|
|
/review -i |
Incremental PR Review
PR Analysis
PR Feedback
How to use
|
Add Score Review Feature
There was a problem hiding this comment.
GitHub Copilot Review
Title: feat: 🎸 move to prod changes
Description:
Changes:
server/util/constants.js
@@ -47,11 +47,11 @@ module.exports = {
LOCAL: 'local',
DEV: 'dev',
PRODUCTION: 'production'
},
- DEVELOPERS_EMAIL: '[email protected]',
- DEVELOPERS_EMAIL: '[email protected]',
SES_HOST: 'email-smtp.us-east-2.amazonaws.com',
- AWS_S3_PUBLIC_BUCKET: 'dev-use2-assests',
- AWS_S3_PUBLIC_BUCKET: process.env.AWS_S3_PUBLIC_BUCKET,
CLIENT_INFO: {
EMAIL: '[email protected]'
},
DATE_FORMAT: 'MM/DD/YYYY',
package.json
@@ -6,9 +6,9 @@
"scripts": {
"prestart": "NODE_ENV=local node_modules/.bin/sequelize db:migrate",
"prestart:dev": "NODE_ENV=development node_modules/.bin/sequelize db:migrate",
"prestart:stg": "NODE_ENV=staging node_modules/.bin/sequelize db:migrate",
- "prestart:prod": "NODE_ENV=production node getEnvs.js && NODE_ENV=production node_modules/.bin/sequelize db:migrate",
- "prestart:prod": "NODE_ENV=production node_modules/.bin/sequelize db:migrate",
"start": "NODE_ENV=local nodemon .",
"start:dev": "NODE_ENV=development node index.js",
"start:stg": "NODE_ENV=staging node index.js",
"start:prod": "NODE_ENV=production node index.js",
There was a problem hiding this comment.
GitHub Copilot Review Guide 🤖
PR Overview
| Category | Details |
|---|---|
| Title | feat: 🎸 move to prod changes |
| Description |
|
Code Analysis
| Category | Details |
|---|---|
| Changes Overview | Analyzing 34 lines of changes |
| Main Focus | Code quality and potential improvements |
Key Findings
- Code Structure
- Best Practices
- Potential Improvements
Detailed Review
server/util/constants.js
@@ -47,11 +47,11 @@ module.exports = {
LOCAL: 'local',
DEV: 'dev',
PRODUCTION: 'production'
},
- DEVELOPERS_EMAIL: '[email protected]',
- DEVELOPERS_EMAIL: '[email protected]',
SES_HOST: 'email-smtp.us-east-2.amazonaws.com',
- AWS_S3_PUBLIC_BUCKET: 'dev-use2-assests',
- AWS_S3_PUBLIC_BUCKET: process.env.AWS_S3_PUBLIC_BUCKET,
CLIENT_INFO: {
EMAIL: '[email protected]'
},
DATE_FORMAT: 'MM/DD/YYYY',
package.json
@@ -6,9 +6,9 @@
"scripts": {
"prestart": "NODE_ENV=local node_modules/.bin/sequelize db:migrate",
"prestart:dev": "NODE_ENV=development node_modules/.bin/sequelize db:migrate",
"prestart:stg": "NODE_ENV=staging node_modules/.bin/sequelize db:migrate",
- "prestart:prod": "NODE_ENV=production node getEnvs.js && NODE_ENV=production node_modules/.bin/sequelize db:migrate",
- "prestart:prod": "NODE_ENV=production node_modules/.bin/sequelize db:migrate",
"start": "NODE_ENV=local nodemon .",
"start:dev": "NODE_ENV=development node index.js",
"start:stg": "NODE_ENV=staging node index.js",
"start:prod": "NODE_ENV=production node index.js",
Type of PR:
Enhancement
PR Description:
This PR introduces a new feature to the PR Reviewer tool, allowing it to calculate a score for each PR. The score is a float value between 0 and 100, representing the quality of the PR. The PR also includes updates to the configuration files to enable or disable this feature.
PR Main Files Walkthrough:
-
pr_agent/algo/utils.py: Added a new emoji for the score in the markdown conversion function.-
pr_agent/tools/pr_reviewer.py: Added a new variable 'require_score' to the PR Reviewer's variables, which is fetched from the settings.-
CONFIGURATION.md: Updated the documentation to include the new 'require_score_review' flag.-
pr_agent/settings/configuration.toml: Added a new flag 'require_score_review' to enable or disable the score review feature.-
pr_agent/settings/pr_reviewer_prompts.toml: Updated the prompts file to include a new 'Score' field in the JSON schema if the 'require_score' flag is set to true.