Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script to remove duplicate issues on declarations repository #1115

Merged
merged 8 commits into from
Oct 29, 2024

Conversation

Ndpnt
Copy link
Member

@Ndpnt Ndpnt commented Oct 29, 2024

Output example:

↳ node scripts/reporter/duplicate/index.js
Getting issues from repository OpenTermsArchive/contrib-declarations…
Found 2074 issues

Found 24 duplicates for issue #1049 "`CafeMom` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #3706: https://github.com/OpenTermsArchive/contrib-declarations/issues/3706
Closed issue #3499: https://github.com/OpenTermsArchive/contrib-declarations/issues/3499
Closed issue #3454: https://github.com/OpenTermsArchive/contrib-declarations/issues/3454
Closed issue #3440: https://github.com/OpenTermsArchive/contrib-declarations/issues/3440
Closed issue #3410: https://github.com/OpenTermsArchive/contrib-declarations/issues/3410
Closed issue #3336: https://github.com/OpenTermsArchive/contrib-declarations/issues/3336
Closed issue #3164: https://github.com/OpenTermsArchive/contrib-declarations/issues/3164
Closed issue #3138: https://github.com/OpenTermsArchive/contrib-declarations/issues/3138
Closed issue #2962: https://github.com/OpenTermsArchive/contrib-declarations/issues/2962
Closed issue #2933: https://github.com/OpenTermsArchive/contrib-declarations/issues/2933
Closed issue #2920: https://github.com/OpenTermsArchive/contrib-declarations/issues/2920
Closed issue #2841: https://github.com/OpenTermsArchive/contrib-declarations/issues/2841
Closed issue #2782: https://github.com/OpenTermsArchive/contrib-declarations/issues/2782
Closed issue #2741: https://github.com/OpenTermsArchive/contrib-declarations/issues/2741
Closed issue #2718: https://github.com/OpenTermsArchive/contrib-declarations/issues/2718
Closed issue #2626: https://github.com/OpenTermsArchive/contrib-declarations/issues/2626
Closed issue #2568: https://github.com/OpenTermsArchive/contrib-declarations/issues/2568
Closed issue #2407: https://github.com/OpenTermsArchive/contrib-declarations/issues/2407
Closed issue #2390: https://github.com/OpenTermsArchive/contrib-declarations/issues/2390
Closed issue #2340: https://github.com/OpenTermsArchive/contrib-declarations/issues/2340
Closed issue #2159: https://github.com/OpenTermsArchive/contrib-declarations/issues/2159
Closed issue #2129: https://github.com/OpenTermsArchive/contrib-declarations/issues/2129
Closed issue #2074: https://github.com/OpenTermsArchive/contrib-declarations/issues/2074
Closed issue #1860: https://github.com/OpenTermsArchive/contrib-declarations/issues/1860

[…]

Found 3 duplicates for issue #1127 "`USA.gov` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #2062: https://github.com/OpenTermsArchive/contrib-declarations/issues/2062
Closed issue #1796: https://github.com/OpenTermsArchive/contrib-declarations/issues/1796
Closed issue #1691: https://github.com/OpenTermsArchive/contrib-declarations/issues/1691

Found 4 duplicates for issue #1131 "`Vinted` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #2018: https://github.com/OpenTermsArchive/contrib-declarations/issues/2018
Closed issue #1798: https://github.com/OpenTermsArchive/contrib-declarations/issues/1798
Closed issue #1722: https://github.com/OpenTermsArchive/contrib-declarations/issues/1722
Closed issue #1644: https://github.com/OpenTermsArchive/contrib-declarations/issues/1644

Found 5 duplicates for issue #1133 "`Visible` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #2017: https://github.com/OpenTermsArchive/contrib-declarations/issues/2017
Closed issue #1879: https://github.com/OpenTermsArchive/contrib-declarations/issues/1879
Closed issue #1700: https://github.com/OpenTermsArchive/contrib-declarations/issues/1700
Closed issue #1646: https://github.com/OpenTermsArchive/contrib-declarations/issues/1646
Closed issue #1401: https://github.com/OpenTermsArchive/contrib-declarations/issues/1401

Found 3 duplicates for issue #1561 "`Twitter` ‧ `Trackers Policy` ‧ not tracked anymore"
Closed issue #2011: https://github.com/OpenTermsArchive/contrib-declarations/issues/2011
Closed issue #1791: https://github.com/OpenTermsArchive/contrib-declarations/issues/1791
Closed issue #1693: https://github.com/OpenTermsArchive/contrib-declarations/issues/1693

Found 3 duplicates for issue #1562 "`Twitter` ‧ `Copyright Claims Policy` ‧ not tracked anymore"
Closed issue #2009: https://github.com/OpenTermsArchive/contrib-declarations/issues/2009
Closed issue #1831: https://github.com/OpenTermsArchive/contrib-declarations/issues/1831
Closed issue #1792: https://github.com/OpenTermsArchive/contrib-declarations/issues/1792

Found 7 duplicates for issue #1149 "`Webs` ‧ `Terms of Service` ‧ not tracked anymore"
Closed issue #1975: https://github.com/OpenTermsArchive/contrib-declarations/issues/1975
Closed issue #1931: https://github.com/OpenTermsArchive/contrib-declarations/issues/1931
Closed issue #1880: https://github.com/OpenTermsArchive/contrib-declarations/issues/1880
Closed issue #1799: https://github.com/OpenTermsArchive/contrib-declarations/issues/1799
Closed issue #1728: https://github.com/OpenTermsArchive/contrib-declarations/issues/1728
Closed issue #1697: https://github.com/OpenTermsArchive/contrib-declarations/issues/1697
Closed issue #1647: https://github.com/OpenTermsArchive/contrib-declarations/issues/1647

Found 1 duplicates for issue #1126 "`ucla.edu` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #1694: https://github.com/OpenTermsArchive/contrib-declarations/issues/1694

Found 1 duplicates for issue #1138 "`Windstream` ‧ `Terms of Service` ‧ not tracked anymore"
Closed issue #1672: https://github.com/OpenTermsArchive/contrib-declarations/issues/1672

Found 1 duplicates for issue #1139 "`Windstream` ‧ `Privacy Policy` ‧ not tracked anymore"
Closed issue #1648: https://github.com/OpenTermsArchive/contrib-declarations/issues/1648

Found 1 duplicates for issue #1119 "`The Reach Group` ‧ `Terms of Service` ‧ not tracked anymore"
Closed issue #1398: https://github.com/OpenTermsArchive/contrib-declarations/issues/1398

Duplicate removal process completed; 1882 issues closed

@Ndpnt Ndpnt requested review from clementbiron and MattiSG October 29, 2024 10:13
@Ndpnt
Copy link
Member Author

Ndpnt commented Oct 29, 2024

Example result: OpenTermsArchive/contrib-declarations#3499

Copy link
Member

@clementbiron clementbiron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@@ -2,6 +2,14 @@

All changes that impact users of this module are documented in this file, in the [Common Changelog](https://common-changelog.org) format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased [minor]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say this is a no-release considering there is strictly no change in behavior exposed to reusers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that initially, but if we ask our partners to update to the latest version to access this script, they won’t be able to do so if there’s no official release available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, true 😅

scripts/reporter/duplicate/README.md Outdated Show resolved Hide resolved
scripts/reporter/duplicate/README.md Outdated Show resolved Hide resolved
OTA_ENGINE_GITHUB_TOKEN=your_github_token
```

2. Configure the target repository in `config/development.json`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it really have to be in development? Or just in the environment that will be loaded at config loading time?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be in any config file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

continue;
}

await octokit.request('PATCH /repos/{owner}/{repo}/issues/{issue_number}', { /* eslint-disable-line no-await-in-loop */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we await? Couldn't this be done fully asynchronously?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could, but I find logs much easier to read when they’re sequential. Since we need to not send our requests in parallel to avoid hitting GitHub’s rate limit, I opted for a setup that maintains clear, readable output

Comment on lines 56 to 57
state: 'closed',
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't we use state_reason to avoid the following comment request? 🙂

Copy link
Member Author

@Ndpnt Ndpnt Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is not what you think:

state_reason string or null
The reason for the state change. Ignored unless state is changed.
Can be one of: completed, not_planned, reopened, null

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, sorry.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could still set not_planned then 😉

scripts/reporter/duplicate/index.js Outdated Show resolved Hide resolved

console.log(`\nDuplicate removal process completed; ${counter} issues closed`);
} catch (error) {
console.log(`Failed to remove duplicate issues: ${error.stack}`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
console.log(`Failed to remove duplicate issues: ${error.stack}`);
console.log(`Failed to remove some duplicate issues: ${error.stack}`);

}

console.log(`\nDuplicate removal process completed; ${counter} issues closed`);
} catch (error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do a try / catch and exit instead of simply not catching any error, since we call the function from the main event loop anyway? 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of habit. But it can be removed.

Copy link
Member

@MattiSG MattiSG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would still be happy to see my comments handled, but nothing blocking 🙂

@Ndpnt Ndpnt merged commit 7f64e98 into main Oct 29, 2024
8 checks passed
@Ndpnt Ndpnt deleted the script-delete-duplicate-issues branch October 29, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants