Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: call db to retrieve permits instead of parsing github comments #164

Merged
merged 12 commits into from
Feb 21, 2024
Merged

feat: call db to retrieve permits instead of parsing github comments #164

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
35d0b99
feat: call db to retrieve permits instead of parsing github
gentlementlegen Feb 17, 2024
7528f3e
chore: issue number now uses the value from git get request
gentlementlegen Feb 17, 2024
2b21d66
Merge branch 'development' into permit-urls-from-db
Feb 17, 2024
655ead5
chore: fixed merged characters
gentlementlegen Feb 17, 2024
72b23f2
Merge remote-tracking branch 'origin/development' into permit-urls-fr…
gentlementlegen Feb 19, 2024
3b7c3d2
Merge remote-tracking branch 'pay.ubiquity/development' into permit-u…
gentlementlegen Feb 19, 2024
2f11234
chore: merged development branch
gentlementlegen Feb 19, 2024
0878552
chore: added Action secrets to the build.yml
gentlementlegen Feb 19, 2024
a1c6e4b
chore: renamed supabase key to SUPABASE_PUBLIC_ANON_KEY
gentlementlegen Feb 21, 2024
0ba50af
chore: renamed supabase key to SUPABASE_ANON_KEY
gentlementlegen Feb 21, 2024
5a504fd
chore: allow personal repo to be searched
gentlementlegen Feb 21, 2024
ba87b12
chore: renamed supabase env variables
gentlementlegen Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion build/esbuild-config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import * as dotenv from "dotenv";
import esbuild from "esbuild";
import { invertColors } from "./plugins/invert-colors";

const typescriptEntries = [
"static/scripts/rewards/index.ts",
"static/scripts/audit-report/audit.ts",
Expand All @@ -24,4 +25,19 @@ export let esBuildContext = {
".svg": "dataurl",
},
outdir: "static/out",
define: createEnvDefines(["SUPABASE_URL", "SUPABASE_KEY"]),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2024-02-17 at 16 26 50

Looks like we need to add this to our repo secrets?

Copy link
Member

@0x4007 0x4007 Feb 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the necessary values and tried rerunning the CI but it failed again. Perhaps you can try pushing to force it to run again? I have a feeling you may not be loading in the secrets correctly though. Did you prove this works on your local?

Screenshot 2024-02-17 at 16 29 24

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My local works fine, I'll have a look at the logs. Also it uses the Supabase you shared to me

} as esbuild.BuildOptions;

function createEnvDefines(variableNames: string[]): Record<string, string> {
const defines: Record<string, string> = {};
dotenv.config();
for (const name of variableNames) {
const envVar = process.env[name];
if (envVar !== undefined) {
defines[`process.env.${name}`] = JSON.stringify(envVar);
} else {
throw new Error(`Missing environment variable: ${name}`);
}
}
return defines;
}
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"@octokit/plugin-throttling": "^5.1.0",
"@octokit/rest": "^19.0.7",
"@sinclair/typebox": "^0.31.28",
"@supabase/supabase-js": "2.39.6",
"@types/libsodium-wrappers": "^0.7.10",
"@types/node": "^18.15.11",
"@uniswap/permit2-sdk": "^1.2.0",
Expand Down
236 changes: 70 additions & 166 deletions static/scripts/audit-report/audit.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
import { ethers } from "ethers";
import { Octokit } from "@octokit/rest";
import { throttling } from "@octokit/plugin-throttling";
import { Octokit } from "@octokit/rest";
import { createClient } from "@supabase/supabase-js";
import axios from "axios";
import { ethers } from "ethers";
import GoDB from "godb";
import { permit2Abi } from "../rewards/abis";
import { ObserverKeys, ElemInterface, QuickImport, StandardInterface, TxData, GoDBSchema, GitHubUrlParts, ChainScanResult, SavedData } from "./types";
import { Chain, ChainScan, DatabaseName, NULL_HASH, NULL_ID } from "./constants";
import {
getCurrency,
getGitHubUrlPartsArray,
getRandomAPIKey,
getRandomRpcUrl,
getOptimalRPC,
isValidUrl,
parseRepoUrl,
Expand All @@ -18,8 +19,12 @@ import {
RateLimitOptions,
secondaryRateLimitHandler,
} from "./helpers";
import { ChainScanResult, ElemInterface, GitHubUrlParts, GoDBSchema, ObserverKeys, QuickImport, SavedData, StandardInterface, TxData } from "./types";
=======
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you need to redo fixing the merge conflict

import { getTxInfo } from "./utils/getTransaction";

const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_KEY!);

const rateOctokit = Octokit.plugin(throttling);

let BOT_WALLET_ADDRESS = "";
Expand Down Expand Up @@ -292,129 +297,6 @@ class QueueSet {
const updateQueue = new smartQueue();
const rpcQueue = new QueueSet();

const commentFetcher = async () => {
if (isComment) {
const commentIntervalID = setInterval(async () => {
clearInterval(commentIntervalID);
try {
if (issueList.length !== 0) {
const octokit = new Octokit({
auth: GITHUB_PAT,
throttle: {
onRateLimit: (retryAfter, options) => {
return primaryRateLimitHandler(retryAfter, options as RateLimitOptions);
},
onSecondaryRateLimit: (retryAfter, options) => {
return secondaryRateLimitHandler(retryAfter, options as RateLimitOptions);
},
},
});
let [owner, repo] = parseRepoUrl(issueList[0].html_url);
const { data } = await octokit.rest.issues.listComments({
owner,
repo,
issue_number: issueList[0].number,
per_page: offset,
page: commentPageNumber,
});
if (data.length === 0) {
commentPageNumber = 1;
await issueList.shift();
if (issueList.length > 0) {
commentFetcher();
} else {
isComment = false;
finishedQueue.mutate("isComment", true);
}
} else {
let isFound = false;
for (let comment of data) {
if (comment.user && comment.user.node_id === botNodeId && comment.body) {
const match = comment.body.match(urlRegex);
if (match && isValidUrl(match[1])) {
const url = new URL(match[1]);
const params = new URLSearchParams(url.search);
const base64Payload = params.get("claim");
let network = getCurrency(comment.body); // Might change it to `const claimNetwork = params.get("network");` later because previous permits are missing network query
if (base64Payload) {
const {
owner: ownerAddress,
signature,
permit: {
deadline,
nonce,
permitted: { amount, token },
},
transferDetails: { to },
} = JSON.parse(window.atob(base64Payload)) as TxData;
await updateQueue.add(signature, {
k: signature,
t: "git",
c: {
nonce,
owner: ownerAddress,
token,
amount,
to,
deadline,
signature,
},
s: {
git: {
issue_title: issueList[0].title,
issue_number: issueList[0].number,
owner,
repo,
bounty_hunter: {
name: issueList[0].assignee ? issueList[0].assignee.login : issueList[0].assignees[0]?.login,
url: issueList[0].assignee ? issueList[0].assignee.html_url : issueList[0].assignees[0]?.html_url,
},
},
ether: undefined,
network: network as string,
},
});
isFound = true;
}
} else {
console.log("URL not found, skipping");
}
}
}

if (isFound) {
commentPageNumber = 1;
await issueList.shift();
} else {
commentPageNumber++;
}

if (issueList.length > 0) {
commentFetcher();
} else {
isComment = false;
finishedQueue.mutate("isComment", true);
}
}
} else {
isComment = false;
finishedQueue.mutate("isComment", true);
}
} catch (error) {
console.error(error);
finishedQueue.raise();
await issueList.shift();
if (issueList.length > 0) {
commentFetcher();
} else {
isComment = false;
finishedQueue.mutate("isComment", true);
}
}
}, GIT_INTERVAL);
}
};

const gitFetcher = async (repoUrls: GitHubUrlParts[]) => {
if (isGit) {
const octokit = new rateOctokit({
Expand All @@ -432,45 +314,26 @@ const gitFetcher = async (repoUrls: GitHubUrlParts[]) => {
const getIssuesForRepo = async (owner: string, repo: string) => {
const issueList: any[] = [];

while (true) {
try {
const { data } = await octokit.rest.issues.listForRepo({
owner,
repo,
state: "closed",
per_page: offset,
page: gitPageNumber,
});

if (data.length === 0) break;

const issues = data.filter(issue => !issue.pull_request && issue.comments > 0);
if (issues.length > 0) {
if (!lastGitID) {
lastGitID = issues[0].number;
}
let iEF = true;
for (let i of issues) {
if (i.number !== gitID) {
await issueList.push(i);
} else {
iEF = false;
break;
}
}

if (iEF) {
gitPageNumber++;
} else {
break;
}
} else {
break;
try {
const { data: gitData } = await octokit.rest.repos.get({
owner,
repo,
});
if (!gitData.organization) {
console.warn(`${owner}/${repo} is not an organization, skipping.`);
} else {
const { data } = await supabase
.from("permits")
.select("*, locations(*), users(*, wallets(*)), tokens(*)")
.eq("locations.organization_id", gitData.organization?.id)
.not("locations", "is", null);
if (data) {
issueList.push(...data.map(d => ({ ...d, owner, repo })));
}
} catch (error: any) {
console.error(error);
throw error;
}
} catch (error: any) {
console.error(error);
throw error;
}

return issueList;
Expand All @@ -483,14 +346,56 @@ const gitFetcher = async (repoUrls: GitHubUrlParts[]) => {
for (let i = 0; i < allIssues.length; i++) {
const issues = allIssues[i];
issueList.push(...issues);
console.log(`Fetched ${issues.length} issues for repository ${repoUrls[i].owner}/${repoUrls[i].repo}`);
console.log(`Fetched ${issues.length} permits for repository ${repoUrls[i].owner}/${repoUrls[i].repo}`);
}

isGit = false;
finishedQueue.mutate("isGit", true);
commentFetcher();
for (const issue of issueList) {
const { data: userData } = await octokit.request("GET /user/:id", { id: issue.locations.user_id });
const { data } = await supabase.from("locations").select("*").eq("issue_id", issue.locations.issue_id).single();
const lastSlashIndex = data.node_url.lastIndexOf("/");
const hashIndex = data.node_url.lastIndexOf("#") || data.node_url.length;
const issueNumber = Number(data.node_url.substring(lastSlashIndex + 1, hashIndex));
const { data: issueData } = await octokit.rest.issues.get({
issue_number: issueNumber,
owner: issue.owner,
repo: issue.repo,
});
updateQueue.add(issue.signature, {
c: {
amount: issue.amount,
deadline: issue.deadline,
nonce: issue.nonce,
owner: issue.owner,
signature: issue.signature,
to: issue.users.wallets.address,
token: issue.tokens.address,
},
k: issue.signature,
s: {
ether: undefined,
git: {
bounty_hunter: {
name: userData.login,
url: userData.html_url,
},
issue_number: issueData.number,
issue_title: issueData.title,
owner: issue.owner,
repo: issue.repo,
},
network: getCurrency(issue.network_id)!,
},
t: "git",
});
}
isComment = false;
finishedQueue.mutate("isComment", true);
} catch (error: any) {
console.error("Error fetching issues:", error);
isComment = false;
finishedQueue.mutate("isComment", true);
}
}
};
Expand Down Expand Up @@ -567,7 +472,6 @@ const rpcFetcher = async () => {
const data = await rpcQueue.read();
if (data) {
const { hash, chain } = data;

const providerUrl = await getOptimalRPC(chain);
const txInfo = await getTxInfo(hash, providerUrl, chain);

Expand Down
6 changes: 3 additions & 3 deletions static/scripts/audit-report/helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,10 +177,10 @@ export const isValidUrl = (urlString: string) => {
}
};

export const getCurrency = (comment: string) => {
if (comment.includes("WXDAI")) {
export const getCurrency = (id: number) => {
if (id === 100) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be handled dynamically I think. We should be able to support any network and any token.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem on using it as is, but it seems to be converted into a string value corresponding to its currency, so I have no clue if this value impacts any other project, so please let me know and I can use the number instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no clue if this value impacts any other project

I don't understand what you're saying but network 100 is Gnosis Chain which we use for most of our payouts now. However we want partners to be able to use any network (just as long as there is a permit2 smart contract deployment on it, it should be fine!)

return Chain.Gnosis;
} else if (comment.includes("DAI")) {
} else if (id === 1) {
return Chain.Ethereum;
}
return null;
Expand Down
Loading