Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: added more detailed logging for scaling up and down #1222

Merged
merged 1 commit into from
Oct 1, 2021
Merged

fix: added more detailed logging for scaling up and down #1222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion modules/runners/lambdas/runners/src/scale-runners/gh-auth.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { createOctoClient, createGithubAppAuth, createGithubInstallationAuth } from './gh-auth';
import { createOctoClient, createGithubAppAuth } from './gh-auth';
import nock from 'nock';
import { createAppAuth } from '@octokit/auth-app';

Expand Down
10 changes: 8 additions & 2 deletions modules/runners/lambdas/runners/src/scale-runners/scale-down.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ async function evaluateAndRemoveRunners(

for (const ownerTag of ownerTags) {
const ec2RunnersFiltered = ec2Runners.filter((runner) => runner.owner === ownerTag);
console.debug(`Found: '${ec2RunnersFiltered.length}' active GitHub runners with owner tag: '${ownerTag}'`);
for (const ec2Runner of ec2RunnersFiltered) {
const ghRunners = await listGitHubRunners(ec2Runner);
const ghRunner = ghRunners.find((runner) => runner.name === ec2Runner.instanceId);
Expand Down Expand Up @@ -190,15 +191,20 @@ export async function scaleDown(): Promise<void> {

// list and sort runners, newest first. This ensure we keep the newest runners longer.
const ec2Runners = await listAndSortRunners(environment);
const activeEc2RunnersCount = ec2Runners.length;
console.info(`Found: '${activeEc2RunnersCount}' active GitHub EC2 runner instances before clean-up.`);

if (ec2Runners.length === 0) {
if (activeEc2RunnersCount === 0) {
console.debug(`No active runners found for environment: '${environment}'`);
return;
}
const legacyRunners = filterLegacyRunners(ec2Runners);
console.log(JSON.stringify(legacyRunners));
console.debug(JSON.stringify(legacyRunners));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this debug logging intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only changed it to debug. I assume is wad added with the intentio to see if still legacy runners needs to be termimnated by @mcaulifn

const runners = filterRunners(ec2Runners);

await evaluateAndRemoveRunners(runners, scaleDownConfigs);
await evaluateAndRemoveRunners(legacyRunners, scaleDownConfigs);

const activeEc2RunnersCountAfter = (await listAndSortRunners(environment)).length;
console.info(`Found: '${activeEc2RunnersCountAfter}' active GitHub EC2 runners instances after clean-up.`);
}
7 changes: 4 additions & 3 deletions modules/runners/lambdas/runners/src/scale-runners/scale-up.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,14 @@ async function getJobStatus(githubInstallationClient: Octokit, payload: ActionRe
export async function createRunnerLoop(runnerParameters: RunnerInputParameters): Promise<void> {
const launchTemplateNames = process.env.LAUNCH_TEMPLATE_NAME?.split(',') as string[];
let launched = false;
for (const launchTemplateName of launchTemplateNames) {
console.info(`Attempting to launch instance using ${launchTemplateName}.`);
for (let i = 0; i < launchTemplateNames.length; i++) {
console.info(`Attempt '${i}' to launch instance using ${launchTemplateNames[i]}.`);
try {
await createRunner(runnerParameters, launchTemplateName);
await createRunner(runnerParameters, launchTemplateNames[i]);
launched = true;
break;
} catch (error) {
console.debug(`Attempt '${i}' to launch instance using ${launchTemplateNames[i]} FAILED.`);
console.error(error);
}
}
Expand Down