Skip to content

Commit

Permalink
feat: ec2 instance will now be started when there are running builds
Browse files Browse the repository at this point in the history
Allows the EC2 Instance to correctly start if it is currently stopped,
but a build has become stuck in the 'running' state. This typically
occurs when a build begins in the 'dead zone' between an EC2 Instance
beginning the shutdown, but before the Drone Worker process is
terminated.

This patch should allow recovery from this state by starting the EC2
instance back up again to resume the job.
  • Loading branch information
wms committed May 2, 2019
1 parent 10b0b7f commit b344569
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 1 deletion.
5 changes: 5 additions & 0 deletions src/plan.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,11 @@ describe('plan', () => {
instanceState: 'stopped',
name: 'builds are pending and EC2 instance is stopped',
queue: [{ status: 'pending' }],
}, {
action: plan.Action.Start,
instanceState: 'stopped',
name: 'a build is stuck in the `running` state but the EC2 instance is stopped',
queue: [{ status: 'pending' }, { status: 'pending' }, { status: 'running' }],
}, {
action: plan.Action.ScheduleStop,
instanceState: 'running',
Expand Down
2 changes: 1 addition & 1 deletion src/plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ export const determineNextAction =
}
}

if (droneQueueState === DroneQueueState.Pending) {
if (droneQueueState === DroneQueueState.Pending || droneQueueState === DroneQueueState.Running) {
if (ec2InstanceState === Ec2InstanceState.Stopping || ec2InstanceState === Ec2InstanceState.Stopped) {
const nextAction = Action.Start;
log.debug({ phase: 'finish', nextAction });
Expand Down

0 comments on commit b344569

Please sign in to comment.