Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark last instance unhealthy if BuildkiteTerminateInstanceAfterJob is enabled #1245

Merged
merged 13 commits into from
Oct 23, 2023

Commits on Oct 20, 2023

  1. Correctly terminate single-job instances

    There's a situation that can occur when using the TerminateAfterJob mode of the agent, where when the ASG gets down to its MinSize, after running a job, the agent will attempt to self-terminate using the `aws autoscaling terminate-instance-in-autoscaling-group` command, but becuase that action would cause the ASG to go below its minsize, the call fails. This fails the entire systemd `ExecPostStop` hook, which fails the entire unit, which causes systemd to restart the unit.
    
    This means that in certain circumstances (when the instance attempted to self-terminate but the call failed), single-job instances can be caused to retain state from previous jobs, which is the opposite of what we want for these instances.
    
    This PR changes the autoterminate script so that if the instance is in TerminateAfterJob mode, if the call to terminate fails, the instance will call `shutdown` on itself, causing the instance to be removed from the ASG no matter what EC2 has to say about it
    moskyb authored and triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    4a5abec View commit details
    Browse the repository at this point in the history
  2. Fix missing equals

    moskyb authored and triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    7eec130 View commit details
    Browse the repository at this point in the history
  3. FFFFFFFF

    moskyb authored and triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    4e95386 View commit details
    Browse the repository at this point in the history
  4. Accursed plusses

    moskyb authored and triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    11a5832 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    916dfe1 View commit details
    Browse the repository at this point in the history
  6. Also set BUILDKITE_TERMINATE_INSTANCE_AFTER_JOB in cfn-env

    Then it will appear in the job logs, so customers will know it is set by
    the elastic stack.
    triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    cee477c View commit details
    Browse the repository at this point in the history
  7. Use systemd service overrides instead of systemctl set-env

    The latter affects every subsequently started service (at least). The
    former only affects the service that needs the env variable.
    triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    f05546d View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    d9fe204 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    f9d7731 View commit details
    Browse the repository at this point in the history
  10. Mark as unhealthy on windows

    triarius committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    d2927cf View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2023

  1. Configuration menu
    Copy the full SHA
    c3ebaa5 View commit details
    Browse the repository at this point in the history
  2. Remove unused condition

    triarius committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    5b02d95 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Configuration menu
    Copy the full SHA
    198db6e View commit details
    Browse the repository at this point in the history