Skip to content

infrastructure jenkins

Jump to bottom
Brian Barrett edited this page Apr 6, 2017 · 9 revisions

OMPI Jenkins Setup

The Open MPI community runs a Jenkins build server (and fleet of build workers) to check both pull requests and new commits. The server runs on aws.open-mpi.org, with builders running at a number of institutions as well as AWS. In addition, Mellanox and IBM run their own Jenkins master, as there is some significant level of trust between master and slave. This page does not document how to use Jenkins, but instead is notes on how Jenkins is currently configured. Brian and Howard have been doing most of the configuration lately.

EC2 Builders

EC2 Builders run in the Open MPI Production account (same place aws.open-mpi.org runs), using the Jenkins EC2 plugin, which does auto scaling and instance stop when idle and all that. The configuration is pretty straight forward, configured in the master configuration screen on Jenkins. aws.open-mpi.org has a role that allows all the permissions needed to modify EC2 instance state, meaning that we don't embed AWS creds in jenkins, but instead use the Use EC2 instance profile to obtain credentials option. Yay.

In general, we use CloudInit to run a script hosted in the ompi-scripts repo for configuring Linux AMIs, and do not build custom AMIs for Linux. The Linux AMIs all look the same today, so this document doesn't break them out. Today, we support Amazon Linux (multiple variants) and Ubuntu (multiple variants). The BSD instances generally use custom AMIs built off the "official" AMIs to work around challenges with their package installation / user-data hooks.

Linux Instances

Configuration:

Instance Type: T2Micro
Availability Zone: us-west-2c
Security group names: sg-f87d7a9e
Remote FS root: /home/DEFAULT_USER
Remote user: DEFAULT_USER
AMI Type: unix
Root command prefix: sudo
Remote ssh port: 22
Labels: ec2 linux DISTRO_VERSION
Idle termination time: -5
User Date:
TBD
Number of Executors: 1
Stop/Disconnect on Idle Timeout: True
Subnet ID for VPC: subnet-a9e154f1
Instance Cap: 5
IAM Instance Profile: arn:aws:iam::518752846868:instance-profile/jenkins-worker
Associate Public IP: True

The Idle termination time: -5 is not a typo. EC2 bills in hour increments. So if an instance gets stopped after running 30 minutes or after 59 minutes, it's the same cost. -5 means Jenkins won't stop an instance until 5 minutes before the hour is up, maximizing instance reuse if there are multiple builds in the hour.

FreeBSD

FreeBSD doesn't have a nice CloudInit hook, and it's FirstBoot hook doesn't play nice with the Openjdk package they distribute. The easiest solution was to build a custom AMI. As of FreeBSD 10.3, official RELEASE amis are part of the release announcement.

Current AMIs:

  • ami-b80096d8 (based on FreeBSD 11.0-RELEASE-amd64 (ami-6926f809))

To build a new AMI:

  • Create a t2.micro instance using the official RELEASE ami
  • run pkg install openjdk autoconf automake libtool gcc wget curl git as root on the AMI (default user: ec2-user)
  • Now's a good time to checkout / build Open MPI, just to verify everything's there:
  • Follow (these instructions)[http://www.daemonology.net/freebsd-on-ec2/] to create a new ami
    • Delete the file /root/.ssh/authorized_keys or /home/ec2-user/.ssh/authorized_keys (otherwise your SSH key will be able to log in to any instances launched from this AMI, and Jenkins won't be able to).
    • Create the file /firstboot on 10.0-RELEASE or /root/firstboot on earlier releases (otherwise the AMI you create won't know to download the new SSH public key for logins).
    • Stop the instance (but do not terminate it).
    • Use the ec2-create-image tool or the "Create Image" command in the EC2 Management Console to create an AMI from the instance.
    • Terminate the instance.

Configuration:

Instance Type: T2Micro
Availability Zone: us-west-2c
Security group names: sg-f87d7a9e
Remote FS root: /home/ec2-user
Remote user: ec2-user
AMI Type: unix
Root command prefix: su -m root -c
Remote ssh port: 22
Labels: ec2 freebsd freebsd_11.0
Idle termination time: -5
User Date:
TBD
Number of Executors: 1
Stop/Disconnect on Idle Timeout: True
Subnet ID for VPC: subnet-a9e154f1
Instance Cap: 5
IAM Instance Profile: arn:aws:iam::518752846868:instance-profile/jenkins-worker
Associate Public IP: True
Clone this wiki locally