Skip to content

Commit

Permalink
feat: add new diagnostic check to validate the agent status (#126)
Browse files Browse the repository at this point in the history 
### Description

OB-32327 add new diagnostic check to validate the agent status, update
integration tests to separate config initialization from running the
diagnose check

### Checklist
- [x] Created tests which fail without the change (if possible)
- [ ] Extended the README / documentation, if necessary
  • Loading branch information
@obs-gh-mattcotter
obs-gh-mattcotter authored Nov 21, 2024
1 parent 42ca91a commit 9ade1cf
Show file tree
Hide file tree
Showing 25 changed files with 863 additions and 605 deletions.
121 changes: 52 additions & 69 deletions integration/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,35 +1,35 @@
## Integration Tests

# Integration Tests

The root of this module location is intended to run integration tests using the terraform test framework. The tests are located at `integration/tests`

The tests are run using the `terraform test -verbose` command from this folder `observe-agent/integration`
The tests are run using the `terraform test -verbose` command from this folder `observe-agent/integration`

When the above command is run, the tests in the `integration/tests` directory are ran using the variables provided. The tests are ran in the order of the run blocks provided in `<test>.tftest.hcl`
When the above command is run, the tests in the `integration/tests` directory are ran using the variables provided. The tests are ran in the order of the run blocks provided in `<test>.tftest.hcl`

Generally a test will do the following for any given EC2 Machine:

- Create a machine using the variables provided below in `us-west-1`
- Run a test using `observeinc/collection/aws//modules/testing/exec` module to accept python scripts located at `integration/tests/scripts`
- Run a test using `observeinc/collection/aws//modules/testing/exec` module to accept python scripts located at `integration/tests/scripts`

### Pre-requisites
## Pre-requisites

Ensure you have the following:
- Built version of the agent ( in `observe-repos/observe-agent/dist`) using `go-releaser`
Ensure you have the following:

- Built version of the agent ( in `observe-agent/dist`) using `go-releaser`
- Blunderdome Admin Access in AWS (used to assume the `gh-observe_agent-repo` role for testing )
- Observe Collection URL & Datastream Token to test with
- Generated Private & public key pair ( name to `test_key.pub` & `test_key.pem`)
- Observe Collection URL & Datastream Token to test with
- Generated Private & public key pair ( name to `test_key.pub` & `test_key.pem`)
- Terraform provider overide and terraform variables (see below section on how to do this)


**Building the agent**:

```
```sh
observe-agent git:(nikhil/update-RM) ✗ goreleaser release --snapshot --clean --verbose
```

If agent distributable is not built, you may get the following message:

```
```txt
│ Error: Test assertion failed
│
│ on tests/integration.tftest.hcl line 65, in run "test_install":
Expand All @@ -40,78 +40,69 @@ If agent distributable is not built, you may get the following message:
│ Error in Installation Test
```


**SSH Key Pairs**:
**SSH Key Pairs**:

Generate in PEM format for the OpenSSH Key that will be used by Terraform EC2 Modules:

```
```sh
ssh-keygen -m PEM
```

For more info on generating SSH keys, see [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws)

Ensure you have `./test_key.pub` and `./test_key.pem` in the `integration` directory. This can also be changed and specifed in the `integration/tests.auto.tfvars` file.

Ensure the extensions are correct!
Ensure you have `./test_key.pub` and `./test_key.pem` in the `integration` directory. This can also be changed and specifed in the `integration/tests.auto.tfvars` file.

Ensure the extensions are correct!

**AWS UI Access (optional)**:

For AWS UI access for viewing machines:
1. Login to Britive Blunderdome
For AWS UI access for viewing machines:

2. Navigate to AWS Console
1. Login to Britive Blunderdome

<p align="left">
<img src="screenshots/aws-console.png" alt="AWS Console" width="300"/>
</p>
2. Navigate to AWS Console

3. Click Switch role and input details for `nikhil-ps-account` which is the member account for integration testing within Blunderdome. Details are the following:
<p align="left">
<img src="screenshots/member-account.png" alt="Member Account" width="500"/>
</p>
![AWS Console](screenshots/aws-console.png)

4. You can now access EC2 machines and download key pairs (same as what Github Actions workflow uses)
3. Click Switch role and input details for `nikhil-ps-account` which is the member account for integration testing within Blunderdome. Details are the following:

5. Ensure you're in `us-west-1`
![Member Account](screenshots/member-account.png)

4. You can now access EC2 machines and download key pairs (same as what Github Actions workflow uses)

5. Ensure you're in `us-west-1`

### Terraform Variables
## Terraform Variables

The tests are run using the following variables. These can be set in the `integration/tests.auto.tfvars` file for local testing.
The tests are run using the following variables. These can be set in the `integration/tests.auto.tfvars` file for local testing.

```
name_format = "tf-observe-agent-test-%s"
AWS_MACHINE= "AMAZON_LINUX_2023" #Choose the AWS Machine to run the tests on
```terraform
name_format = "tf-observe-agent-test-%s"
AWS_MACHINE = "AMAZON_LINUX_2023" #Choose the AWS Machine to run the tests on
PUBLIC_KEY_PATH = "./test_key.pub" #Path to Public Key for EC2
PRIVATE_KEY_PATH = "./test_key.pem" #Path to Private Key for EC2
OBSERVE_URL = "https://<TENANT_ID.collect.observe-staging.com>" #Observe URL to use for testing
OBSERVE_TOKEN ="<DATASTREAM_TOKEN_TO_TEST_OBSERVE_AGENT_WITH">
```
OBSERVE_URL = "https://<TENANT_ID.collect.observe-staging.com>" #Observe URL to use for testing
OBSERVE_TOKEN = "<DATASTREAM_TOKEN_TO_TEST_OBSERVE_AGENT_WITH">
```

The PUBLIC & PRIVATE key pair can be generated by following the instructions in the "SSH Key Pairs" section above
The PUBLIC & PRIVATE key pair can be generated by following the instructions in the "SSH Key Pairs" section above

Example of what the `integrations` folder contains after creating `.*tfvars` file and SSH Key Pair:

![SSH Example](screenshots/ssh-example.png)]

<p align="left">
<img src="screenshots/ssh-example.png" alt="SSH Example" width="200"/>
</p>


### Terraform Provider
## Terraform Provider

Note: You must also set the provider correctly. We use the following settings:

- Region: `us-west-1`
- Profile: `blunderdome`
- IAM Role Assumed: `gh-observe_agent-repo`
- IAM Role Assumed: `gh-observe_agent-repo`
- The above role has permissions to create and destroy EC2 instances. See `modules/setup_aws_backend/role.tf` for more details.

The provider can be directly set in the `integration/tests/integration.tftest.hcl` as below:

```
```terraform
provider "aws" {
region = "us-west-1" # Specify the AWS region
profile = "blunderdome"
Expand All @@ -126,34 +117,28 @@ or through a `provider_override.tf` placed in `modules/create_ec2` directory.

Example of this:

<p align="left">
<img src="screenshots/provider-example.png" alt="SSH Example" width="200"/>
</p>



![Provider Example](screenshots/provider-example.png)

> [!NOTE]
> For Terraform to access and assume the role properly, you MUST be logged into Blunderdome Admin in console and have the correct permissions!
> For Terraform to access and assume the role properly, you MUST be logged into Blunderdome Admin in console and have the correct permissions!
Example of this:

```
```sh
observe git:(master) ✗ s/aws-creds checkout blunderdome

Checked out 'AWS Blunderdome Organization/460044344528 (observe-blunderdome)/BritiveBlunderdome-FullAWSAdmin' into awscli profile 'blunderdome'

observe git:(master) ✗ export AWS_PROFILE=blunderdome
```


### Local Testing (without terraform test)
## Local Testing (without terraform test)

Any of the python scripts in the `/scripts` directory can be tested by running them directly, granted an EC2 Machine exists. As the scripts rely on the outputs of `create_ec2` and `setup_observe_variables` modules to be passed in as environment variables, these environment variables can be manually set if the set up modules are not ran.

The `/scripts/<test_xyz.py` expects the following environment variables to be set:
The `/scripts/<test_xyz>.py` expects the following environment variables to be set:

```
```sh
HOST="54.177.249.99" #HOST IP Address
USER="ubuntu" #HOST user to login as
KEY_FILENAME="./test_key.pem" #Private path to key
Expand All @@ -162,19 +147,20 @@ MACHINE_CONFIG="ami_description:Ubuntu Server 22.04 LTS (HVM)- EBS General Purpo
OBSERVE_URL="" #Observe URL to use for testing
OBSERVE_TOKEN="" #Observe Token to use for testing
PASSWORD="WindowsPassword to be used for testing" # Set to None for testing
```

Run the scripts from the folder as below:
```

```sh
➜ integration git:(nikhil/integration-testing-windows) ✗ pwd
/Users/nikhil.dua/Documents/observe-repos/observe-agent/integration
➜ integration git:(nikhil/integration-testing-windows) ✗ python3 scripts/test_installation.py
➜ integration git:(nikhil/integration-testing-windows) ✗ python3 scripts/test_install.py
```

Note: If testing Windows machines, the RDP password is redacted by default in the python scripts.
Note: If testing Windows machines, the RDP password is redacted by default in the python scripts.
This can be turned off when disabling mask by setting below environment variable to `False` before running these scripts
```

```sh
export MASK=False
python3 scripts/test_ec2_connection.py
------------------------------
Expand All @@ -185,9 +171,6 @@ Env vars set to:
Testing SSH connection to host 54.177.26.178 with timeout 120s
```

### Architecture
## Architecture

The architecture diagram can be found ![here](screenshots/Observe-Agent.png)



Loading

0 comments on commit 9ade1cf

Please sign in to comment.