Skip to content

feat: add new diagnostic check to validate the agent status #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 21, 2024
Merged

feat: add new diagnostic check to validate the agent status #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d5337c2
feat: add diagnostic check to validate agent status, update integrati…
obs-gh-mattcotter Oct 30, 2024
4ea844c
rm comment, clean up template text, unify whitespace in check output
obs-gh-mattcotter Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 52 additions & 69 deletions integration/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,35 @@
## Integration Tests

# Integration Tests

The root of this module location is intended to run integration tests using the terraform test framework. The tests are located at `integration/tests`

The tests are run using the `terraform test -verbose` command from this folder `observe-agent/integration`
The tests are run using the `terraform test -verbose` command from this folder `observe-agent/integration`

When the above command is run, the tests in the `integration/tests` directory are ran using the variables provided. The tests are ran in the order of the run blocks provided in `<test>.tftest.hcl`
When the above command is run, the tests in the `integration/tests` directory are ran using the variables provided. The tests are ran in the order of the run blocks provided in `<test>.tftest.hcl`

Generally a test will do the following for any given EC2 Machine:

- Create a machine using the variables provided below in `us-west-1`
- Run a test using `observeinc/collection/aws//modules/testing/exec` module to accept python scripts located at `integration/tests/scripts`
- Run a test using `observeinc/collection/aws//modules/testing/exec` module to accept python scripts located at `integration/tests/scripts`

### Pre-requisites
## Pre-requisites

Ensure you have the following:
- Built version of the agent ( in `observe-repos/observe-agent/dist`) using `go-releaser`
Ensure you have the following:

- Built version of the agent ( in `observe-agent/dist`) using `go-releaser`
- Blunderdome Admin Access in AWS (used to assume the `gh-observe_agent-repo` role for testing )
- Observe Collection URL & Datastream Token to test with
- Generated Private & public key pair ( name to `test_key.pub` & `test_key.pem`)
- Observe Collection URL & Datastream Token to test with
- Generated Private & public key pair ( name to `test_key.pub` & `test_key.pem`)
- Terraform provider overide and terraform variables (see below section on how to do this)


**Building the agent**:

```
```sh
observe-agent git:(nikhil/update-RM) ✗ goreleaser release --snapshot --clean --verbose
```

If agent distributable is not built, you may get the following message:

```
```txt
│ Error: Test assertion failed
│
│ on tests/integration.tftest.hcl line 65, in run "test_install":
Expand All @@ -40,78 +40,69 @@ If agent distributable is not built, you may get the following message:
│ Error in Installation Test
```


**SSH Key Pairs**:
**SSH Key Pairs**:

Generate in PEM format for the OpenSSH Key that will be used by Terraform EC2 Modules:

```
```sh
ssh-keygen -m PEM
```

For more info on generating SSH keys, see [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws)

Ensure you have `./test_key.pub` and `./test_key.pem` in the `integration` directory. This can also be changed and specifed in the `integration/tests.auto.tfvars` file.

Ensure the extensions are correct!
Ensure you have `./test_key.pub` and `./test_key.pem` in the `integration` directory. This can also be changed and specifed in the `integration/tests.auto.tfvars` file.

Ensure the extensions are correct!

**AWS UI Access (optional)**:

For AWS UI access for viewing machines:
1. Login to Britive Blunderdome
For AWS UI access for viewing machines:

2. Navigate to AWS Console
1. Login to Britive Blunderdome

<p align="left">
<img src="screenshots/aws-console.png" alt="AWS Console" width="300"/>
</p>
2. Navigate to AWS Console

3. Click Switch role and input details for `nikhil-ps-account` which is the member account for integration testing within Blunderdome. Details are the following:
<p align="left">
<img src="screenshots/member-account.png" alt="Member Account" width="500"/>
</p>
![AWS Console](screenshots/aws-console.png)

4. You can now access EC2 machines and download key pairs (same as what Github Actions workflow uses)
3. Click Switch role and input details for `nikhil-ps-account` which is the member account for integration testing within Blunderdome. Details are the following:

5. Ensure you're in `us-west-1`
![Member Account](screenshots/member-account.png)

4. You can now access EC2 machines and download key pairs (same as what Github Actions workflow uses)

5. Ensure you're in `us-west-1`

### Terraform Variables
## Terraform Variables

The tests are run using the following variables. These can be set in the `integration/tests.auto.tfvars` file for local testing.
The tests are run using the following variables. These can be set in the `integration/tests.auto.tfvars` file for local testing.

```
name_format = "tf-observe-agent-test-%s"
AWS_MACHINE= "AMAZON_LINUX_2023" #Choose the AWS Machine to run the tests on
```terraform
name_format = "tf-observe-agent-test-%s"
AWS_MACHINE = "AMAZON_LINUX_2023" #Choose the AWS Machine to run the tests on
PUBLIC_KEY_PATH = "./test_key.pub" #Path to Public Key for EC2
PRIVATE_KEY_PATH = "./test_key.pem" #Path to Private Key for EC2
OBSERVE_URL = "https://<TENANT_ID.collect.observe-staging.com>" #Observe URL to use for testing
OBSERVE_TOKEN ="<DATASTREAM_TOKEN_TO_TEST_OBSERVE_AGENT_WITH">
```
OBSERVE_URL = "https://<TENANT_ID.collect.observe-staging.com>" #Observe URL to use for testing
OBSERVE_TOKEN = "<DATASTREAM_TOKEN_TO_TEST_OBSERVE_AGENT_WITH">
```

The PUBLIC & PRIVATE key pair can be generated by following the instructions in the "SSH Key Pairs" section above
The PUBLIC & PRIVATE key pair can be generated by following the instructions in the "SSH Key Pairs" section above

Example of what the `integrations` folder contains after creating `.*tfvars` file and SSH Key Pair:

![SSH Example](screenshots/ssh-example.png)]

<p align="left">
<img src="screenshots/ssh-example.png" alt="SSH Example" width="200"/>
</p>


### Terraform Provider
## Terraform Provider

Note: You must also set the provider correctly. We use the following settings:

- Region: `us-west-1`
- Profile: `blunderdome`
- IAM Role Assumed: `gh-observe_agent-repo`
- IAM Role Assumed: `gh-observe_agent-repo`
- The above role has permissions to create and destroy EC2 instances. See `modules/setup_aws_backend/role.tf` for more details.

The provider can be directly set in the `integration/tests/integration.tftest.hcl` as below:

```
```terraform
provider "aws" {
region = "us-west-1" # Specify the AWS region
profile = "blunderdome"
Expand All @@ -126,34 +117,28 @@ or through a `provider_override.tf` placed in `modules/create_ec2` directory.

Example of this:

<p align="left">
<img src="screenshots/provider-example.png" alt="SSH Example" width="200"/>
</p>



![Provider Example](screenshots/provider-example.png)

> [!NOTE]
> For Terraform to access and assume the role properly, you MUST be logged into Blunderdome Admin in console and have the correct permissions!
> For Terraform to access and assume the role properly, you MUST be logged into Blunderdome Admin in console and have the correct permissions!

Example of this:

```
```sh
observe git:(master) ✗ s/aws-creds checkout blunderdome

Checked out 'AWS Blunderdome Organization/460044344528 (observe-blunderdome)/BritiveBlunderdome-FullAWSAdmin' into awscli profile 'blunderdome'

observe git:(master) ✗ export AWS_PROFILE=blunderdome
```


### Local Testing (without terraform test)
## Local Testing (without terraform test)

Any of the python scripts in the `/scripts` directory can be tested by running them directly, granted an EC2 Machine exists. As the scripts rely on the outputs of `create_ec2` and `setup_observe_variables` modules to be passed in as environment variables, these environment variables can be manually set if the set up modules are not ran.

The `/scripts/<test_xyz.py` expects the following environment variables to be set:
The `/scripts/<test_xyz>.py` expects the following environment variables to be set:

```
```sh
HOST="54.177.249.99" #HOST IP Address
USER="ubuntu" #HOST user to login as
KEY_FILENAME="./test_key.pem" #Private path to key
Expand All @@ -162,19 +147,20 @@ MACHINE_CONFIG="ami_description:Ubuntu Server 22.04 LTS (HVM)- EBS General Purpo
OBSERVE_URL="" #Observe URL to use for testing
OBSERVE_TOKEN="" #Observe Token to use for testing
PASSWORD="WindowsPassword to be used for testing" # Set to None for testing

```

Run the scripts from the folder as below:
```

```sh
➜ integration git:(nikhil/integration-testing-windows) ✗ pwd
/Users/nikhil.dua/Documents/observe-repos/observe-agent/integration
➜ integration git:(nikhil/integration-testing-windows) ✗ python3 scripts/test_installation.py
➜ integration git:(nikhil/integration-testing-windows) ✗ python3 scripts/test_install.py
```

Note: If testing Windows machines, the RDP password is redacted by default in the python scripts.
Note: If testing Windows machines, the RDP password is redacted by default in the python scripts.
This can be turned off when disabling mask by setting below environment variable to `False` before running these scripts
```

```sh
export MASK=False
python3 scripts/test_ec2_connection.py
------------------------------
Expand All @@ -185,9 +171,6 @@ Env vars set to:
Testing SSH connection to host 54.177.26.178 with timeout 120s
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More spacing/formatting in this file? @obs-gh-mattcotter

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I just used markdownlint plus a few minor changes;

  • scripts/test_installation.py was a typo (or missed name update) for scripts/test_install.py
  • removed reference to your local observe-repos folder
  • switched images from html to md


### Architecture
## Architecture

The architecture diagram can be found ![here](screenshots/Observe-Agent.png)



Loading
Loading