Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Install LME in the testbed from a single script #150

Merged
merged 105 commits into from
Feb 8, 2024
Merged

Install LME in the testbed from a single script #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
f37af1e
Adding the configure scripts
Dec 19, 2023
a93311f
Add scripts to zip and copy to a container for downloading in the server
Dec 19, 2023
2d5d146
Grab the expiry time properly in copy file
Dec 19, 2023
d2d3413
Overwrite the blob if it exists
Dec 19, 2023
89ddba1
Add the script to download file into DC
Dec 19, 2023
1d62f03
Script that unzips the files in a container
Dec 19, 2023
8f8ab45
Adds username argument to download files
Dec 20, 2023
eb59d3a
Add script to run scripts in container
Dec 20, 2023
67dbfa6
Adds username argument to gpo script
Dec 20, 2023
072f2be
Modifies the url name in the client GPO
Dec 20, 2023
11d479d
Adds the functionality for chapter 1 and first half of chapter 2
Dec 21, 2023
ea0ba16
Imports the sysmon GPO
Dec 22, 2023
daf3018
Update the variables for sysmon gpo
Dec 22, 2023
fe5ca77
Name the scripts so they are grouped together in a listing
Dec 22, 2023
26cad54
Echos the file download url
Dec 26, 2023
3b80b44
Expands the domain name correctly in create ou
Dec 26, 2023
5ad0a5e
Write the url output of copy file to container to a different output …
Dec 26, 2023
8f71057
Create a new LME folder for our scripts and files
Dec 26, 2023
55f76af
Set path for extract to lme
Dec 26, 2023
70a2754
Update paths for scripts to /lme
Dec 26, 2023
68e8e2b
Fix the wec server name setting
Dec 27, 2023
6265149
Adds the scripts to install chapter 1 and 2
Dec 28, 2023
c89aa88
Allows azure to download in linux and windows
Dec 28, 2023
3047296
Adds linux install scripts.
Dec 28, 2023
5c12a1f
Adds winlogbeat installer
Dec 29, 2023
2daf25b
emove garbage in update server name
Dec 29, 2023
eaea87c
Tweak several scripts to get the scp of files_for_windows
Jan 3, 2024
a4467fd
Adds installer script to run all the scripts
Jan 4, 2024
03444bf
Fixes the formatting method for az output
Jan 4, 2024
486a2a1
Clean up the scripts and add documentation
Jan 5, 2024
54c3a0b
Fixes outputting format errors
Jan 5, 2024
b4e2d87
Fixes hanging on adding ls1 to domain
Jan 5, 2024
ac65188
Fix formatting errors on responses
Jan 8, 2024
61a1085
Update linux expect script for different prompts.
Jan 9, 2024
89552ab
Handle the reboot message for linux expect script
Jan 10, 2024
e398c0a
Echos the file download url
Dec 26, 2023
65a347e
Create a new LME folder for our scripts and files
Dec 26, 2023
f74a823
Set path for extract to lme
Dec 26, 2023
e4609c5
Update paths for scripts to /lme
Dec 26, 2023
ca696d8
Update paths for scripts to /lme
Dec 26, 2023
a49f985
Fix the wec server name setting
Dec 27, 2023
b2feae7
Adds the scripts to install chapter 1 and 2
Dec 28, 2023
1222979
Allows azure to download in linux and windows
Dec 28, 2023
b8efb21
Adds linux install scripts.
Dec 28, 2023
ec19b8d
Adds winlogbeat installer
Dec 29, 2023
eda0bfe
emove garbage in update server name
Dec 29, 2023
4160a75
Tweak several scripts to get the scp of files_for_windows
Jan 3, 2024
91b8e1c
Adds installer script to run all of the scripts
Jan 4, 2024
38823a9
Fixes the formatting method for az output
Jan 4, 2024
9d9afa0
Clean up the scripts and add documentation
Jan 5, 2024
2574f6b
Fixes outputting format errors
Jan 5, 2024
246b6fa
Fixes hanging on adding ls1 to domain
Jan 5, 2024
f8d2344
Fix formatting errors on responses
Jan 8, 2024
82cc7b4
Update linux expect script for different prompts.
Jan 9, 2024
f7d6965
Handle the reboot message for linux expect script
Jan 10, 2024
6888fc0
Adds InstallTestbed instructions to Readme.md
Jan 12, 2024
acf6cc9
Adds InstallTestbed instructions to Readme.md
Jan 12, 2024
71c448c
Adds InstallTestbed instructions to Readme.md
Jan 12, 2024
b6ff436
Adds InstallTestbed instructions to Readme.md
Jan 12, 2024
0a49480
Adds InstallTestbed instructions to Readme.md
Jan 12, 2024
5f09bfb
Adds InstallTestbed instructions to Readme.md
Jan 13, 2024
afe7589
Adds InstallTestbed instructions to Readme.md
Jan 13, 2024
2a34842
Adds InstallTestbed instructions to Readme.md
Jan 13, 2024
a92125a
Adds InstallTestbed instructions to Readme.md
Jan 13, 2024
a7e03f6
Adds InstallTestbed instructions to Readme.md
Jan 13, 2024
beeb745
Merge branch 'main' into cbaxley-122-testbed_from_scripts
Jan 13, 2024
e052052
Adds InstallTestbed instructions to Readme.md
Jan 14, 2024
52edaa6
Modifies parameters to be pascal case
Jan 16, 2024
9c86489
ls1 not being set on DC1
Jan 16, 2024
84241f0
Adds Linux Only install to SetupTestbed
Jan 18, 2024
4788351
Remove separate linux only script
Jan 18, 2024
1a1aa97
Update testing/Readme.md
cbaxley Jan 18, 2024
6986f69
Make number of clients consisten between scripts
Jan 18, 2024
84eb65b
Add ports for elk stack for testing
Jan 19, 2024
5295599
Update readmes to change ResourceGroupName to ResourceGroup
Jan 22, 2024
e64ab4c
Adds a switch to install linux only
Jan 22, 2024
437648c
Adds simple tests to check install
Jan 23, 2024
066166e
Adds simple tests to check install
Jan 23, 2024
7b874c2
Removes the error if the old configure zip is not found.
Jan 23, 2024
84eda1d
Removes the error if the old configure zip is not found.
Jan 24, 2024
7727bed
Adds variables to linux tests run command
Jan 24, 2024
07e17a2
Move credential extraction to lib for use by other scripts.
Jan 24, 2024
246a5db
Adds npm for other testing
Jan 24, 2024
5523a1f
Adds latest version of nodejs for testing
Jan 24, 2024
9435d10
Make output.log readable for tests
Jan 24, 2024
bd10676
Merge branch 'main' into cbaxley-122-testbed_from_scripts
cbaxley Jan 25, 2024
71381f1
Add the -m parameter in the testing readme
Jan 25, 2024
6b52688
Download the latest version or a specified version
Jan 25, 2024
03a98f0
Download the latest version or a specified version
Jan 25, 2024
05f548c
Download the latest version or a specified version
Jan 25, 2024
488e40b
Reboot for 1.3.0
Jan 26, 2024
7818afb
Notes that we could have different expect scripts
Jan 26, 2024
9bd9722
Put back in the restart after all of the domain updates
Jan 26, 2024
52225a7
Scp uses ls1 instead of ls1.lme.local
Jan 26, 2024
0fabcc3
Up the timeout of the adding ls1.lme.local
Jan 26, 2024
9cb4dfc
Up the timeout of the adding ls1.lme.local
Jan 29, 2024
df245bc
Fixes chmod of the output.log for tests
Jan 30, 2024
cfc8e0e
Adds venv to the gitignore
Jan 30, 2024
ed148c3
Adds venv to the gitignore
Jan 30, 2024
e9204af
Adds the ability to pass a branch to the installer
Jan 31, 2024
184f6a4
Remove node installer
Feb 1, 2024
7b25456
Change timeout in expect script for slow connections
Feb 2, 2024
780f5e1
Change timeout in expect script for slow connections
Feb 2, 2024
29a0169
Change timeout in expect script for slow connections
Feb 2, 2024
934813e
Make shell files executable
cbaxley Feb 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,5 @@ dashboard_update.sh
files_for_windows.zip
lme.conf
lme_update.sh
/testing/tests/.env
/testing/tests/venv/
386 changes: 386 additions & 0 deletions testing/InstallTestbed.ps1
View file
Open in desktop

Large diffs are not rendered by default.

67 changes: 51 additions & 16 deletions testing/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,29 +13,31 @@ Using the Azure CLI, it creates the following:
This script does not install LME; it simply creates a fresh environment that's ready to have LME installed.

## Usage
| **Parameter** | **Alias** | **Description** | **Required** |
|------------------------|-----------|----------------------------------------------------------------------------------------|---------------------------------------|
| $ResourceGroup | -g | The name of the resource group that will be created for storing all testbed resources. | Yes |
| $NumClients | -n | The number of Windows clients to create; maximum 16; defaults to 1 | No |
| $AutoShutdownTime | | The auto-shutdown time in UTC (HHMM, e.g. 2230, 0000, 1900); auto-shutdown not configured if not provided | No |
| $AutoShutdownEmail | | An email to be notified if a VM is auto-shutdown. | No |
| $AllowedSources | -s | Comma-Separated list of CIDR prefixes or IP ranges, e.g. XX.XX.XX.XX/YY,XX.XX.XX.XX/YY,etc..., that are allowed to connect to the VMs via RDP and ssh. | Yes |
| $NoPrompt | -y | Switch, run the script with no prompt (useful for automated runs). By default, the script will prompt the user to review paramters and confirm before continuing. | No |
| **Parameter** | **Alias** | **Description** | **Required** |
|--------------------|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| $ResourceGroup | -g | The name of the resource group that will be created for storing all testbed resources. | Yes |
| $NumClients | -n | The number of Windows clients to create; maximum 16; defaults to 2 | No |
| $AutoShutdownTime | | The auto-shutdown time in UTC (HHMM, e.g. 2230, 0000, 1900); auto-shutdown not configured if not provided | No |
| $AutoShutdownEmail | | An email to be notified if a VM is auto-shutdown. | No |
| $AllowedSources | -s | Comma-Separated list of CIDR prefixes or IP ranges, e.g. XX.XX.XX.XX/YY,XX.XX.XX.XX/YY,etc..., that are allowed to connect to the VMs via RDP and ssh. | Yes |
| $Location | -l | The region you would like to build the assets in. Defaults to westus | No |
| $NoPrompt | -y | Switch, run the script with no prompt (useful for automated runs). By default, the script will prompt the user to review paramters and confirm before continuing. | No |
| $LinuxOnly | -m | Run a minimal install of only the linux server | No |

Example:
```
./SetupTestbed.ps1 -ResourceGroup Example1 -NumClients 2 -AutoShutdownTime 0000 -AllowedSources "1.2.3.4,1.2.3.5" -y
```

## Running Using Azure Shell
| **#** | **Step** | **Screenshot** |
|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
| 1 | Open a cloud shell by navigating to portal.azure.com and clicking the shell icon. | ![image](/docs/imgs/testing-screenshots/shell.png) |
| 2 | Select PowerShell. | ![image](/docs/imgs/testing-secreenshots/shell2.png) |
| 3 | Upload `SetupTestbed.ps1` by clicking the "Upload/Download files" icon | ![image](/docs/imgs/testing-screenshots/shell3.png) |
| 4 | Run the script, providing values for the parameters when promoted (see [Usage](#usage)). The script will take ~20 minutes to run to completion. | ![image](/docs/imgs/testing-screenshots/shell4.png) |
| 5 | Save the login credentials printed to the terminal at the end. At this point you can login to each VM using RDP (for the Windows servers) or SSH (for the Linux server). | ![image](/docs/imgs/testing-screenshots/shell5.png) |
| 6 | When you're done testing, simply delete the resource group to clean up all resources created. | ![image](/docs/imgs/testing-screenshots/delete.png) |
| **#** | **Step** | **Screenshot** |
|-------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------|
| 1 | Open a cloud shell by navigating to portal.azure.com and clicking the shell icon. | ![image](/docs/imgs/testing-screenshots/shell.png) |
| 2 | Select PowerShell. | ![image](/docs/imgs/testing-secreenshots/shell2.png) |
| 3 | Clone the repo `git clone https://github.com/cisagov/LME.git` and then `cd LME\testing` | |
| 4 | Run the script, providing values for the parameters when promoted (see [Usage](#usage)). The script will take ~20 minutes to run to completion. | ![image](/docs/imgs/testing-screenshots/shell4.png) |
| 5 | Save the login credentials printed to the terminal at the end (They will also be in a file called `<$ResourceGroup>.password.txt`). At this point you can login to each VM using RDP (for the Windows servers) or SSH (for the Linux server). | ![image](/docs/imgs/testing-screenshots/shell5.png) |
| 6 | When you're done testing, simply delete the resource group to clean up all resources created. | ![image](/docs/imgs/testing-screenshots/delete.png) |

# Extra Functionality:

Expand All @@ -55,3 +57,36 @@ Flags:
- enable: deletes the DENYINTERNET/DENYLOADBALANCER rules
- NSG: sets NSG to a custom NSG if desired [NSG1 default]

## Install LME on the cluster:
### InstallTestbed.ps1
## Usage
| **Parameter** | **Alias** | **Description** | **Required** |
|-------------------|-----------|----------------------------------------------------------------------------------------|--------------|
| $ResourceGroup | -g | The name of the resource group that will be created for storing all testbed resources. | Yes |
| $NumClients | -n | The number of Windows clients you have created; defaults to 2 | No |
| $DomainController | -w | The name of the domain controller in the cluster; defaults to "DC1" | No |
| $LinuxVm | -l | The name of the linux server in the cluster; defaults to "LS1" | No |
| $LinuxOnly | -m | Run a minimal install of only the linux server | No |
| $Version | -v | Optionally provide a version to install if you want a specific one. `-v 1.3.2` | No |
| $Branch | -b | Optionally provide a branch to install if you want a specific one `-b your_branch` | No |

Example:
```
./InstallTestbed.ps1 -ResourceGroup YourResourceGroup
# Or if you want to save the output to a file
./InstallTestbed.ps1 -ResourceGroup YourResourceGroup | Tee-Object -FilePath "./YourResourceGroup.output.log"
```
| **#** | **Step** | **Screenshot** |
|-------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
| 1 | Open a cloud shell by navigating to portal.azure.com and clicking the shell icon. | ![image](/docs/imgs/testing-screenshots/shell.png) |
| 2 | Select PowerShell. | ![image](/docs/imgs/testing-secreenshots/shell2.png) |
| 3.a | If you have already cloned the LME repo then make sure you are in the `LME\testing` directory and run git pull before changing to the testing directory. | |
| 3.b | If you haven't cloned it, clone the github repo in the home directory. `git clone https://github.com/cisagov/LME.git` and then `cd LME\testing`. | |
| 4 | Now you can run one of the commands from the Examples above. | |
| 5 | Save the login credentials printed to the terminal at the end. *See note* | |
| 6 | When you're done testing, simply delete the resource group to clean up all resources created. | |

Note: When the script finishes you will be in the azure_scripts directory, and you should see the elasticsearch credentials printed to the terminal.
You will need to `cd ../../` to get back to the LME directory. All the passwords should also be in the `<$ResourceGroup>.password.txt` file.


Loading
Loading