This document provides instructions for setting up the environment, running the application, and performing various tasks such as compiling, testing, and contributing to the project.
Any distribution of the Java 17 JDK.
To run the application directly, execute...
./gradlew clean run
This runs the web API on port 8080. The app reads/writes data to a local file (unless you have a DB configured)
You can view the API documentation at /openapi
.
- Run
brew install mike-engel/jwt-cli/jwt-cli
- Replace
PATH_TO_FILE_ON_YOUR_MACHINE
in this command with the actual path, then run it:jwt encode --exp='+5min' --jti $(uuidgen) --alg RS256 --no-iat -S @/PATH_TO_FILE_ON_YOUR_MACHINE/trusted-intermediary/mock_credentials/organization-trusted-intermediary-private-key-local.pem
- Copy token from terminal and paste into your postman body with the key
client_assertion
- Add a key to the body with the key
scope
and value oftrusted-intermediary
- Body type should be
x-wwww-form-urlencoded
- You should be able to run the post call against the
v1/auth/token
endpoint to receive a bearer token to be used in this step
The additional requirements needed to contribute towards development are...
To set up the necessary environment variables, run the generate_env.sh
script. This script
creates a .env
file in the resource folder with the required configuration
-
Navigate to the project directory.
-
Run the
generate_env.sh
script:./generate_env.sh
-
If you run TI using Docker rather than Gradle, update the DB and port values in the
.env
file (the alternate values are in comments)
Use docker-compose.yml to run your local DB. In IntelliJ, you can click the play arrow to start it
Apply all outstanding migrations:
liquibase update --changelog-file ./etor/databaseMigrations/root.yml --url jdbc:postgresql://localhost:5433/intermediary --username intermediary --password 'changeIT!' --label-filter '!azure'
If running in Windows, use double quotes instead:
liquibase update --changelog-file ./etor/databaseMigrations/root.yml --url jdbc:postgresql://localhost:5433/intermediary --username intermediary --password "changeIT!" --label-filter "!azure"
To compile the application, execute...
./gradlew shadowJar
Once compiled, the built artifact is /app/build/libs/app-all.jar
.
To run the unit tests, execute...
./gradlew clean allUnitTests
End-to-end tests are designed to interact with the API and verify that its overall flow operates correctly. They require that the API to be running already. The end-to-end tests use whatever database configuration is already in place - if you're using the local filesystem, so will the e2e tests (this is how they work on github), and if you're using a DB, so will the tests
To run them, execute...
./gradlew e2e:clean e2e:test
The previous command requires the API to be running already. To help streamline the execution of this flow, a helper Bash script can be executed...
./e2e-execute.sh
This will start the API, wait for it to respond, run the end-to-end tests against that running API, and then stop the API.
These tests are located under the e2e
Gradle sub-project directory. Like any Gradle project, there are the main
and test
directories.
The test
directory contains the tests. The main
directory contains our custom framework that helps us interact with the API.
These tests cover the integration between ReportStream and TI. They run automatically every weekday via Github actions. See the rs-e2e readme for more details.
Load tests are completed with Locust.io. Run the load tests by running...
./gradle-load-execute.sh
./docker-load-execute.sh
Currently, we are migrating to using Azure. Local load testing is using gradle, however a docker load test is available to mimic the Azure environment settings until the azure migration is complete.
This will run the API for you, so no need to run it manually.
Note:
If you are already running the API, stop it before running the load tests or the cleanup steps won't work.
The load tests will also spin up (and clean up) a local test DB on port 5434 that should not interfere with the local dev DB.
The locustfile.py
that specifies the load test is located at
./operations/locustfile.py
.
If you want to run the load test in an interactive mode, run...
locust -f ./operations/locustfile.py
The terminal will start a local web interface, and you can enter
the swarm parameters for the test and the local url where the app is running
(usually http://localhost:8080
). You can also set time limits for the tests under 'Advanced Settings'.
The project comes with an attached remote jvm configuration for debugging the container.
If you check your remote JVM settings, under Run/Edit Configurations
,
you will see the Debug TI
. If you want to add a new remote JVM configuration, follow the steps below,
under "Docker Container Debugging Using Java Debug Wire Protocol"
Go into the Dockerfile
file and change CMD ["java", "-jar", "app.jar"]
to CMD ["java", "-agentlib:jdwp=transport=dt_socket,address=*:6006,server=y,suspend=n", "-jar", "app.jar"]
- In Intellij, click on Run and select Edit Configurations
- Create a new Remote JVM Debug
- Set up the configuration for the remote JVM debug to look like this.
- In your code, set your breakpoint, and then start your docker container with
docker-compose up --build
- Once your docker container is running, in order to attach, select Run again.
- Select Debug (not Attach to Process)
- Select your Docker Debug that you set up in step 3
- A console window will pop up that will show you that it is connected to Docker, and at that point, you can interact with your container and then step through the code at your breakpoints.
We have a number of environments that are split between CDC and non-CDC Azure Entra domains and subscriptions.
The Internal environment is designed to be the Wild West, meaning anyone can push changes without restrictions. It allows for testing various configurations without the requirement that only stable builds be pushed. Use the Internal environment if you want to test something in a deployed environment in a non-CDC Azure Entra domain and subscription. See below:
Before starting...
Remember to ping the Engineering Channel to make sure someone is not already using the enviroment.
To deploy to the Internal environment...
-
Check with the team that no one is already using it.
-
Find the
internal
branch and delete it inGitHub. -
Delete your local
internal
branch if needed.git branch -D internal
-
From the branch you want to test, create a new
internal
branch.git checkout -b internal
-
Push the branch to GitHub.
git push --set-upstream origin internal
Then the deploy will run.
Remember that you now have the internal
branch checked out locally. If you make subsequent code changes, you will
make them on the internal
branch instead of your original branch.
The Dev environment is similar to the Internal environment but deploys to a CDC Azure Entra domain and subscription. It
is also meant to be the Wild West. Dev deploys similarly to the Internal environment, but you interact with the
dev
branch.
The Staging environment is production-like and meant to be stable. It deploys to a CDC Azure Entra domain and
subscription. Deployments occur when a commit is made to the main
branch. main
is a protected branch and requires
PR reviews before merge.
The Production environment is the real deal. It deploys to a CDC Azure Entra domain and subscription. Deployments occur when a release is published.
There is minimal set-up to do to get Terraform setup before you can run the Terraform commands in
a new Azure environment in the Flexion Entra domain. For example, the internal
environment. This does not apply to the CDC
Entra domains and subscriptions.
- Create a resource group.
- Create a storage account inside the aforementioned resource group.
- Within the new storage account, create a Container.
- Within Azure Entra...
- Create an App Registration.
- Add federated credentials to the App Registration
repo:CDCgov/trusted-intermediary:ref:refs/heads/main
(for terraform apply).repo:CDCgov/trusted-intermediary:environment:staging
(for staging webapp deploy).- And presumably other repo paths needed in the future for other environments and branches.
- Within your Subscription, assign the Contributor role to the previously created App Registration.
- Add GitHub Action secrets to your GitHub repository.
- A secret with the tenant ID from Azure Entra directory.
- A secret with the ID from the subscription that everything should be deployed into.
- A secret with the ID of the App Registration created previously.
- Create a copy of one of the environments under the operations folder.
- Name the copy off of the name of the new environment. Ex:
internal
- Edit the
main.tf
file with the names of the resources previously created:resource_group_name
,storage_account_name
,container_name
. Also update theenvironment
to match the new folder name.
- Name the copy off of the name of the new environment. Ex:
- Create a GitHub Action workflow so that automatic deploys can occur. You can take inspiration from our
Internal environment deployment. Make sure you set the
AZURE_CLIENT_ID
,AZURE_TENANT_ID
, andAZURE_SUBSCRIPTION_ID
based on the secrets created previously.
The PR and Internal environment is available on the public Internet and able to be interacted with directly.
The Dev, Staging, and Prod environment are deployed inside a Vnet and require special steps to interact with these.
The application basically has a firewall in place. You need to add (and remove when you're done) your IP address to the firewall allow list.
- Log into CyberArk and then into Azure with your -SU account.
- Navigate to the environment's app service.
- Click on Networking in the left pane.
- Click on the "Enabled with access restrictions" link under "Inbound traffic configuration".
- Add a new rule to allow your public IP address. Provide an appropriate name with your name. The priority will
need a lower number than the existing denies. It will look like your IP address with a
/32
appended. E.g.192.168.0.1/32
. - Click "Save".
You will now be able to interact with that environment's application.
Note:
Don't forget to remove your rule and save when you are done.
You will need to connect to the VPN for the given environment first, and then you can interact with the database. Notion contains the instructions for connecting to the VPN. After connecting, you can follow the database documentation to gain access.
We use pre-commit
to run some hooks on every commit. These
hooks do linting to ensure things are in a good spot before a commit is made. Please install pre-commit
and then
install the hooks.
pre-commit install
Anyone is encouraged to contribute to the repository by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.
Please read CONTRIBUTING.md
for additional details.
All comments, messages, pull requests, and other submissions received through CDC including this GitHub page may be subject to applicable federal law, including but not limited to the Federal Records Act, and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.
For database documentation: /docs/database.md
- Checkout
main
branch forCDCgov/trusted-intermediary
- Run
./generate_env.sh
to generate.env
file with required environment variables - Run TI with
./gradlew clean run
For Apple Silicon you will want to enable the Docker option for Use Rosetta for x86/amd64 emulation on Apple Silicon
.
After enabling this option it is recommended that you delete all docker images and containers and rebuild them
with this option enabled.
- Checkout
master
branch forCDCgov/prime-reportstream
- Build RS (for more information please refer to the ReportStream docs):
- If building for the first time, run:
./cleanslate
inprime-reportstream/prime-router
- Otherwise run:
./gradlew clean package
inprime-reportstream
root folder - If attempting to access the metadata endpoint in RS add the variable
ETOR_TI_baseurl="http://host.docker.internal:8080"
toprime-router/.vault/env/.env.local
file before building the container
- If building for the first time, run:
- Run RS with
docker compose up -d
. You may also use./gradlew quickRun
- Run the RS setup script in this repository:
/scripts/setup/setup-reportstream.sh
- Before running the script, make sure to follow the instructions in /scripts/README.md
- You can verify the script created vault secrets successfully by going to
http://localhost:8200/
in your browser, use the token inprime-router/.vault/env/.env.local
to authenticate, and then go toSecrets engines
>secret/
to check the available secrets
To test sending from a simulated hospital:
curl --header 'Content-Type: application/hl7-v2' --header 'Client: flexion.simulated-hospital' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/orm_message.hl7' 'http://localhost:7071/api/waters'
To test sending from TI:
curl --header 'Content-Type: application/fhir+ndjson' --header 'Client: flexion.etor-service-sender' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oml_message.fhir' 'http://localhost:7071/api/waters'
To test sending from a simulated lab:
curl --header 'Content-Type: application/hl7-v2' --header 'Client: flexion.simulated-lab' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oru_message.hl7' 'http://localhost:7071/api/waters'
To test sending from TI:
curl --header 'Content-Type: application/fhir+ndjson' --header 'Client: flexion.etor-service-sender' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oru_message.fhir' 'http://localhost:7071/api/waters'
After one or two minutes, check that hl7 files have been dropped to prime-reportstream/prime-router/build/sftp
folder
In order to submit a request, you'll need to authenticate with ReportStream using JWT auth:
-
Create a JWT for the sender (e.g.
flexion.simulated-hospital
) using the sender's private key, which should be stored in Keybase. You may use this CLI tool to create the JWT:jwt encode --exp='+5min' --jti $(uuidgen) --alg RS256 -k <sender> -i <sender> -s <sender> -a staging.prime.cdc.gov --no-iat -S @/path/to/sender_private.pem
-
Use the generated JWT to authenticate with ReportStream and get the token, which will be in the
access_token
responsecurl --header 'Content-Type: application/x-www-form-urlencoded' --data 'scope=flexion.*.report' --data 'client_assertion=<jwt>' --data 'client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer' --data 'grant_type=client_credentials' 'http://localhost:7071/api/token'
-
Submit an Order or Result using the returned token in the
'Authorization: Bearer <token>'
header
We use DORA Metrics to measure our DevOps performance. We currently are tracking Deployment Frequency, Change Fail Rate and Mean Time to Recovery.
The metrics are produced weekly using a Github Action and written into CSV files which are available for download in the workflow job's artifacts.
- Open Practices
- Rules of Behavior
- Thanks and Acknowledgements
- Disclaimer
- Contribution Notice
- Code of Conduct
This repository constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This repository is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this repository will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.
The repository utilizes code licensed under the terms of the Apache Software License and therefore is licensed under ASL v2 or later.
This source code in this repository is free: you can redistribute it and/or modify it under the terms of the Apache Software License version 2, or (at your option) any later version.
This source code in this repository is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License for more details.
You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html
The source code forked from other open source projects will inherit its license.
This repository contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/other/privacy.html.
This repository is not a source of government records, but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC website.
Please refer to CDC's Template Repository for more information about contributing to this repository, public domain notices and disclaimers, and code of conduct.
Error: copier: stat: "/app/build/libs/app-all.jar": no such file or directory Solution: Run ./gradlew shadowjar first