HTTP service that accepts a collection of "components"
then fans-out requests and returns aggregated content
- Components should be defined in a YAML page configuration file
- Components are requested concurrently via goroutines
- Components can be marked as "mandatory" (if they fail, the request summary is set to "failure")
components:
- id: google
url: http://google.com
- id: integralist
url: http://integralist.co.uk
mandatory: true
- id: not-found
url: http://httpstat.us/404
- id: timeout
url: http://httpstat.us/408
- id: server-error
url: http://httpstat.us/500
- id: service-unavailable
url: http://httpstat.us/503Note:
example config provided as part of this repo
./config/page.yaml
{
"summary": "success",
"components": [
{
"id": "google",
"status": 200,
"body": "<!doctype html> ... </html>",
"mandatory": false
},
{
"id": "integralist",
"status": 200,
"body": "<!doctype html> ... </html>",
"mandatory": true
},
{
"id": "slooow",
"status": 408,
"body": "Get http://localhost:3000/pugs: net/http: request canceled (Client.Timeout exceeded while awaiting headers)",
"mandatory": false
}
{
"id": "not-found",
"status": 404,
"body": "404 Not Found",
"mandatory": false
}
]
}Note: the toplevel
summarykey's value will befailureif any mandatory components fail
You'll need Glide installed:
go get github.com/Masterminds/glideIf using Go 1.6 you're OK, otherwise you'll need to set the vendor experiment:
export GO15VENDOREXPERIMENT=1To install the dependencies found in the Glide.lock file:
glide installgo test $(glide novendor)go buildOnce the application is built and installed into your binary path, you can run it:
go-requester <path/to/your/page/config>There are two ways to run this application locally:
- Host Machine
- Docker
go run main.go ./config/page.yamlAlternative:
godo server --watch
Once the application is running, view: http://localhost:8080/
If you want to see how latency/slow responses effect the application, then also try running: https://github.com/Integralist/go-slow-http-server which was specifically designed to be used alongside go-requester for testing purposes.
docker build -t gorequester .
docker run --rm --name gr -v "$PWD":/go/src/app -p 5000:8080 gorequesterOnce the application is running, view: http://<docker-machine-ip>:5000/
To debug the container:
docker exec -it <container_id> /bin/bash
docker run -it --entrypoint /bin/bash -v "$PWD":/go/src/app -p 5000:8080 gorequesterIf you specify a private repo as a dependency, then you'll need to pass in your SSH credentials:
docker run \
-it \
-v "$PWD":/go/src/app \
-v "$HOME/.ssh/github_bbc_rsa":/.ssh/github_rsa \
-p 5000:8080 \
gorequester ssh_setupNotice we've passed ssh_setup. Our bootstrap script (which runs inside the Docker container) will identify when this argument is provided and subsequently load up the SSH agent within the container.
Because we've added the -it flag we're able to manually type in our SSH private key's passphrase (you do use a passphrase don't you?).
Having to type in your SSH passphrase can be annoying an restrict your ability to automate. Although I personally wouldn't advise it, you can make a change to the bootstrap script and have it use the ssh.exp script included in this repository.
Using this Expect script will mean you don't have to manually enter the passphrase for your SSH private key. The MASSIVE downside to this process is that you'll need to edit the ssh.exp script to include your passphrase. This means your secret passphrase has now been codified into the file.
If your laptop is compromised or you commit the change accidentally, then you're in serious trouble. So really... don't use it unless you're really sure you know what you're doing.
You'll also need to comment out the line ssh-add /.ssh/github_rsa from the bootstrap script and replace it with /ssh.exp.
We can verify the performance of this application using curl timings, like so:
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/Note: I've included a
curl-format.txtfile within the repo
- Refactor code so that certain aspects are loaded from other packages
- Add logic for loading page config remotely
- Dynamically change port number when run as binary
- Tests!
Copyright (c) 2015 Mark McDonnell