NAME

cloudfront_curl

SYNOPSIS

cloudfront_curl [OPTIONS] -- [CURL_OPTIONS] <URL>

A curl wrapper to CloudFront PoP edge nodes. Tailored to be an OpsView/Nagios monitoring script.

• Accumulates a list of PoP IP addresses from a RIPE Atlas DNS measurement using probes around the US and Canada

• Run a curl call with any extra parameters provided, prepending --resolve to the appropriate PoP IP address

• Grab the x-amz-cf-pop response header, store timings for each PoP

• Look up the PoP location

• print out statistics after all HTTP requests are finished

• exit with a success/warning/error status if any requests failed

REQUIRED PACKAGES

• Getopt::Long (perl-Getopt-Long)

• Pod::Usage (perl-Pod-Usage)

• JSON (perl-JSON)

• Time::HiRes (perl-Time-HiRes)

• HTTP::Request (perl-libwww-perl)

• LWP::UserAgent (perl-libwww-perl)

• URI (perl-URI)

• Net::DNS (libnet-dns-perl or perl-Net-DNS)

• XML::LibXML (libxml-libxml-perl or perl-XML-LibXML)

• threads (perl-threads)

• threads::shared (perl-threads-shared)

OPTIONS

--help

Show usage information

--man

Show the full manual page.

CURL_OPTIONS URL

You may add extra options to curl. For example, to add request headers:

$ cloudfront_curl -- -H "Authorization: 1234" https://my-cdn.example.com

You only need the -- end-of-options separator with extra curl options. The most simple execution of this script might look like

$ cloudfront_curl https://my-cdn.example.com

--ca_file /full/path/to/ca-bundle.crt

Internally, this uses the perl LWP::UserAgent module for HTTP requests to RIPE Atlas. If you have problems with this script showing an error that it can't contact atlas.ripe.net, specify the location to a freshly updated root CA bundle.

For RH based distros, it's usually /etc/pki/tls/certs/ca-bundle.crt For Debian based distros, it's /etc/ssl/certs/ca-certificats.crt

--msm_id <MEASUREMENT ID>

By default, cloudfront_curl uses a RIPE Atlas measurement that is publicly available and contains the DNS lookups for PoP locations across the US and Canada.

If you want to create your own measurement, specify the ID here.

--key <RIPE API KEY>

If you've created your own private measurement, provide your RIPE Atlas API key

--verbose -v

By default, cloudfront_curl is pretty quiet, only printing errors and warnings for use with monitoring checks. Turn on extra output with --verbose

--debug -d

Turn on very detailed debugging messages for each PoP location

--threads <INTEGER>

Thread out the curl calls for a faster runtime. Default is 1 thread. If OpsView times out from its default 60 second agent timeout, increase the thread count enough to finish the check under the allotted time. Don't go too far though, since too many concurrent HTTP checks on the client could adversely affect the check.

--random

Append a random jQuery-like _=12345678 random GET query to each CloudFront curl call. This allows you to enforce a cache miss on each PoP location to test the added latency of hitting the origin.

Without this parameter, repeated calls will most likely cause a cache hit.

--print_ip_stats

Print some request time statistics for each IP address hit with cloudfront_curl.

If --verbose is supplied, print these statistics

--print_pop_stats

Similar to --print_ip_stats, but aggregate the timings and show averages for each PoP location hit.

If --verbose is supplied, print these statistics

--validate_pops [FILE INPUT]

After running the curl calls, validate that we actually hit all of the expected PoP locations. If no FILE INPUT is provided, the valid list of PoPs are queried from an online list.

If some PoP locations are missed, return with a WARNING exit status.

The file format is one entry per line.

Tip: If you have CloudFront logging to S3 enabled, use this Athena query to pull an active list of your PoPs:

SELECT location, status, count(*) as hit_count 
FROM "my_cdn" 
group by location, status order by status desc

--ip_list <FILE INPUT>

By default, the list of PoP location IP addresses are pulled from a RIPE Atlas measurement. Override this with a local file, one IP address entry per line.

Tip: If you have ELB access logs, the CloudFront IP addresses should be in there in the client_ip field.

EXAMPLES

Validate a list of PoP locations

There is a list of USA and Canada PoP names in the lists/ directory.

$ ./cloudfront_curl --validate_pops ./lists/validate_us_ca.txt \
  --verbose --print_ip_stats --print_pop_stats \
  https://my-cdn.example.com
 

Add an Authorization header to your request

Add any extra curl parameters after the end-of-options marker:

$ ./cloudfront_curl --verbose -- -H "Authorization: 1234" https://my-cdn.example.com

OPSVIEW INTEGRATION

Passive Checks

• Create a new OpsView host for the CDN

• Create a new passive service check for that host

• Attach the passive service check to the CDN host

• Find the location of your results.sock file on the machine running OpsView

  Use the opsview_passive script to wrap the cloudfront_curl script. Add a cronjob entry on the machine running OpsView similar to

  */5 * * * * opsview_passive "my-cdn.example.com" "CloudFront service check" "https://my-cdn.example.com"  

• Reload OpsView

Repeat this process for additional CDNs. The checks should run within 2 - 3 minutes and post the results to the passive service. The OpsView service definition can also define what state to use if the script has not run for N minutes. (Default is UNKNOWN)

Active Checks

• Create a new OpsView host for the CDN

• Clone the cloudfront_curl repo to the machine running OpsView

• Symlink /opt/opsview/monitoringscripts/plugins/check_cloudfront_curl to the location of the cloudfront_curl script

• Create a new active service check for that host. Add parameters to cloudfront_curl for this specific CDN check.

• Attach the service check to the CDN host

• Reload OpsView. If the service check times out, add more threads with the --threads parameter and see if it finishes in time.

  It's not recommended to increase the global 60 second timeout in OpsView for plugin checks.