Skip to content

Home

Jump to bottom Edit New page
egregory edited this page May 19, 2018 · 61 revisions

Hyperpotamus logo

Quickstart

If you don't already have hyperpotamus installed, you can install it according to this guide. If you are in a rush:

sudo npm install -g hyperpotamus

To get started, create a new text file with the following contents and save it as "hello.yml":

- print: Hello World!

And now run it from your command-prompt/terminal with

hyperpotamus hello.yml

Now let's make our script a bit more interactive:

- prompt:
    name: What is your name?
- print: Hello, <% name %>!

Here is a quick-list of some hyperpotamus features:

  • JSON/YAML format for scripts is easy to read and edit.
  • <% macro %> syntax for replacing values.
  • powerful macro transformation pipeline to format data <% value | substring,3 | upcase | urlencode %>
  • if/then/else and boolean logical (and/or/not) operations to control script flow
  • Data driven input for bulk operations (run a script for each row of a .csv file)
  • array manipulation, looping and iteration and parallel processing
  • easy processing for JSON, HTML, and XML using JsonPath, JQuery, and XPath
  • regex/text matching and capture from HTTP responses
  • support for date/time parsing, formatting, and manipulation
  • powerful HTTP request control (user-agent impersonation, custom headers, cookie handling, proxy handling and authentication)
  • randomization for numerical values or array element selection
  • custom javascript functions and plugins for extensible control
  • save binary/text response data to files
  • base64 and url encoding/decoding
  • sha1/md5 hashing
  • loading in additional JSON/YAML data files and lookup data
  • supports file/directory inclusion to modularize common scripts portions
  • tons more...

Overview

Hyperpotamus allows users to write simple, human-readable scripts using YAML or JSON to automate the processing of computer tasks. Hyperpotamus is commonly used to parse and print out data, execute other programs or send HTTP requests. Hyperpotamus allows dynamic variables to be used to parameterize actions and can validate or capture data from action responses to use in subsequent actions.

The values for these dynamic variables can be:

  1. Set or overridden directly in the script
  2. Read from the user in response to an input prompt
  3. Passed in as parameters on the command-line
  4. Read as columns from a .csv data-file
  5. Loaded from one or more YAML/JSON files

Hyperpotamus includes specific support for parsing JSON/HTML/XML HTTP responses, however it can work with any text-based format. The extensible plugin framework and the embedded javascript engine allow developers to easily extend the processing capabilities beyond what is included out-of-the-box.

Even though hyperpotamus scripts aim to be very short and readable, powerful multi-step processes are also supported.

Example

As an example, let's say that we want to make a request to a web-page. To do this, we can use the request action. Hyperpotamus will make a request to the specified URL and by default will raise an error if the response does not return with a 200 status code (after following any redirects).

- request: https://github.com/pmarkert/hyperpotamus

There are many options that can be set to customize the request and response handling.

Have you ever used curl to automate sending an HTTP request? If you have ever tried creating multi-step scripts, or dynamically replacing parameter values, or parsing data from the response of one request to use as the input to another request then you know that it can be done, but it typically results in ugly, hard to maintain shell scripts. Hyperpotamus aims to provide a simpler and more elegant solution.

Let's consider the following curl script, which makes a POST request to a hypothetical API to set the value of a user's favorite person:

curl 'http://hypothetical-api.org/users/$USERNAME/favorite' \
-X POST \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'favorite=$FAVORITE'

Here is the hyperpotamus equivalent.

- request:
    url: http://hypothetical-api.org/users/<% username %>/favorite
    method: POST
    form:
      favorite: <% favorite %>

Notice the <% username %> and <% favorite %> tokens in the script. Those are dynamic macros and hyperpotamus will replace them with the specified values when needed. Where does hyperpotamus get the values to use? As mentioned above, there are a few options.

The values could also be set directly in the script.

- set:
    username: Ironman
    favorite: Pepper Potts
- request: 
    ...etc, etc...

That's not terribly useful because if we wanted to hard-code them into the script, then we could have just put them directly into the request element. Changing the first action's name from set to defaults will cause the value to only be set if the value has not already been made available via one of the other methods.

The values could also be passed in on the command-line (this is not a great idea for passwords or sensitive data):

hyperpotamus login.csv --qs "username=Batman&favorite=Superman"

A common and powerful use-case with hyperpotamus is to create a CSV file with columns that match the macro names. The script can be executed using the CSV file as input. Hyperpotamus will run the script once for each row in the .csv file, using the values from each row to populate the macros. This makes it easy to automate a process that needs to be repeated many times for different values.

** users.csv **

username,favorite
Agent_Smith,Neo
DarthVader,Padme

This will result in the script being for both users:

hyperpotamus --csv users.csv login.yml

We can build interactive scripts that prompt the user for input. By default, the user will only be prompted if the value is not available via one of the other methods.

- prompt:
    username: Please enter the username
    favorite: Who is their favorite person?
- request: ...

Values can be dynamically calculated or parsed from the results of a previous request. Hyperpotamus excels at dealing with multi-step requests

- request:
    url: http://hypothetical-api.org/users/<% username %>
  response:
    json:
      favorite: "$.preferences.favorite"

- request:
    url: http://hypothetical-api.org/users/<% username %>/favorite
    method: POST
    form:
      favorite: Still <% favorite %>

Have you ever needed to describe the parameters or options supported by an HTTP endpoint? Perhaps you were documenting an API or giving an example of a request that produced a problem. Many times people will include sample requests using curl command-lines. Since hyperpotamus uses an easily-readable format for describing requests with parameters, hyperpotamus scripts can be excellent examples for documentation.

Spotify Music Image example

Let's take a peek at a more complex, but fun example. This prompts the user for the name (or part of a name) of a musical artist and then queries the Spotify JSON API for matches. For each matching artist on the first page of results, the script downloads and saves the first linked image into a sub-folder called "images", naming the image after the name of the artist. The most complex part of this script is the JSONPath expressions which include a filter to remove any items that do not have any images. To try this one out, just save the contents into a text file and run it with hyperpotamus.

- prompt:
    artist: Please enter all or part of an artist's name

- request:
    url: https://api.spotify.com/v1/search?q=<% artist | urlencode %>&type=artist
  response:
    - status: 200
    - json: # Capture arrays of the names and first image url. Some artists don't have any images, so filter.
        names: [ "$.artists.items.*.name" ]
        images: [ "$.artists.items.*.images[0].url" ]

- request:
    url: <% images | current %>
    encoding: null # Important for downloading binary data
  response:
   - save: 'images/<% names | current %>.jpeg'
   - iterate: [ names, images ] # move to the next

NOTE: For simplicity, this example eliminates some of the error handling and special cases that would be in a production-level script. Depending upon the search term entered, this script may fail if Spotify returns results that do not have images or if no results can be found. Hyperpotamus does support many techniques to work through these types of issues, which are covered in the documentation.

Word of the day example

Let's create another example that will scrape the "word of the day" from the Merriam Webster dictionary site.

request: http://www.merriam-webster.com/word-of-the-day/
response:
  - jquery: div.word-header h1
    capture:
      todays_word: text
  - print: <% todays_word %>

NOTE: With YAML, indentation is important

This script says that we want to go to the url http://www.merriam-webster.com/word-of-the-day/ and use the jquery action to look for an html element that has a class of "main_entry_word". Once we find that element on the page, we capture the text of the element into a session variable named "todays_word".

Hyperpotamus Documentation - home

Index

Clone this wiki locally