Shell Scripting Templates and Utilities

A collection of BASH utility functions and script templates used to ease the creation of portable and hardened BASH scripts with sane defaults.

Usage

There are two Script Templates located in the root level of this repository. The usage of these templates is described in detail below.

BASH Utility functions are located within the utilities/ folder.

Complex sed find/replace operations are supported with the files located in sedfiles/. Read the usage instructions.

Automated testing is provided using BATS. All tests are in the tests/ repo. A git pre-commit hook provides automated testing is located in the .hooks/ directory. Read about how to install the hook.

Bash Script Templates Usage

To create a new script, copy one of the script templates to a new file and make it executable chmod 755 [newscript].sh. Place your custom script logic within the _mainScript_ function at the top of the script.

Script Templates

There are two templates located at the root level of this repository.

• template.sh - A lean template which attempts to source all the utility functions from this repository. You will need to update the path to the utilities folder sent to _sourceUtilities_ at the bottom of the script. This template will not function correctly if the utilities are not found.
• template_standalone.sh - For portability, the standalone template does not assume that this repository is available. Copy and paste the individual utility functions under the ### Custom utility functions line.

Code Organization

The script templates are roughly split into three sections:

• TOP: Write the main logic of your script within the _mainScript_ function. It is placed at the top of the file for easy access and editing. However, it is invoked at the end of the script after options are parsed and functions are sourced.
• MIDDLE: Functions and default variable settings are located just below _mainScript_.
• BOTTOM: Script initialization (BASH options, traps, call to _mainScript_, etc.) is at the bottom of the template

Default Options

These default options and global variables are included in the templates and used throughout the utility functions. CLI flags to set/unset them are:

• -h, --help: Prints the contents of the _usage_ function. Edit the text in that function to provide help
• --logfile [FILE] Full PATH to logfile. (Default is ${HOME}/logs/$(basename "$0").log)
• loglevel [LEVEL]: Log level of the script. One of: FATAL, ERROR, WARN, INFO, DEBUG, ALL, OFF (Default is 'ERROR')
• -n, --dryrun: Dryrun, sets $DRYRUN to true allowing you to write functions that will work non-destructively using the _execute_ function
• -q, --quiet: Runs in quiet mode, suppressing all output to stdout. Will still write to log files
• -v, --verbose: Sets $VERBOSE to true and prints all debug messages to stdout
• --force: If using the _seekConfirmation_ utility function, this skips all user interaction. Implied Yes to all confirmations.

You can add custom script options and flags to the _parseOptions_ function.

Script Initialization

The bottom of the script template file contains a block which initializes the script. Comment, uncomment, or change the settings here for your needs

trap '_trapCleanup_ ${LINENO} ${BASH_LINENO} "${BASH_COMMAND}" "${FUNCNAME[*]}" "${0}" "${BASH_SOURCE[0]}"' EXIT INT TERM SIGINT SIGQUIT SIGTERM ERR

# Trap errors in subshells and functions
set -o errtrace

# Exit on error. Append '||true' if you expect an error
set -o errexit

# Use last non-zero exit code in a pipeline
set -o pipefail

# Confirm we have BASH greater than v4
[ "${BASH_VERSINFO:-0}" -ge 4 ] || {
    printf "%s\n" "ERROR: BASH_VERSINFO is '${BASH_VERSINFO:-0}'.  This script requires BASH v4 or greater."
    exit 1
}

# Make `for f in *.txt` work when `*.txt` matches zero files
shopt -s nullglob globstar

# Set IFS to preferred implementation
IFS=$' \n\t'

# Run in debug mode
# set -o xtrace

# Source utility functions
_sourceUtilities_

# Initialize color constants
_setColors_

# Disallow expansion of unset variables
set -o nounset

# Force arguments when invoking the script
# [[ $# -eq 0 ]] && _parseOptions_ "-h"

# Parse arguments passed to script
_parseOptions_ "$@"

# Create a temp directory '$TMP_DIR'
# _makeTempDir_ "$(basename "$0")"

# Acquire script lock
# _acquireScriptLock_

# Source GNU utilities for use on MacOS
# _useGNUUtils_

# Run the main logic script
_mainScript_

# Exit cleanly
_safeExit_

Utility Functions

The files within utilities/ contain BASH functions which can be used in your scripts. Each included function includes detailed usage information. Read the inline comments within the code for detailed usage instructions.

Including Utility Functions

Within the utilities folder are many BASH functions meant to ease development of more complicated scripts. These can be included in the template in two ways.

1. Copy and paste into template_standalone.sh

You can copy any complete function from the Utilities and place it into your script. Copy it beneath the ### Custom utility functions line. Scripts created this way are fully portable among systems

2. Source all the utility files by using template.sh

template.sh contains a function to source all the utility files into the script. IMPORTANT: You will need to update the paths within the _sourceUtilities_ function to ensure your script can find this repository.

alerts.bash

• _columns_ Prints a two column output from a key/value pair
• -_printFuncStack_ Prints the function stack in use. Used for debugging, and error reporting
• _alert_ Performs alerting functions including writing to a log file and printing to screen
• _centerOutput_ Prints text in the center of the terminal window
• _setColors_ Sets color constants for alerting (Note: Colors default to a dark theme.)

Basic alerting, logging, and setting color functions (included in scriptTemplate.sh by default). Print messages to stdout and to a user specified logfile using the following functions.

debug "some text"     # Printed only when in verbose (-v) mode
info "some text"      # Basic informational messages
notice "some text"    # Messages which should be read. Brighter than 'info'
warning "some text"   # Non-critical warnings
error "some text"     # Prints errors and the function stack but does not stop the script.
fatal "some text"     # Fatal errors. Exits the script
success "some text"   # Prints a success message
header "some text"    # Prints a header element
dryrun "some text"    # Prints commands that would be run if not in dry run (-n) mode

The following global variables must be set for the alert functions to work

• $DEBUG - If true, prints debug level alerts to stdout. (Default: false)
• $DRYRUN - If true does not eval commands passed to _execute_ function. (Default: false)
• $LOGFILE - Path to a log file
• $LOGLEVEL - One of: FATAL, ERROR, WARN, INFO, DEBUG, ALL, OFF (Default: ERROR)
• $QUIET - If true, prints to log file but not stdout. (Default: false)

arrays.bash

Utility functions for working with arrays.

• _dedupeArray_ Removes duplicate array elements
• _forEachDo_ Iterates over elements and passes each to a function
• _forEachFilter_ Iterates over elements, returning only those that are validated by a function
• _forEachFind_ Iterates over elements, returning the first value that is validated by a function
• _forEachReject_ Iterates over elements, returning only those that are NOT validated by a function
• _forEachValidate_ Iterates over elements and passes each to a function for validation
• _inArray_ Determine if a value is in an array
• _isEmptyArray_ Checks if an array is empty
• _joinArray_ Joins items together with a user specified separator
• _mergeArrays_ Merges the values of two arrays together
• _randomArrayElement_ Selects a random item from an array
• _reverseSortArray_ Sorts an array from highest to lowest
• _setdiff_ Return items that exist in ARRAY1 that are do not exist in ARRAY2
• _sortArray_ Sorts an array from lowest to highest

checks.bash

Functions for validating common use-cases

• _commandExists_ Check if a command or binary exists in the PATH
• _functionExists_ Tests if a function is available in current scope
• _isInternetAvailable_ Checks if Internet connections are possible
• _isAlpha_ Validate that a given variable contains only alphabetic characters
• _isAlphaDash_ Validate that a given variable contains only alpha-numeric characters, as well as dashes and underscores
• _isAlphaNum_ Validate that a given variable contains only alpha-numeric characters
• _isDir_ Validate that a given input points to a valid directory
• _isEmail_ Validates that an input is a valid email address
• _isFQDN_ Determines if a given input is a fully qualified domain name
• _isFile_ Validate that a given input points to a valid file
• _isIPv4_ Validates that an input is a valid IPv4 address
• _isIPv6_ Validates that an input is a valid IPv6 address
• _isNum_ Validate that a given variable contains only numeric characters
• _isTerminal_ Checks if script is run in an interactive terminal
• _rootAvailable_ Validate we have superuser access as root (via sudo if requested)
• _varIsEmpty_ Checks if a given variable is empty or null
• _varIsFalse_ Checks if a given variable is false
• _varIsTrue_ Checks if a given variable is true

dates.bash

Functions for working with dates and time.

• _convertToUnixTimestamp_ Converts a date to unix timestamp
• _countdown_ Sleep for a specified amount of time
• _dateUnixTimestamp_ Current time in unix timestamp
• _formatDate_ Reformats dates into user specified formats
• _fromSeconds_ Convert seconds to HH:MM:SS
• _monthToNumber_ Convert a month name to a number
• _numberToMonth_ Convert a month number to its name
• _parseDate_ Takes a string as input and attempts to find a date within it to parse into component parts (day, month, year)
• _readableUnixTimestamp_ Format unix timestamp to human readable format
• _toSeconds_ Converts HH:MM:SS to seconds

debug.bash

Functions to aid in debugging BASH scripts

• _pauseScript_ Pause a script at any point and continue after user input
• _printAnsi_ Helps debug ansi escape sequence in text by displaying the escape codes
• _printArray_ Prints the content of array as key value pairs for easier debugging

files.bash

Functions for working with files.

• _backupFile_ Creates a backup of a specified file with .bak extension or optionally to a specified directory.
• _decryptFile_ Decrypts a file with openssl
• _encryptFile_ Encrypts a file with openssl
• _extractArchive_ Extract a compressed file
• _fileBasename_ Gets the basename of a file from a file name
• _fileContains_ Tests whether a file contains a given pattern
• _filePath_ Gets the absolute path to a file
• _fileExtension_ Gets the extension of a file
• _fileName_ Prints a filename from a path
• _json2yaml_ Convert JSON to YAML uses python
• _listFiles_ Find files in a directory. Use either glob or regex.
• _makeSymlink_ Creates a symlink and backs up a file which may be overwritten by the new symlink. If the exact same symlink already exists, nothing is done.
• _parseYAML_ Convert a YAML file into BASH variables for use in a shell script
• _printFileBetween_ Prints block of text in a file between two regex patterns
• _randomLineFromFile_ Prints a random line from a file
• _readFile_ Prints each line of a file
• _sourceFile_ Source a file into a script
• _createUniqueFilename_ Ensure a file to be created has a unique filename to avoid overwriting other files
• _yaml2json_ Convert a YAML file to JSON with python

macOS.bash

Functions useful when writing scripts to be run on macOS

• _guiInput_ Ask for user input using a Mac dialog box
• _haveScriptableFinder_ Determine whether we can script the Finder or not
• _homebrewPath_ Adds Homebrew bin directory to PATH
• _useGNUUtils_ Add GNU utilities to PATH to allow consistent use of sed/grep/tar/etc. on MacOS

misc.bash

Miscellaneous functions

• _acquireScriptLock_ Acquire script lock to prevent running the same script a second time before the first instance exits
• _detectLinuxDistro_ Detects the host computer's distribution of Linux
• _detectMacOSVersion_ Detects the host computer's version of macOS
• _detectOS_ Detect the the host computer's operating system
• _endspin_ Clears output from the spinner
• _execute_ Executes commands with safety and logging options. Respects DRYRUN and VERBOSE flags.
• _findBaseDir_ Locates the real directory of the script being run. Similar to GNU readlink -n
• _generateUUID_ Generates a unique UUID
• _progressBar_ Prints a progress bar within a for/while loop
• _runAsRoot_ Run the requested command as root (via sudo if requested)
• _seekConfirmation_ Seek user input for yes/no question
• _spinner_ Creates a spinner within a for/while loop.
• _trapCleanup_ Cleans up after a trapped error.

services.bash

Functions to work with external services

• _haveInternet_ Tests to see if there is an active Internet connection
• _httpStatus_ Report the HTTP status of a specified URL
• _pushover_ Sends a notification via Pushover (Requires API keys)

strings.bash

Functions for string manipulation

• _cleanString_ Cleans a string of text
• _decodeHTML_ Decode HTML characters with sed. (Requires sed file)
• _decodeURL_ Decode a URL encoded string
• _encodeHTML_ Encode HTML characters with sed (Requires sed file)
• _encodeURL_ URL encode a string
• _escapeString_ Escapes a string by adding \ before special chars
• _lower_ Convert a string to lowercase
• _ltrim_ Removes all leading whitespace (from the left)
• _regexCapture_ Use regex to validate and parse strings
• _rtrim_ Removes all leading whitespace (from the right)
• _splitString_ Split a string based on a given delimiter
• _stringContains_ Tests whether a string matches a substring
• _stringRegex_ Tests whether a string matches a regex pattern
• _stripANSI_ Strips ANSI escape sequences from text
• _stripStopwords_ Removes common stopwords from a string using a list of sed replacements located in an external file.
• _trim_ Removes all leading/trailing whitespace
• _upper_ Convert a string to uppercase

template_utils.bash

Functions required to allow the script template and alert functions to be used

• _makeTempDir_ Creates a temp directory to house temporary files
• _safeExit_ Cleans up temporary files before exiting a script
• _setPATH_ Add directories to $PATH so script can find executables

Coding conventions

• Function names use camel case surrounded by underscores: _nameOfFunction_
• Local variable names use camel case with a starting underscore: _localVariable
• Global variables are in ALL_CAPS with underscores separating words
• Exceptions to the variable an function naming rules are made for alerting functions and colors to ease my speed of programming. (Breaking years of habits is hard...) I.e. notice "Some log item: ${blue}blue text${reset} Where notice is a function and $blue and $reset are global variables but are lowercase.
• Variables are always surrounded by quotes and brackets "${1}" (Overly verbose true, but a safe practice)
• Formatting is provided by shfmt using 4 spaces for indentation
• All scripts and functions are fully Shellcheck compliant
• Where possible, I follow defensive BASH programming principles.

A Note on Code Reuse and Prior Art

I compiled these scripting utilities over many years without having an intention to make them public. As a novice programmer, I have Googled, GitHubbed, and StackExchanged a path to solve my own scripting needs. I often lift a function whole-cloth from a GitHub repo don't keep track of its original location. I have done my best within these files to recreate my footsteps and give credit to the original creators of the code when possible. Unfortunately, I fear that I missed as many as I found. My goal in making this repository public is not to take credit for the code written by others. If you recognize something that I didn't credit, please let me know.

Contributing

Setup

1. Install Python 3.11 and Poetry
2. Clone this repository. git clone https://github.com/natelandau/shell-scripting-templates.git
3. Install the Poetry environment with poetry install.
4. Activate your Poetry environment with poetry shell.
5. Install the pre-commit hooks with pre-commit install --install-hooks.

Developing

• Activate your Poetry environment with poetry shell.
• This project follows the Conventional Commits standard to automate Semantic Versioning and Keep A Changelog with Commitizen.
  • When you're ready to commit changes run cz c
• Run poe from within the development environment to print a list of Poe the Poet tasks available to run on this project. Common commands:
  • poe lint runs all linters and tests
• Run poetry add {package} from within the development environment to install a runtime dependency and add it to pyproject.toml and poetry.lock.
• Run poetry remove {package} from within the development environment to uninstall a runtime dependency and remove it from pyproject.toml and poetry.lock.
• Run poetry update from within the development environment to upgrade all dependencies to the latest versions allowed by pyproject.toml.

License

MIT