Fast s3 utility is a faster version of s3cmd's ls and del functions ideal for listing and deleting buckets containing millions of keys.

#Installation

go get github.com/TuneOSS/fasts3
cd $GOPATH/src/github.com/TuneOSS/fasts3
go build

#Configuration

use fasts3 init command which will create a template file in ~/.fs3cfg

[default]
access_key=<access_key>
secret_key=<secret_key>

fill in the template file with your s3 credentials

alternatively you can set these environment variables:

export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>

#Usage

usage: fasts3 <command> [<flags>] [<args> ...]

Multi-threaded s3 utility

Flags:
  --help  Show help.

Commands:
  help [<command>]
    Show help for a command.

  ls [<flags>] <s3uri>
    List s3 prefixes.

  del [<flags>] [<prefixes>]
    Delete s3 keys

  get [<flags>] [<prefixes>]
    Fetch files from s3

  stream [<flags>] [<prefixes>]
    Stream s3 files to stdout

  init
    Initialize .fs3cfg file in home directory

#####ls

usage: fasts3 [<flags>] ls [<flags>] <s3uri>

List s3 prefixes.

Flags:
  --help            Show help.
  -r, --recursive   Get all keys for this prefix.
  --search-depth=0  search depth to search for work.
  -H, --human-readable  human readable key size.
  -d, --with-date   include the last modified date.

Args:
  <s3uri>  paritial s3 uri to list, ex: s3://mary/had/a/little/lamb/

#####del

usage: fasts3 [<flags>] del [<flags>] [<prefixes>]

Delete s3 keys

Flags:
  --help            Show help.
  -r, --recursive   Delete all keys with prefix
  --search-depth=0  search depth to search for work.

Args:
  [<prefixes>]  1 or more partial s3 uris to delete delimited by space

#####get

usage: fasts3 get [<flags>] [<prefixes>]

Fetch files from s3

Flags:
  --search-depth=0  search depth to search for work.

Args:
  [<prefixes>]  list of prefixes or s3Uris to retrieve

#####stream

usage: fasts3 stream [<flags>] [<prefixes>]

Stream s3 files to stdout

Flags:
  --search-depth=0  search depth to search for work.
  --key-regex=KEY-REGEX
                    regex filter for keys

Args:
  [<prefixes>]  list of prefixes or s3Uris to retrieve

####Using search depth to go faster Many times you know the structure of your s3 bucket, this can be used to optimize listings. Say you have a structure like so:

fasts3 ls s3://mybuck/logs/

DIR s3://mybuck/logs/2010/
DIR s3://mybuck/logs/2012/
DIR s3://mybuck/logs/2013/
DIR s3://mybuck/logs/2014/
DIR s3://mybuck/logs/2015/

doing a fasts3 ls -r s3://mybuck/logs/ will read all keys under logs sequentially. We can make this faster by adding a --search-depth 1 flag to the command which gives each of the underlying directories it's own thread increasing throughput.

####Examples

# ls
fasts3 ls s3://mybucket/ # lists top level directories and keys
fasts3 ls -r s3://mybucket/ # lists all keys in the bucket
fasts3 ls -r --search-depth 1 s3://mybucket/ # lists all keys in the bucket using the directories 1 level down to thread

# del
fasts3 del -r s3://mybuck/logs/ # deletes all keys in the prefix
fasts3 del s3://mybuck/logs/2015/01/12/api.log.201501122359.gz # deletes single key
fasts3 del $(fasts3 ls s3://mybuck/logs/2015/01/12 | awk -F " " '/api.log/{print $2}') # delete all keys that have "api.log" in them

#get
fasts3 get s3://mybuck/logs/ # fetches all logs in the prefix

# stream
fasts3 stream s3://mybuck/logs/ # streams all logs under prefix to stdout
fasts3 stream --key-filter ".*2015-01-01" s3://mybuck/logs/ # streams all logs with 2015-01-01 in the key name stdout