Skip to content

Commit

Permalink
s3sync
Browse files Browse the repository at this point in the history
  • Loading branch information
grosser committed Sep 25, 2013
1 parent a3d9ab6 commit 81819ba
Show file tree
Hide file tree
Showing 18 changed files with 108 additions and 116 deletions.
2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
source 'https://rubygems.org'

# Specify your gem's dependencies in s3ranger.gemspec
# Specify your gem's dependencies in s3sync.gemspec
gemspec
4 changes: 2 additions & 2 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
PATH
remote: .
specs:
s3ranger (0.3.1)
s3sync (0.3.1)
aws-sdk
cmdparse

Expand Down Expand Up @@ -49,5 +49,5 @@ DEPENDENCIES
debugger
rake
rspec
s3ranger!
s3sync!
simplecov
32 changes: 12 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,37 @@
# S3Ranger

> Watch out, there's a new sheriff in town
*-- Said the waiter while he was polishing a glass*
# S3Sync

## Intro

I needed to backup some stuff once when I was in the woods. Unfortunately, I
didn't find anything easy but elegant enough to sync my stuff with Amazon S3.

The best guy I found was [s3sync](http://s3sync.net), but he was kinda
abandoned! So I decided to take care of him, I taught him some Kung fu, shown
him the mountains and he became S3Ranger!

S3Ranger uses the [official aws sdk for ruby](https://github.com/aws/aws-sdk-ruby)
S3Sync uses the [official aws sdk for ruby](https://github.com/aws/aws-sdk-ruby)
so we expect it to be stable. The most sensitive parts of the code are tested
and that only tends to get better, I'm crazy about testing code! :)

## Installation

$ gem install s3ranger
$ gem install s3sync

## Usage

S3Range's help command is pretty powerful, so you can get all the help you need
S3Sync's help command is pretty powerful, so you can get all the help you need
from him. He's always ready to answer your questions:

$ s3ranger help [SUBCOMMAND]
$ s3sync help [SUBCOMMAND]

If you want to learn more about a specific command, you just need to inform
the optional [SUBCOMMAND] argument:

$ s3ranger help sync
$ s3sync help sync

### Managing buckets

The following commands are used to manage buckets themselves

* `s3ranger listbuckets`: Show all available buckets
* `s3ranger createbucket <name>`: Create a new bucket
* `s3ranger deletebucket <name> [-f]`: Delete a bucket
* `s3sync listbuckets`: Show all available buckets
* `s3sync createbucket <name>`: Create a new bucket
* `s3sync deletebucket <name> [-f]`: Delete a bucket

### Managing content

Expand All @@ -54,7 +46,7 @@ The following commands are used to manage buckets themselves
If you want to sync up an s3 folder with a local folder (both directions are
accepted), you can use the `sync` command. e.g.:

$ s3ranger sync Work/reports disc.company.com:reports/2013/08
$ s3sync sync Work/reports disc.company.com:reports/2013/08

The above line will sync the local folder `Work/reports` with the remote node
`disc.company.com:reports/2013/08`.
Expand All @@ -76,9 +68,9 @@ The most important options of this command are:
### Feedback

Reporting bugs and giving any other feedback is highly appreciated. Feel free
to [create a new issue](https://github.com/clarete/s3ranger/issues/new) if you
to [create a new issue](https://github.com/clarete/s3sync/issues/new) if you
find anything wrong or if you have any ideas to improve the ranger!


[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/clarete/s3ranger/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/clarete/s3sync/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

24 changes: 12 additions & 12 deletions bin/s3ranger
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env ruby
#!/usr/bin/env ruby
#
# s3ranger - Tool belt for managing your S3 buckets
# s3sync - Tool belt for managing your S3 buckets
#
# The MIT License (MIT)
#
Expand All @@ -26,16 +26,16 @@

$:.unshift(File.dirname(__FILE__) + '/../lib') unless $:.include?(File.dirname(__FILE__) + '/../lib')

require "s3ranger/exceptions"
require "s3ranger/config"
require "s3ranger/cli"
require "s3sync/exceptions"
require "s3sync/config"
require "s3sync/cli"

conf = S3Ranger::Config.new
conf = S3Sync::Config.new

# Time to load config and see if we've got everything we need to cook our salad
begin
conf.read
rescue S3Ranger::NoConfigFound => exc
rescue S3Sync::NoConfigFound => exc
# We can't proceed without having those two vars set
$stderr.puts "You didn't set up the following environment variables:"
$stderr.puts
Expand All @@ -47,27 +47,27 @@ rescue S3Ranger::NoConfigFound => exc
exc.paths_checked.each {|path| $stderr.puts " * #{path}"}
$stderr.puts

$stderr.puts "You could try to set the `S3RANGER_PATH' environment variable"
$stderr.puts "You could try to set the `S3SYNC_PATH' environment variable"
$stderr.puts "pointing to a file to be loaded as your config file or just"
$stderr.puts "export those variables to your environment like this:"
$stderr.puts
exc.missing_vars.each {|var|
$stderr.puts " $ export #{var}=<value-provided-by-amazon>"
}
$stderr.puts
$stderr.puts "Learn how to do that here: https://github.com/clarete/s3ranger"
$stderr.puts "Learn how to do that here: https://github.com/clarete/s3sync"
exit
end

# Step aside, the star of this show is here. Let's try to create the
# environment to run the requested command. And feed the user back if
# information needed was not enough
begin
S3Ranger::CLI::run conf
rescue S3Ranger::FailureFeedback => exc
S3Sync::CLI::run conf
rescue S3Sync::FailureFeedback => exc
$stderr.puts exc.message
exit 1
rescue S3Ranger::WrongUsage => exc
rescue S3Sync::WrongUsage => exc
$stderr.puts "Error:\n #{exc.msg}\n" if exc.msg
exit exc.error_code
end
62 changes: 31 additions & 31 deletions doc/old/README.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Welcome to s3ranger.rb
Welcome to s3sync.rb
--------------------
Home page, wiki, forum, bug reports, etc: http://s3sync.net

Expand All @@ -7,19 +7,19 @@ directory and an S3 bucket:prefix. It behaves somewhat, but not precisely, like
the rsync program. In particular, it shares rsync's peculiar behavior that
trailing slashes on the source side are meaningful. See examples below.

One benefit over some other comparable tools is that s3ranger goes out of its way
to mirror the directory structure on S3. Meaning you don't *need* to use s3ranger
One benefit over some other comparable tools is that s3sync goes out of its way
to mirror the directory structure on S3. Meaning you don't *need* to use s3sync
later in order to view your files on S3. You can just as easily use an S3
shell, a web browser (if you used the --public-read option), etc. Note that
s3ranger is NOT necessarily going to be able to read files you uploaded via some
s3sync is NOT necessarily going to be able to read files you uploaded via some
other tool. This includes things uploaded with the old perl version! For best
results, start fresh!

s3ranger runs happily on linux, probably other *ix, and also Windows (except that
s3sync runs happily on linux, probably other *ix, and also Windows (except that
symlinks and permissions management features don't do anything on Windows). If
you get it running somewhere interesting let me know (see below)

s3ranger is free, and license terms are included in all the source files. If you
s3sync is free, and license terms are included in all the source files. If you
decide to make it better, or find bugs, please let me know.

The original inspiration for this tool is the perl script by the same name which
Expand All @@ -32,20 +32,20 @@ Examples:
---------
(using S3 bucket 'mybucket' and prefix 'pre')
Put the local etc directory itself into S3
s3ranger.rb -r /etc mybucket:pre
s3sync.rb -r /etc mybucket:pre
(This will yield S3 keys named pre/etc/...)
Put the contents of the local /etc dir into S3, rename dir:
s3ranger.rb -r /etc/ mybucket:pre/etcbackup
s3sync.rb -r /etc/ mybucket:pre/etcbackup
(This will yield S3 keys named pre/etcbackup/...)
Put contents of S3 "directory" etc into local dir
s3ranger.rb -r mybucket:pre/etc/ /root/etcrestore
s3sync.rb -r mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)
Put the contents of S3 "directory" etc into a local dir named etc
s3ranger.rb -r mybucket:pre/etc /root
s3sync.rb -r mybucket:pre/etc /root
(This will yield local files at /root/etc/...)
Put S3 nodes under the key pre/etc/ to the local dir etcrestore
**and create local dirs even if S3 side lacks dir nodes**
s3ranger.rb -r --make-dirs mybucket:pre/etc/ /root/etcrestore
s3sync.rb -r --make-dirs mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)


Expand All @@ -63,7 +63,7 @@ things), there are apt packages available for ruby and the open ssl lib.

Your environment:
-----------------
s3ranger needs to know several interesting values to work right. It looks for
s3sync needs to know several interesting values to work right. It looks for
them in the following environment variables -or- a s3config.yml file.
In the yml case, the names need to be lowercase (see example file).
Furthermore, the yml is searched for in the following locations, in order:
Expand All @@ -75,7 +75,7 @@ Required:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

If you don't know what these are, then s3ranger is probably not the
If you don't know what these are, then s3sync is probably not the
right tool for you to be starting out with.
Optional:
AWS_S3_HOST - I don't see why the default would ever be wrong
Expand All @@ -98,15 +98,15 @@ to REGULAR
I use "envdir" from the daemontools package to set up my env
variables easily: http://cr.yp.to/daemontools/envdir.html
For example:
envdir /root/s3ranger/env /root/s3ranger/s3ranger.rb -etc etc etc
envdir /root/s3sync/env /root/s3sync/s3sync.rb -etc etc etc
I know there are other similar tools out there as well.

You can also just call it in a shell script where you have exported the vars
first such as:
#!/bin/bash
export AWS_ACCESS_KEY_ID=valueGoesHere
...
s3ranger.rb -etc etc etc
s3sync.rb -etc etc etc

But by far the easiest (and newest) way to set this up is to put the name:value
pairs in a file named s3config.yml and let the yaml parser pick them up. There
Expand All @@ -125,7 +125,7 @@ companion utility s3cmd.rb. See README_s3cmd.txt.

About single files
------------------
s3ranger lacks the special case code that would be needed in order to handle a
s3sync lacks the special case code that would be needed in order to handle a
source/dest that's a single file. This isn't one of the supported use cases so
don't expect it to work. You can use the companion utility s3cmd.rb for single
get/puts.
Expand All @@ -137,17 +137,17 @@ In S3 there's no actual concept of folders, just keys and nodes. So, every tool
uses its own proprietary way of storing dir info (my scheme being the best
naturally) and in general the methods are not compatible.

If you populate S3 by some means *other than* s3ranger and then try to use s3ranger
If you populate S3 by some means *other than* s3sync and then try to use s3sync
to "get" the S3 stuff to a local filesystem, you will want to use the
--make-dirs option. This causes the local dirs to be created even if there is no
s3ranger-compatible directory node info stored on the S3 side. In other words,
s3sync-compatible directory node info stored on the S3 side. In other words,
local folders are conjured into existence whenever they are needed to make the
"get" succeed.


About MD5 hashes
----------------
s3ranger's normal operation is to compare the file size and MD5 hash of each item
s3sync's normal operation is to compare the file size and MD5 hash of each item
to decide whether it needs syncing. On the S3 side, these hashes are stored and
returned to us as the "ETag" of each item when the bucket is listed, so it's
very easy. On the local side, the MD5 must be calculated by pushing every byte
Expand Down Expand Up @@ -188,7 +188,7 @@ using more than one CA.

Getting started:
----------------
Invoke by typing s3ranger.rb and you should get a nice usage screen.
Invoke by typing s3sync.rb and you should get a nice usage screen.
Options can be specified in short or long form (except --delete, which has no
short form)

Expand All @@ -200,10 +200,10 @@ precious, precious files.
If you use the --public-read(-p) option, items sent to S3 will be ACL'd so that
anonymous web users can download them, given the correct URL. This could be
useful if you intend to publish directories of information for others to see.
For example, I use s3ranger to publish itself to its home on S3 via the following
command: s3ranger.rb -v -p publish/ ServEdge_pub:s3ranger Where the files live in a
For example, I use s3sync to publish itself to its home on S3 via the following
command: s3sync.rb -v -p publish/ ServEdge_pub:s3sync Where the files live in a
local folder called "publish" and I wish them to be copied to the URL:
http://s3.amazonaws.com/ServEdge_pub/s3ranger/... If you use --ssl(-s) then your
http://s3.amazonaws.com/ServEdge_pub/s3sync/... If you use --ssl(-s) then your
connections with S3 will be encrypted. Otherwise your data will be sent in clear
form, i.e. easy to intercept by malicious parties.

Expand All @@ -215,8 +215,8 @@ cherished and irreplaceable data.

Updates and other discussion:
-----------------------------
The latest version of s3ranger should normally be at:
http://s3.amazonaws.com/ServEdge_pub/s3ranger/s3ranger.tar.gz
The latest version of s3sync should normally be at:
http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
and the Amazon S3 forums probably have a few threads going on it at any given
time. I may not always see things posted to the threads, so if you want you can
contact me at [email protected] too.
Expand Down Expand Up @@ -329,7 +329,7 @@ which we need in order to make some decisios in the local generator.
prefix.
- Fixed S3->local when there's no "/" in the source so it doesn't try to create
a folder with the bucket name.
- Updated s3try and s3_s3ranger_mod to allow SSL_CERT_FILE
- Updated s3try and s3_s3sync_mod to allow SSL_CERT_FILE
----------

2007-02-22
Expand All @@ -346,16 +346,16 @@ Added --progress
Version 1.1.3
IMPORTANT!
Pursuant to http://s3sync.net/forum/index.php?topic=49.0 , the tar.gz now
expands into its own sub-directory named "s3ranger" instead of dumping all the
expands into its own sub-directory named "s3sync" instead of dumping all the
files into the current directory.

In the case of commands of the form:
s3ranger -r somedir somebucket:
s3sync -r somedir somebucket:
The root directory node in s3 was being stored as "somedir/" instead of "somedir"
which caused restores to mess up when you say:
s3ranger -r somebucket: restoredir
s3sync -r somebucket: restoredir
The fix to this, by coincidence, actually makes s3fox work even *less* well with
s3ranger. I really need to build my own xul+javascript s3 GUI some day.
s3sync. I really need to build my own xul+javascript s3 GUI some day.

Also fixed some of the NoMethodError stuff for when --progress is used
and caught Errno::ETIMEDOUT
Expand Down Expand Up @@ -403,4 +403,4 @@ Version 1.2.6
Catch connect errors and retry.
----------

FNORD
FNORD
Loading

0 comments on commit 81819ba

Please sign in to comment.