Separate duplicacy backup from duplicacy copy operations

In some backup scenerios, you may want to do a "duplicacy backup"
operation without doing a "duplicacy copy" operation, or you may
want to do a "duplicacy copy" operation on it's own. Modify command
line options to allow for this.

This fixes issue #18.

README.md

@@ -418,26 +418,34 @@ This will generate output similar to:
 
 ```text
 Usage of ./duplicacy-util:
-  -a	Perform all duplicacy operations (backup/copy, purge, check)
-  -b	Perform duplicacy backup/copy operation
-  -c	Perform duplicacy check operation
-  -d	Enable debug output (implies verbose)
+  -a    Perform all duplicacy operations (backup, copy, purge, check)
+  -b    Perform duplicacy backup operation (deprecated; use -backup -copy)
+  -backup
+        Perform duplicacy backup operation
+  -c    Perform duplicacy check operation (deprecated; use -check)
+  -check
+        Perform duplicacy check operation
+  -copy
+        Perform duplicacy copy operation
+  -d    Enable debug output (implies verbose)
   -f string
-    	Configuration file for storage definitions (must be specified)
+        Configuration file for storage definitions (must be specified)
   -g string
-    	Global configuration file name
-  -m	(Deprecated) Send E-Mail with results of operations (implies quiet)
-  -p	Perform duplicacy prune operation
-  -q	Quiet operations (generate output only in case of error)
+        Global configuration file name
+  -m    (Deprecated) Send E-Mail with results of operations (implies quiet)
+  -p    Perform duplicacy prune operation (deprecated; use -prune)
+  -prune
+        Perform duplicacy prune operation
+  -q    Quiet operations (generate output only in case of error)
   -sd string
-    	Full path to storage directory for configuration/log files
+        Full path to storage directory for configuration/log files
   -tm
-    	(Deprecated: Use -tn instead) Send a test message via E-Mail
+        (Deprecated: Use -tn instead) Send a test message via E-Mail
   -tn
-    	Test notifications
-  -v	Enable verbose output
+        Test notifications
+  -v    Enable verbose output
   -version
-    	Display version number
+        Display version number
 ```
 
 Exit codes from `duplicacy-util` are as follows:

UPGRADING.md

@@ -0,0 +1,158 @@
+# UPGRADING.md - Notes on upgrading from prior versions
+
+Table of contents:
+
+- [Explanation of Upgrade Policy](#explanation-of-upgrade-policy)
+- [Changes made in `v1.4`(#changes-made-in-v1.4)
+  - [Notifications](#v1.4-notifications)
+  - [Command Line Changes](#v1.4-command-line-changes]
+
+### Explanation of Upgrade Policy
+
+While reasonable attempts are made to insure compatibility from release
+to release, sometimes changes are necessary.
+
+As a policy, when an incompatible change is made, old behavior will be
+maintained for two releases (major or minor). So, for example, if
+`duplicacy-util v1.4` introduces incompatible changes, old behavior will
+be maintained until `v1.6` (assuming no major releases).
+
+When incompatible changes are made, old code paths will not get new
+features. They simply become stale until removed. If you want to use a
+new feature, and that section was involved in an incompatible change,
+then you must move to new functionality to utilize new features.
+
+As an example: the `email` configuration formatting changed in `v1.4`
+(in the global configuration file). Additionally, `acceptInsecureCerts`
+was added to `email` to support insure certificates. If you wish to use
+this new setting, then you MUST update to the new `email` configuration.
+
+As a result, it is recommended to immediately incorporate updates to
+scripting or configuration files when upgrading to a new version.
+
+** Note: This file only describes incompatible changes made in a release,
+not ALL changes made in a release. Refer to release notes on GitHub for a
+full description of all changes made, or new features that did not create
+a compatibility issue. **
+
+Finally, note that [README.md](README.md) only describes most current
+configurations. If you're running the master branch before releases,
+you should monitor issues raised and commits made so you know when changes
+are made to the master branch.
+
+### Changes made in v1.4
+
+Incompatible changes in `v1.4` involve to areas of the code base:
+
+1. Notifications (and how email is affected)
+1. Command line changes
+
+#### v1.4: Notifications
+
+This release introduces notifications, an extendible mechanism to notify
+of backup results. In the past, `duplicacy-util` only had the notion of
+sending email. With extendable notificaitons, `duplicacy-util` can easily
+have new forms of notifications added (push notifications to mobile devices,
+desktop notificaitons, etc). This change involved removing email-specific
+commnd line options (namely `-m` and `-tm`) and global configuration changes.
+
+The `-m` (send mail) option was deprecated because, with notifications, that
+behavior is automatic. You don't need to specify a command line option to
+trigger notifications.
+
+The `-tm` (test mail) option was deprecated because it only tests for email.
+A new option, `-tn` (test notifications), was created to test notifications.
+Note that `-tn` only works if you actually set up notifications in the
+global configuration file.
+
+Notifications are triggered at certain times in the backup process:
+
+| Type      | When triggered |
+| ----      | -------------- |
+| onStart   | Triggered at the beginning of a backup |
+| onSkip    | Triggered when a backup is skipped (if already running) |
+| onSuccess | Triggered when a backup completes successfully |
+| onfailure | Triggered when a backup fails due to an error |
+
+If you wish to be notified, mention the notification type in a
+comma-separated (if multiple notifications are to be triggered) list.
+Be aware that, in `v1.4`, only `email` notifications are supported. If
+you do not wish to be notified, do not specify the notification type
+in the notifier.
+
+Notifications are specified in the global configuration file with a
+section like:
+
+```
+notifications:
+  onStart: []
+  onSkip: ['email']
+  onSuccess: ['email']
+  onFailure: ['email']
+```
+
+In this example, no notification would be triggered when a backup was
+started, but email would be sent when a backup was skipped, succeeded,
+or failed.
+
+Additionally, the `email` configuration was changed as follows:
+
+| Old email setting | New email setting |
+| ----------------- | ----------------- |
+| emailFromAddress | fromAddress (in section email) |
+| emailToAddress | toAdddress (in section email) |
+| emailserverHostname | serverHostname (in section email) |
+| emailServerPort | serverPort (in section email) |
+| emailAuthUsername | authUsername (in section email) |
+| emailAuthPassword | authPassword (in section email) |
+
+Thus, if you had an email configuration such as:
+
+```
+emailFromAddress: "Donald Duck <[email protected]>"
+emailToAddress: "Donald Duck <[email protected]>"
+emailServerHostname: smtp.gmail.com
+emailServerPort: 465
+emailAuthUsername: [email protected]
+emailAuthPassword: gaozqlwbztypagwt
+```
+
+then you should replace that with the following section:
+
+```
+email:
+  fromAddress: "Donald Duck <[email protected]>"
+  toAddress: "Donald Duck <[email protected]>"
+  serverHostname: smtp.gmail.com
+  serverPort: 465
+  authUsername: [email protected]
+  authPassword: gaozqlwbztypagwt
+```
+
+Note that old `email*` flags will be removed in `v1.6`. Additionally,
+new email options are available, but you must use new format to utilize
+them.
+
+
+#### v1.4: Command Line Changes
+
+New command line options:
+
+| Option | Purpose |
+| ------ | ------- |
+| -backup | Specifically perform a `duplicacy backup` operation. Can be combined with `-copy`, `-check`, and `-prune`. |
+| -copy | Specifically perform a `duplicacy copy` operation. Can be combine with `-backup`, `-check`, and `-prune`. |
+| -check | Specifically perform a `duplicacy check` operation. Can be combined with `-backup`, `-copy`, and `-prune`. |
+| -prune | Specifically perform a `duplicacy prune` operation. Can be combined with `-backup`, `-copy`, and `-check`. |
+| -tn | Test notifications (trigger each notification type configured in global configuration file).
+
+Deprecated command line options:
+
+| Option | Purpose |
+| ------ | ------- |
+| -b | Performed a `duplicacy backup` operation. Use `-backup` instead. |
+| -c | Performed a `duplicacy check` operaiton. Use `-check` instead. |
+| -p | Performed a `duplicacy prune` operation. Use `-prune` instead. |
+| -tm | Triggered test email messages. Configure notifications in global configuration and use `-tn` instead. |
+
+Note that deprecated command line options will be removed in `v1.6`.

backup-ops.go

@@ -78,6 +78,13 @@ func performBackup() error {
 		}
 	}
 
+	// Perform "duplicacy copy" if required
+	if cmdCopy {
+		if err := performDuplicacyCopy(logger, []string{}); err != nil {
+			return err
+		}
+	}
+
 	// Perform "duplicacy prune" if required
 	if cmdPrune {
 		if err := performDuplicacyPrune(logger, []string{}); err != nil {
@@ -106,7 +113,6 @@ func performBackup() error {
 func performDuplicacyBackup(logger *log.Logger, testArgs []string) error {
 	// Handling when processing output from "duplicacy backup" command
 	var backupEntry backupRevision
-	var copyEntry copyRevision
 
 	backupLogger := func(line string) {
 		switch {
@@ -153,26 +159,6 @@ func performDuplicacyBackup(logger *log.Logger, testArgs []string) error {
 		}
 	}
 
-	copyLogger := func(line string) {
-		switch {
-		// Copy complete, 107 total chunks, 0 chunks copied, 107 skipped
-		case strings.HasPrefix(line, "Copy complete, "):
-			logger.Println(line)
-			logMessage(logger, fmt.Sprint("  ", line))
-
-			// Save chunk data for inclusion into HTML portion of E-Mail message
-			re := regexp.MustCompile(`Copy complete, (\S+) total chunks, (\S+) chunks copied, (\S+) skipped`)
-			elements := re.FindStringSubmatch(line)
-			if len(elements) >= 4 {
-				copyEntry.chunkTotalCount = elements[1]
-				copyEntry.chunkCopyCount = elements[2]
-				copyEntry.chunkSkipCount = elements[3]
-			}
-		default:
-			logger.Println(line)
-		}
-	}
-
 	// Perform backup/copy operations
 	for i, backupInfo := range configFile.backupInfo {
 		backupStartTime := time.Now()
@@ -223,6 +209,33 @@ func performDuplicacyBackup(logger *log.Logger, testArgs []string) error {
 		backupTable = append(backupTable, backupEntry)
 	}
 
+	return nil
+}
+
+func performDuplicacyCopy(logger *log.Logger, testArgs []string) error {
+	// Handling when processing output from "duplicacy backup" command
+	var copyEntry copyRevision
+
+	copyLogger := func(line string) {
+		switch {
+		// Copy complete, 107 total chunks, 0 chunks copied, 107 skipped
+		case strings.HasPrefix(line, "Copy complete, "):
+			logger.Println(line)
+			logMessage(logger, fmt.Sprint("  ", line))
+
+			// Save chunk data for inclusion into HTML portion of E-Mail message
+			re := regexp.MustCompile(`Copy complete, (\S+) total chunks, (\S+) chunks copied, (\S+) skipped`)
+			elements := re.FindStringSubmatch(line)
+			if len(elements) >= 4 {
+				copyEntry.chunkTotalCount = elements[1]
+				copyEntry.chunkCopyCount = elements[2]
+				copyEntry.chunkSkipCount = elements[3]
+			}
+		default:
+			logger.Println(line)
+		}
+	}
+
 	for i, copyInfo := range configFile.copyInfo {
 		copyStartTime := time.Now()
 		logger.Println("######################################################################")

backup-ops_test.go

@@ -53,30 +53,79 @@ func TestRunDuplicacyBackup(t *testing.T) {
 		assetInputFragment string
 		resultsFile        string
 		backupInfo         []map[string]string
-		copyInfo           []map[string]string
 	}{
 		// Dupicacy Error: Enter Backblaze Account ID:Enter Backblaze Application Key:Failed to load the Backblaze B2 storage at b2://hidden-bucket: Authorization failure
 		{
 			"account_id.log", "account_id.log_results_backup",
 			[]map[string]string{
 				{"name": "b2", "threads": "10", "vss": "false"},
 			},
-			[]map[string]string{},
 		},
 		// Duplicacy Error: Enter storage password:Failed to read the password: EOF
 		{
 			"storagepw.log", "storagepw.log_results_backup",
 			[]map[string]string{
 				{"name": "b2", "threads": "5", "vss": "false"},
 			},
-			[]map[string]string{},
 		},
 		// Test of long, very involved backup
 		{"taltos.log", "taltos.log_results_backup",
 			[]map[string]string{
 				{"name": "gcd", "threads": "5", "vss": "false"},
 				{"name": "azure-direct", "threads": "10", "vss": "false"},
 			},
+		},
+	}
+
+	for _, test := range tests {
+		// Set up logging infrastructure
+		logger, file, err := setupLogging()
+		if err != nil {
+			t.Errorf("unexpected error creating log file, got %#v", err)
+		}
+		loggingSystemDisplayTime = false
+		quietFlag = true
+		defer func() {
+			file.Close()
+			os.Remove(file.Name()) // For debugging, might need to leave log file around for perusal
+
+			loggingSystemDisplayTime = true
+			quietFlag = false
+		}()
+
+		// Initialize data structures for test
+		configFile.backupInfo = test.backupInfo
+		mailBody = nil
+		//defer os.Remove(file.Name())
+
+		execCommand = fakeBackupOpsCommand
+		defer func() { execCommand = exec.Command }()
+		if err := performDuplicacyBackup(logger, []string{"testbackup", test.assetInputFragment}); err != nil {
+			t.Errorf("expected nil error, got %v", err)
+		}
+
+		// Check results of anon function
+		expectedOutputInBytes, err := ioutil.ReadFile(path.Join("test/assets", test.resultsFile))
+		if err != nil {
+			t.Errorf("unable to read backup results file %s", err)
+			return
+		}
+		expectedOutput := string(expectedOutputInBytes)
+		actualOutput := strings.Join(mailBody, "\n") + "\n"
+		if actualOutput != expectedOutput {
+			t.Errorf("result was incorrect, got\n=====\n%s=====\nexpected\n=====\n%s=====", actualOutput, expectedOutput)
+		}
+	}
+}
+
+func TestRunDuplicacyCopy(t *testing.T) {
+	tests := []struct {
+		assetInputFragment string
+		resultsFile        string
+		copyInfo           []map[string]string
+	}{
+		// Test of long, very involved copy operation
+		{"taltos.log", "taltos.log_results_copy",
 			[]map[string]string{
 				{"from": "gcd", "to": "azure", "threads": "5"},
 			},
@@ -100,14 +149,13 @@ func TestRunDuplicacyBackup(t *testing.T) {
 		}()
 
 		// Initialize data structures for test
-		configFile.backupInfo = test.backupInfo
 		configFile.copyInfo = test.copyInfo
 		mailBody = nil
 		//defer os.Remove(file.Name())
 
 		execCommand = fakeBackupOpsCommand
 		defer func() { execCommand = exec.Command }()
-		if err := performDuplicacyBackup(logger, []string{"testbackup", test.assetInputFragment}); err != nil {
+		if err := performDuplicacyCopy(logger, []string{"testbackup", test.assetInputFragment}); err != nil {
 			t.Errorf("expected nil error, got %v", err)
 		}