Include/Exclude mechanism

[memeplex]

As go regexps don't support negative lookahead is difficult to exclude some set defined by a barely selective pattern (like ^\. in a home directory) and yet include some small subset of it (say some dotfiles I want to sync).

[odeke-em]

Hello @memeplex, thank you for reporting this and welcome to drive!

Would you mind giving an example for your use case because I don't yet understand the deficiency?

[memeplex]

Sure. Say I want to sync the part of my home directory with contents generated by me. This will include some dotfiles and most directories and files that don't start with a dot. But there is a large number of dotfiles that I don't want to sync at all. So the rules will be something like:

-^\.*$
+^\.mydofile1$'
+^\.mydotfile2$'

I could do that with a negative lookahead as follows:

exclude=^\.(?!(mydotfile1|mydotfile2))$

But (?! is not supported by go.

[memeplex]

See for example rclone, which implements rule based filtering: http://rclone.org/filtering/

[odeke-em]

Alright, thanks for the example, am gonna take a look at this and make a couple of comparisons then get back to you.

[memeplex]

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

[memeplex]

It only seems to work when .mydofile1 and .mydotfile2 are already in the server, then changes are pushed. Is it expected to work like that?

[odeke-em]

So I've ran your tests to compare with other languages e.g Python and none has matched, maybe a revision of the regex might change things. Please see https://gist.github.com/odeke-em/c29ae8bc475fd0fb9eda.

In regards to:

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

drive won't push anything that you've already synced and needs no changes unless you request an explicit push in this case with a --force flag. Please see https://github.com/odeke-em/drive#pushing. Otherwise if the content changes then sure it will push it up or if modTime changes occur.

Would you mind giving me your drive version, ie after running

$ go get github.com/odeke-em/drive/drive-gen && drive-gen
$ driver version

that should give you information kind of similar to this

$ go get github.com/odeke-em/drive/drive-gen && drive-gen && drive version
drive version: 0.3.4
Commit Hash: '6992e27390b35ec51db59ae1d03e0cb439bbb38b'
Go Version: devel +337c5541 Sun Dec 20 20:08:59 2015 -0700
OS: darwin/amd64
BuildTime: 2015-12-27 20:28:07.005553354 -0700 MST

[odeke-em]

Oops, I forgot to mention that hidden files by default are ignored, please use the --hidden flag during a pull or push.

[odeke-em]

I think I can also reproduce the .dot files not being pushed. I'll file a bug about .dot files not being pushed, probably a regression. Thanks for that catch!

[memeplex]

Regarding the regexp: just remove the ending $.

Regarding the push thing: I'm not testing it with hidden files. What I'm doing is:

1. Create a new file or directory xxx.
2. Run drive push xxx.

Then nothing happens. Now if I push the parent directory of xxx, then xxx is added to the server. After that I:

1. Modify the contents of xxx.
2. Run drive push xxx.

Then xxx is updated in the server.

[odeke-em]

Roger that. I can confirm that I broke this functionality with PR #528 and here is a breakdown of what I screwed up:
Before a raw lookup was done for the first file, if none was returned, an appropriate error and nil file was passed back so a need to sync would be made.
Now a channel of files ie get all the matches is returned and if no match was made, that channel is closed and no files are registered as registered whether empty or not hence no syncing ever occurs since matches are within a loop ie

for rem := range remotesChan {
   ....
}

// Will never be run if the file didn't exist upstream.

Am patching this up and thank you very much for bringing this to my attention. Fix coming up soon.

[odeke-em]

@memeplex please get the latest from master and your work around should be working. The bug in your push/pull workaround that you found has been addressed by PR #537.

[odeke-em]

You can do so if you don't already know by

$ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gen

[memeplex]

Thank you very much. Despite include/exclude support would be nice lacking regexp lookahead, I think I can workaround that with some shell glob matching and your fix.

[memeplex]

See for example extglob for bash:
https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html#Pattern-Matching
It allows for powerful selective file expansion in a simple way. Coupled with the ability to push/pull only some paths, this renders the ignore file more or less obsolete.

[odeke-em]

Ah I see, thanks for the references. I'll keep digging and then I'll see if we can get this in, but I do agree the ability to match not all files in a match set is definitely useful.

If you don't mind and can give me feedback, I could hack up for you a solution but basically inside your .driveignore
for each clause that you'd like to exclude from ignoring, you'd have to use the #exclude: tag

^\. #this one gets ignored
#exclude:(mydotfile|only|others)

It is 10:07PM and except for going out to lunches and etc, I don't have to be busy till the 4th of January 2016 so I think I could rig up something for you in this time. Please let me know if you are interested then we can proceed.

[memeplex]

Looks fine to me. I would avoid the word exclude here, I understand you're using it as a "double negation" in "exclude from ignore", but it usually means the same than just ignore. Also some escaping mechanism will be in order as the tag could be part of a valid regexp itself. A leading ! could be just fine, escaped with !.

[odeke-em]

Good point right there, a '!' actually is simpler and is related to the original deficiency ?!.
Aye aye, I'll ping you soon with a PR.

[odeke-em]

@memeplex I got sometime before bed and I've spun out a PR for you #541. Please help me test it out to see if it matches your use cases and it looks alright. Please don't hesitate to reach out in case of any questions or any ideas on this.
Thank you.

[odeke-em]

Hey there @memeplex, I just wanted to let you know that now you can do the inclusion/exclusion you had wanted.

# My .driveignore file
# Note that it can be put in ~/ or ever single directory for which you want one customized
# e.g mySchoolDrive/.driveignore, myWorkDrive/a/b/.driveignore myWorkDrive/a/c/.driveignore
^\.
!^\.dockercfg
!^\.bashrc|\.info

Please get the latest from master and thank you for requesting for this and for the discussion!

[memeplex]

Thank you very much, you're the best.

[ghost]

Is it possible in .driveignore do something similar to .gitignore:

//when you want to commit just the content of magic1 and magic2 
/*
!src/
src/*
!src/someBigFolder/
src/someBigFolder/*
!src/someBigFolder/magic1/
!src/someBigFolder/magic2/

[odeke-em]

Hello there @antigone2, thanks for the question and welcome to drive.

I don't know, never tried it, please try and let us know if it works :)

[ghost]

Well, @odeke-em the thing is I have failed to figure it out, that is why I am writing :)

The goal is this:

Situation:

1. You have established gDrive folder
2. You have created the a symlink ln -s /from/your/wokring/direcotory /to/gDrive/

Now you want to set up .driveignore to ignore all but some sub/sub folders in your working direcotory - typically you want to backup to drive just source codes and config files, no compiled results etc.

Only thing what was working for me was:

example situation

let's say you have in gDrive folder container in which you have test1, test 2, test3 folders which all contain some files, and you want only test1 and all its content to sync.

to ignore all what f.e. starts with /container/, but not what starst with !^/container/test1. That worked, but it is not a perfect solution and it is not working on next levels.

So my question really is if is it just something simple I am too dumb to figure out, or is it impossible to do by design. Thank you very much for your answer, btw nice project, up-to date ppa, thank you and @rakyll for doing it - exactly what you need on Ubuntu.