Feature request: Listen could *really ignore* ignored directories (optimization)

[e2]

TL; DR - UPDATE: this feature doesn't make sense due to technical limitations of the adapters.

Listen currently watches every file in a watched directory and applies the filter after receiving system events.

On optimized backends, this causes lots of unnecessary calls through Ruby upon every file change.

(e.g. changes in db/test.sqlite trigger changes through Listen)

How to implement (minimal work)

1. disable recursion on all optimized adapters: windows.rb (see wdm), darwin.rb (see: rb-fsevent) [note, "disabling" here means ignoring reported subdirs], linux.rb (see: rb-inotify))
2. get listen working again by implementing the recursion in Ruby in the base adapter (rb-inotify has a great Ruby implementation of recursion)
3. get the adapter to turn a watched directory into an array of directories to watch non-recursively here
4. Finally, translate physical paths backs into symlink paths (if they exist) here: https://github.com/guard/listen/blob/master/lib/listen/adapter/base.rb#L47

[e2]

@thibaudgg - is it possible to turn off recursion in rb-fsevent? (and I'm wondering if I haven't already asked this before).

The workaround is to simply ignore events on subdirectories, but in this case means potentially giving each rb-fsevent instance a full list of all subdirectories containing each other, e.g. if I have:

foo/bar/baz/bam/, then rb-fsevent will get: foo, foo/bar, foo/bar/baz, foo/bar/baz/bam.

Would that affect rb-fsevent performance for large projects on the rb-fsevent side? Would listen get multiple events (4 in this case) if something is modified in foo/bar/baz/bam?

[thibaudgg]

@e2 sadly not, I don't think this is possible with the actual implementation of rb-fsevent.

[e2]

@thibaudgg - thanks! You saved me time digging into this - which I appreciate a lot!

[skizzybiz]

This is disastrous when interacting with #273 -- when guard uses listen, it apparently now listens to the whole root of your project, regardless of the watch directives you've added, so even if you add ignore directives to ignore the dirs with symlinks in them, listen pays them no mind when it's scanning. Effectively, guard 2.8.0 is unusable for me.

[e2]

@skizzybiz - thanks for mentioning this!

TL;DR - use Guard's -w option (check README)

First, what you want is guard's -w option to select which directories to watch. I'm sorry if guard's documentation is too extensive/discouraging/confusing.

Maybe it requires rework because of the -w option which is so much more important now? If you have any suggestions about that, let me know, thanks.

This is disastrous when interacting with #273

While this is technically unrelated (because of the -w option), you've just mentioned a great idea to use guard's watch statements for automatically selecting the right directories to watch: guard/guard#674

(Add comments on +1's if you agree).

I'm sorry it's so confusing with how things work, i.e.:

• guard's '-w` option is so important, yet it lacks highlights in the docs
• listen's :ignore option has always confused users, because it doesn't affect watching/scanning (except for polling)

Effectively, guard 2.8.0 is unusable for me.

I hope that will change soon, although I admit I'm doing my best to refactor Guard and Listen, so these feature can be added quickly without breaking things and causing grief.

All I ask for is some patience (and a bit of support, even cheering means a lot), because both projects have a TON of legacy functionality and API assumptions other plugins and projects heavily rely on.

Even removing a useless method deprecated since 2012 caused annoyances.

And some of the existing open bugs (both in Listen and Guard) require MASSIVE changes in the internal architecture - just to fix them (I'm currently working on it).

Anyway, thanks for chiming in - every bit of feedback is precious, because there's too many specific cases I can miss - and too many aren't covered by tests yet.

[skizzybiz]

Thanks for the response! Sorry the codebase is a bit of a mess there. I think that guard/guard#674 is a more urgent need, as far as my problem goes.

[e2]

@skizzybiz - it isn't as much of a mess, it's just that the project grew so fast, it became very difficult to come up with a clear, maintainable internal architecture - because of how submodules are so interdependent (e.g. being able to configure Guard from the commandline, but still be able to reconfigure it at runtime by reloading the Guardfile, how to both be able to notify about actions and still have actions which configure notifications).

Just the user request and demands have been really tough to fulfill within an application, while still keeping backward compatibility (notice Guard has currently few bugs, while still having a 2-year old API!!).

[bradgessler]

I just updated to 2.8 and was surprised to see the error about symlinks. Naturally I thought to myself, "oh, there's probably some ignore directive I can set," so I found ignore, tried it, and it didn't fix the problem. I finally tracked down this thread and #273 and understand why, but it is very confusing and unclear. I'd at least mention the -w switch in the error message as a work-around until the ignore directive is properly sorted out.

[e2]

@bradgessler - thanks for the feedback. I've released 2.8.1 just with better tips on dealing with this (and how the ignore option doesn't work).

[bradgessler]

@e2 that message is much more informative!

[e2]

@bradgessler - thanks, it's the least I can do at the moment...

[rmm5t]

TL;DR - use Guard's -w option (check README)

For now, I actually find this to be the easiest workaround so far. The -w command line option is cumbersome.

  # Until listen has better support for recursive symlinks or until Guardfiles
  # can specify a list of watchdirs.
  # See https://github.com/guard/guard/issues/674
  # See https://github.com/guard/listen/pull/273
  # See https://github.com/guard/listen/pull/274
  gem "listen", "~> 2.7.12"

[bradgessler]

Would it be too late to yank 2.8.x until the symlink issues are resolved?

[rmm5t]

Would it be too late to yank 2.8.x until the symlink issues are resolved?

Considering the 2.8 release notes only mention this new abort behavior, and officially, listen now breaks semver after this change, I'm a 👍 vote on yanking 2.8. I think this should have been released as a deprecation warning in v2.8 first instead of an outright abort. If a true abort is necessary, I think it should have been held back until v3.

[bradgessler]

👍

[e2]

TL;DR - 2.8 behavior is not a bug; 2.7.x behavior is a bug; it doesn't break semver; don't upgrade to 2.8 if it's too inconvenient (that was the point of releasing it as 2.8 instead of 2.7.13).

@rmm5t - all because Listen now protects you from possible filesystem loops and erratic behavior with symlinks, doesn't mean semver was broken (please quote a line from there if you disagree). On the contrary - there was no backward-incompatible API change (unless you treat bugs as part of the specification).

There's no need to yank 2.8, since it doesn't prevent further bugfixes on 2.7.x. If there's a bug in 2.8 or 2.7.x, please report it (as a new issue).

If there's a backward incompatibility, please point me to how Listen 2.8 does not work according to 2.7 documentation.

Also, pull requests are welcome (ideally, with specs).

All because '-w' option is cumbersome doesn't mean 2.8 is botched. If at all, the whole Listen architecture is "botched" since it's inception - and if you dig deep enough you'll basically find Apple responsible for that (their FSevent API/implementation).

Seriously, be professional about this - as expected from someone from Thoughtbot (which I respect). Filesystem notification will NEVER work according to "user expectation", because user expectations are either UNREALISTIC or SERIOUSLY FLAWED.

And yes, abstracting all the leaky crap necessary for Listen to more-or-less "behave as users expect" may be possible, though currently, this isn't a "5 minute fix" - by any stretch. In fact, this change allows further bugfixes without breaking backward compatibility (though you'd have to get deep into the complexities of Listen to understand why).

Summary: If I've screwed up, please point to the code so it can undergo discussion/review. It's not professional to "vote" for reverting based on ... project-specific "inconvenience". If a feature was broken, create a PR with a test case for it, or at least submit an issue. If semver was broken, point out how exactly not just "my stuff stopped working! you brokez semverz!".

P.S. You points are unrelated to this feature request.

[rmm5t]

@e2 I have to respectfully disagree with who you think is acting unprofessional. What did I say that was so offensive? I'm genuinely sorry if I crossed a line with something I said , but I honestly don't know what I'm apologizing for yet.

please quote a line from there if you disagree

"Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API." My argument is that listen's API is not just about how it's Ruby interfaces behave; it's also about how it's filesystem interfaces behave. Listen used to support directories with symlinks in them. Now it doesn't. This is where I believe the violation of semver happened. I have no doubt there was some sporadic behavior like you mentioned. It just didn't appear to affect my projects, and I might have very well been ignorant to problems I didn't notice.

If there's a backward incompatibility, please point me to how Listen 2.8 does not work according to 2.7 documentation.

Just because something isn't documented doesn't mean it's not part of the "API." While semver encourages documenting the full public API, to quote semver again, "For this system to work, you first need to declare a public API. This may consist of documentation or be enforced by the code itself. " While perhaps flawed before, again, directories with symlinks did used to run before, and my argument is that was part of the "public API" whether intended or not. Major projects sometimes have undesirable features and even bugs that require bumping the major version to accommodate the change, because the side effect on just a minor version bump makes it difficult for the semver community to absorb otherwise. That is my main point.

Semver isn't always perfect, and there are times where's it's necessary to break the semver "rules." Maybe this is one of those times, and I respect your opinion if you think v2.8 was the correct place for this change.

Also, pull requests are welcome (ideally, with specs).

Agreed. I've even already looked into the guard codebase to help with some sort of watchdirs declaration in the Guardfile, but upon quick investigation, it looked like the load order of operations for guard somewhat complicates itself with regards to interactions between command line arguments and the Guardfile DSL execution. In the meantime, I punted due to lack of time, and I was more interested in a quick workaround, and I suggested the version as a means to help others.

I agree that none of the true fixes or new features are 5 minute jobs. Maintaining open source projects is a thankless and time-consuming effort sometimes. I know. I'm really glad you're trying to move these projects forward.

All because '-w' option is cumbersome doesn't mean 2.8 is botched.

I never said that. The -w options is a great feature, just not a great workaround to the problems that listen introduced on projects with symlinks, in my opinion. I think adding some sort of watchdirs behavior to Guardfiles is a better future feature.

as expected from someone from Thoughtbot (which I respect)

To be clear, I'm not from Thoughtbot. I'm just one of the few outsiders who have been invited with a commit bit to some of their projects.

It's not professional to "vote" for reverting based on ... project-specific "inconvenience".

How is declaring a vote for something unprofessional? This is open source, and I was trying to communicate my opinion on the topic to start the discussion and work towards a common goal. Maybe I'm misreading you, but I really don't understand why there seems to be so much anger and defensiveness about all of this. I don't have a commit bit to this project, so a comment (all comments are implicit votes, btw) to drive the discussion is the only way this stuff works. It's perfectly valid to veto my argument or vote too, btw, and I would respect that.

P.S. You points are unrelated to this feature request.

That's very fair and a very good point. A discussion about yanking a version probably should have been taken on as a separate issue. I was just responding to @bradgessler's question.

[e2]

What did I say that was so offensive?

Not offensive, just unfair, implied assessment of quality based on a subjective experience.

You're implying shoddy craftsmanship where there was more thought and work actually invested than anyone would ever believe.

It's not helpful, is not respectful towards the contributors and most importantly, it's not encouraging (Though I admit - neither is my attitude probably).

I'm genuinely sorry if I crossed a line

No need to be sorry. I just overreacted seeing a mutiny form undeservingly against 2.8.

There's still a crap load of work to make Listen "good enough" (given the arch vs bugs vs expectations) and statements like "let's yank 2.8" and neither encouraging nor helpful to anyone in anyway.

it's also about how it's filesystem interfaces behave

Unfortunately, that's completely unpredictable, unintuitive and often undefined (in terms of events reported vs actual events).

Listen doesn't implement its own filesystem + OS to make guarantees which fit user "expectations". Filesystem monitoring doesn't "simply work" anywhere. There's lots of magic behind the scenes and a ton of compromises to make "cross-platform filesystem monitoring".

At best, Listen "normalizes" events across adapters, and even that hasn't been clearly defined, forget perfected or "correct".

Listen used to support directories with symlinks in them.

There is no "symlink support" - it's a case-by-case problem that users overgeneralize incorrectly as "symlink support". It's a misnomer at best. No backend offers actually listening to symlinks except polling - everything else listens to the physical directories.

Ironically, symlink support has been an open issue until 2.8.x, because this fix actually allows closing the missing symlink support. Just checkout the README from pre 2.7 - it mentions symlink support as a PENDING FEATURE(!) So, according to the README (the first section, actually), symlinks were OFFICIALLY NOT SUPPORTED AND WERE CONSIDERED BROKEN!

In short: whatever worked was an "accident" and was contrary to what the README stated.

Now it doesn't.

What doesn't work, specifically? Because symlinked directories are followed and the real directories are watched (on Linux), on OSX I have no clue, on Windows people don't even have standard userspace tools for symlinks, and polling just follows everything multiple times.

What doesn't work now is monitoring the same physical directory twice - which makes no sense in terms of performance or resources (a BIG issue, because people are CONSTANTLY having problems with Listen being "slow").

This is where I believe the violation of semver happened.

In pre 2.8, there's no mention of symlinks working in the README, all Travis builds are green on 2.8, nothing was removed, nothing was really done on symlinks since about 2.6.x, etc.

Your perception has absolutely no technical basis. "Thinks worked, now they don't". Well, filesystem loops "worked", (they hanged listen or the backend adapter), now they don't. Once I fix encoding issues, will that also be considered breaking semver? Is crashing on non-UTF filenames considered a bug Listen has to handle and deal with? Or is it a "feature" that I shouldn't touch without a change in the major number?

Just because something isn't documented doesn't mean it's not part of the "API."

All because you imagine something being part of the API based on what "works", doesn't mean you can decide about whether to yank a release or not.

"For this system to work, you first need to declare a public API. This may consist of documentation or be enforced by the code itself.

Yard docs, README, specs ... I've even bent over backward tens of times to keep methods that no one is every using - simply because they were public and the "could" be used somewhere by someone.

While perhaps flawed before, again, directories with symlinks did used to run before, and my argument is that was part of the "public API" whether intended or not.

I get that most people don't bother to even go through READMEs - I don't either. For symlinks to work like you expect, there's work to do (see open issues). Currently, with this fix, there's potential to actually match the users' expectations - but this won't come cheap. (Some encouragement or involvement would be nice).

Personally, I don't use or even own a Mac. Even though, I spent MANY hours dealing with OSX-specific scenarios and functionality in Listen - simply because other people had problems. According to "my API", rb-inotify on Linux works perfect and nothing needs fixing - editor support in Vim is superb. And somehow, no one cared that watching multiple dirs on OSX was broken in Listen for weeks, even after being officially "fixed".

What I'm saying is - I'd appreciate constructive feedback, especially specific cases I can reproduce (ideally, without a Mac). If something doesn't work, let's deal with this like pros - find the root cause, propose a fix, assess the side-effects, etc.

Minor version bump makes it difficult for the semver community to absorb otherwise. That is my main point.

Personally, I see this change as a feature, because it helps people learn how to tweak things, so they are watching only files they are working on - which means faster responses, no 100% CPU issues, no "looping" in the background, polling eats fewer resources (e.g. by not watching .bundle or vendor or public/ or db/test.sqlite_journal, or log files, or git files, etc.).

Just like: why would you make multiple backups of the same folder on the same drive?

Semver isn't always perfect, and there are times where's it's necessary to break the semver "rules."

No rule broken here - just people incorrectly using Listen and their system resources. Sure, tools could be more convenient or "smarter", but those are features, not bugs. Maybe 2.9 will help deal with some of these - maybe they will have to wait until 3.0.

I even considered a 3.0 for this, but there was just no technical basis (other than a perception one - but the benefit of awareness is greater).

Also - if stuff works, why upgrade? And if you want to be on the bleeding edge - why complain? Semver allows you to opt-out if necessary - and that's why I made it 2.8 - just in case people strongly oppose and 2.7 will need immediate bugfixes.

Maybe this is one of those times, and I respect your opinion if you think v2.8 was the correct place for this change.

This had the consent of the whole team - the team which developed Listen over the years in the first place.

Also, pull requests are welcome (ideally, with specs).

I've even already looked into the guard codebase to help with some sort of watchdirs declaration in the Guardfile, but upon quick investigation, it looked like the load order of operations for guard somewhat complicates itself with regards to interactions between command line arguments and the Guardfile DSL execution.

Yes, it's a mess (you've hit the nail on the head) and currently refactoring is almost complete - which is why I haven't started working on making Listen smarter. I'm refactoring Guard, because I'm the only guy both capable and willing to do the cleanup - while being committed to no breaking the current API (which is becoming more and more impossible). Actually, I'm trying to gradually refactor Guard without breaking everything (and until recently, I haven't found a way to refactor in a way which didn't end up with starting from scratch)

In the meantime, I punted due to lack of time, and I was more interested in a quick workaround, and I suggested the version as a means to help others.

Everyone is too busy to dig in deep - I get it and I respect that people have other responsibilities. The quick workaround is to freeze the version - the Release Notes mention freezing, so anyone who upgrades "consciously" (probably few people), there were more than enough warnings and workarounds provided.

I agree that none of the true fixes or new features are 5 minute jobs. Maintaining open source projects is a thankless and time-consuming effort sometimes. I know. I'm really glad you're trying to move these projects forward.

Thanks - I appreciate you saying that. I also know how it sucks when development tools fail users - I really do. There are countless hours spent doing "nothing new", like debugging, refactoring, learning, testing, handling tickets/issues, rebasing, merging, diffing, reviews, releasing, documenting, etc. Especially on mature projects like Guard and Listen, were feature, patches and hacks have been stacked onto each other with no major architecture changes or refactoring throughout the years (the projects just grew that fast towards the demand - and now closing a few bugs takes ages).

I think adding some sort of watchdirs behavior to Guardfiles is a better future feature.

The idea is to have the directory selection automatic. It would be something like find -type d |grep -v "tmp\|log\|...", except it would have to be implemented in Ruby, portable and most importantly - adapted to each backend supported by Listen (with rb-fsevent being the biggest problem - and I don't even have access to a Mac to begin with).

I'm just one of the few outsiders who have been invited with a commit bit to some of their projects.

That still says a lot.

How is declaring a vote for something unprofessional?

It's an opensource software project. There's Ruby, RubyGems, Bundler, Git, RSpec, Travis, forks, patches, diff, git log, git blame - lots of tools which make free you from being locked into a particular version of software (or functionality). Voting for a version revert before research (especially when everything else is available) just doesn't make sense, because there are too many potentially false assumptions (especially technical ones).

This is open source, and I was trying to communicate my opinion on the topic to start the discussion and work towards a common goal.

Your vote carries weight and therefore responsibility - if your reasoning is flawed or incomplete, you're doing the opposite of helping when giving an opinion on "what should be done". You can do things neutrally by mentioning your questions, your issues - and see if the authors/maintainers are responding.

Sure, I could be wrong, I could have screwed up - but at least FIRST give me a chance to explain my actions or decisions before forming an opinion.

If your opinion matters, why doesn't mine? (e.g. asking me about why I released this - was it intentional or not?)

Maybe I'm misreading you, but I really don't understand why there seems to be so much anger and defensiveness about all of this.

I formed an opinion based on your opinion. It's a bit discouraging to be actively working on the issues - and then seeing people unintentionally sabotaging the whole thing. I have only 24 hours a day - so there's a physical limit to how much I can do (and how many issues I can deal with). The projects don't need "management" or "direction" - they need contributions more than anything else.

I don't have a commit bit to this project

But you can fork the project, maintain it and invite people to use your fork - with or without my "consent".

(If you'd look at the README, you'd see both projects (Guard and Listen) are looking for maintainers.)

It's perfectly valid to veto my argument or vote too, btw, and I would respect that.

Seriously - both projects need not more "votes" or "opinions", but more maintenance, more love and pull requests.

Also, behind both projects is a great team and many kind and generous contributors. I refuse to let anyone imply or even hint they did a shitty job. I someone is implying I do a shitty job, it's fair for me to expect technical proof.

Personally, I admit I have an attitude worth fixing. Then again - that same attitude has helped me endure through getting to the bottom of and fixing the most tedious issues.

[bradgessler]

I'd argue the CLI is an important API. If the project I was working on was with my team, I'd run into problems if all of a sudden people had to type in bundle exec guard -w app as opposed to just bundle exec guard. We have bins that act as conventions in our project (for example, foreman start runs the app, bundle guard for testing, etc).

I have a few ideas:

1. Update the README.md to tell people to downgrade to 2.7.x if this symlink issue is affecting them (and explain the consequences, of which I don't fully understand). See https://github.com/bradgessler/listen/tree/symlink-readme#known-symlink-issue-with-28x (I can open a PR)
2. If the ignore directive needs to be deprecated/changed over time, maybe a ignore_symlinks or follow_symlinks false directive could be added to the Guardfile without breaking the existing API (public or private)

[rmm5t]

@e2 I appreciate the time you took on this lengthy reply. I think we went off the tracks a bit though, and I actually regret even starting this discussion, but there's one thing I want to clarify.

There's still a crap load of work to make Listen "good enough" (given the arch vs bugs vs expectations) and statements like "let's yank 2.8" and neither encouraging nor helpful to anyone in anyway.

All I was suggesting by the yank (though I apparently should have been more direct about it) was my vote to yank the 2.8 line from rubygems and re-release it (immediately after) as 3.0. That's it. I never suggested shoddy work. I never meant to imply throwing away any code. I'm sure the changes you're making to listen are best long-term. I just wanted to suggest adhering closer to what I believed to be a more semver-compatible approach.

At the end of the day, keeping it at 2.8 isn't really any skin off my back.

In addition, upon further reflection, and research, rubygems and bundler might not behave as gracefully as I originally assumed with regards to a yanked version that's sitting in a Gemfile.lock. That appears to be a whole other long-debated topic. More here: rubygems/bundler#2277 I actually thought bundler already had this support, but I apparently got that wrong.

[e2]

I'd argue the CLI is an important API.

Listen is a library, not a CLI command (except for the bin/listen tool which I believe is rarely even used). So it's up to the app using it to provide options. So it's up to Guard to pass the directories. If Guard is not using Listen correctly, file an issue in Guard, not here.

And while we're at it, the -w option is an important part of the Cli options in Guard - I don't understand why functionality in Listen is critical, but the -w option is somehow "meh".

I'd run into problems if all of a sudden people had to type in bundle exec guard -w app as opposed to just bundle exec guard

Problems like? Be specific enough so I at least can come up with a test case preventing me (or anyone else) from breaking this "api" in the future.

We have bins that act as conventions in our project (for example, foreman start runs the app, bundle guard for testing, etc).

Just binstub guard, and then edit bin/guard to make it project specific. If you can't do it in a clean way, file an issue in Guard.

Are you a software developer or a software user? If you're a developer, then just be one - you don't need my permission for anything when it comes to open source.

Update the README.md to tell people to downgrade to 2.7.x

Did you check the release notes? (https://github.com/guard/listen/releases).

README is not the place for information about downgrading.

1. The release notes contain PLENTY of information and warnings.
2. The README up till 2.8 contained information about symlinks not supported - and no one cared.
3. Listen aborts with an error explaining the problem and what to do - what's unclear?

If you have suggestions for a better message, you can change it here: lib/listen/record/symlink_detector.rb and submit a PR.

https://github.com/bradgessler/listen/tree/symlink-readme#known-symlink-issue-with-28x

Since Listen follows symlinked directories (a very confusing default - and very uncommon among CLI tools), the error is about Listen watching the same PHYSICAL directories multiple times - it makes just as much sense as e.g. downloading the exact same file twice from 2 different urls, or having the same version of a gem declared twice in a Gemfile.

The behavior people want makes no sense, doesn't provide any new functionality, except being "convenient". You might as well be "conveniently" watching the whole hard drive multiple times. Sure, why not?

If the ignore directive needs to be deprecated/changed over time, maybe a ignore_symlinks or follow_symlinks false directive could be added to the Guardfile without breaking the existing API (public or private)

The ignore directive is to prevent unecessary events from reaching the client app (e.g. Guard) - and so it's "post-filtering" (except for Polling, where it really does what you expect).

What kind of symlinks do you actually have that you need? Maybe there's just a real better way of doing things than "fixing" Listen?

Without a specific example, this is a meaningless discussion (because symlinks mean too many things for too many people).

[e2]

@e2 I appreciate the time you took on this lengthy reply.

Yeah, it might pay off in the future.

I think we went off the tracks a bit though, and I actually regret even starting this discussion

I regret not knowing what I could've said to cut this short respectfully, succinctly and immediately.

but there's one thing I want to clarify.

Sure

All I was suggesting by the yank (though I apparently should have been more direct about it) was my vote to yank the 2.8 line from rubygems and re-release it (immediately after) as 3.0.

Yanking already released versions is bad. It would be better to re-release 2.7.12 as 2.8.2, otherwise those with 2.8.1 would be stuck in a very confusing place. So it doesn't even really matter if it was better to release this as 3.0 or not - the approach was wrong to begin with.

I considered 3.0, but ultimately there's no API change, just a "forced enlightenment" getting people to consider doing things in a much more effective and productive way, while resolving problems they may be having. There's technically no downside.

That's it. I never suggested shoddy work.

Not deliberately. You implied it. If you truly considered 2.8 a "great work", you wouldn't have considered yanking it, right?

I could've just removed the abort and kept the message - and you'd have a big-ass "deprecation" - and that would be 100% ok with semver, even though it would probably piss people off more (and they'd start asking how to "turn it off").

In fact I considered that, expect with filesystem loops, the backend adapter would crash with a truly cryptic stack.

I just wanted to suggest adhering closer to what I believed to be a more semver-compatible approach.

Semver isn't about keeping people mentally comfortable. It's to prevent "dependency hell":

"Dependency hell is when version lock and/or version promiscuity prevent you from easily and safely moving your project forward."

Explain to me how either of those are occuring here.

I'm still baffled about how to go about stuff like this in the future - with the time and energy spent here, I might've refactored Guard enough to be able to quickly and safely added a directories declaration support in Guardfiles.

That's why I'm counting on smart guys like you to move discussion towards "progress" instead of "opinions and management" - whether I'm there to respond or focused on "getting the damn code to finally work".

[bradgessler]

I'm referring to the guard bin CLI. The assumption is that when you run it, it only listens to the files described in the Guardfile. I'd bet that most people just run bundle exec guard and expect everything to Just Work™ just like how bundle exec rspec runs all tests and foreman run runs a project.

Are you a software developer or a software user?

A user in this case. You make a lot of valid points about reading warning messages, release logs, etc, but most people don't bother. Of course the work arounds you proposed would work, but I'm assuming the point of this ticket is to find a more permanant solution.

What kind of symlinks do you actually have that you need? Maybe there's just a real better way of doing things than "fixing" Listen?

Without a specific example, this is a meaningless discussion (because symlinks mean too many things for too many people).

I have a folder in my project that contains deployments. It looks something like:

.
├── head -> versions/47
└── versions
    ├── 46
    └── 47

I tried to ignore the folder that contained these directories but as discussed earlier ignore doesn't work that way.

[bradgessler]

I opened #276 to include error messaging about downgrading to 2.7.x for people who don't read or care about performance.

[e2]

I'd bet that most people just run bundle exec guard and expect everything to Just Work™ just like how bundle exec rspec runs all tests and foreman run runs a project.

First, I ran RSpec : 249 tests passed. Then I symlinked spec/lib/listen->spec/lib/listen2. I reran RSpec. No it reports 498 tests passed. Would you call this " Just Work™"? No warning, no message no error. But is it a good thing? Is it really more valuable than not?

A user in this case.

Well, that turns me into tech support instead of a software developer - which slows down progress.

You make a lot of valid points about reading warning messages, release logs, etc, but most people don't bother.

So what's the cure?

Of course the work arounds you proposed would work, but I'm assuming the point of this ticket is to find a more permanant solution.

First, the -w option ISN'T A WORKAROUND (emphasized, because I know that has trouble comming across). It is HOW YOU'RE SUPPOSED TO USE LISTEN.
(based on the fact that listen doesn't have a clue which files within a project people will decide to edit and change).

A "more permanent solution" is for someone to sit down and take a stab at the 4 steps to implement, outlined in the opening in this ticket.

I have a folder in my project that contains deployments. It looks something like:

├── head -> versions/47
└── versions
├── 46
└── 47

I tried to ignore the folder that contained these directories but as discussed earlier ignore doesn't work that way.

guard -w versions or guard -w head or move either versions or 47 or head to a parent folder (and symlink either way).

I mean, you obviously don't physically keep the contents of 47 and symlinked 47 in the repository, right? Do you edit both at the same time? You could just use rsync to sync the copies. Lots of possibilities of not watching versions/47/* twice.

And if something changes, what do you want listen to report? versions/47 or head or both?

[e2]

I opened #276 to include error messaging (...) for people who don't read (...)

Lol! How is that different from me creating release notes for people who don't read?

[bradgessler]

I mean, you obviously don't physically keep the contents of 47 and symlinked 47 in the repository, right?

No, I ignore this from the git repo; hence, I want to completely ignore it from the project. Structure looks something like:

$ → tree . -L 4
.
├── Guardfile
└── volumes
    └── sites
        └── dev
            ├── head -> versions/47
            └── versions

In my git repo I completely ignore volumes.

Lol! How is that different from me creating release notes for people who don't read?

Because people install listen when they run bundle update, then they run bundle exec guard and it blows up under certain circumstances in 2.8. Nowhere in that process will a user see the release notes.

I'm bowing out of this ticket because downgrading to 2.7.x solved my problems and I don't see how continuing this thread is productive. You'll notice in my previous responses I ignored your soft insults to maintain focus on the issue, but you continued to throw them at me. Please don't assume your users are "blaming" you for "bad software". They're not. Everybody in this ticket is grateful for the work you've done and its high level of quality. We're all just trying to help.

[e2]

In my git repo I completely ignore volumes.

Then what directory do you want to really watch?

I'm asking because:

1. if this issue is implemented (getting directories to be "really" ignored), your case will be solved [in that case you'd have to wait until this issue is closed]
2. until this issue is simplemented, you can specify all the top directories you want to watch [in that case, you don't have to bother with this issue]

I'm guessing there's nothing in volumes you want to watch, right? So are you interested in getting this issue implemented or are you ok with the -w workaround, or do you want a feature in Guard implemented? Or are you just ok with freezing the version?

Nowhere in that process will a user see the release notes.

What I said was supposed to be humorous (because it is!) and with a point - the best solution for people who don't read is smart code (which needs to be written). If people don't read, they won't read - and if they search for answers, they'll likely find someone who will read.

Because people install listen when they run bundle update, then they run bundle exec guard and it blows up under certain circumstances in 2.8

Yes, and if users who do read and who search for answers still don't know what to do - that's a problem.

I'm bowing out of this ticket because downgrading to 2.7.x solved my problems

That's ok. If you can't restructure your project or pick which directories to watch, that's one of the solutions that's left.

I don't see how continuing this thread is productive.

That depends - I do want people to understand what this issue means. If the message could explain better (and with less works), that would be helpful, and I'd gladly release another version. The current message was on I updated based on feedback:

e4cc8a9

and someone said it's much clearer. Obviously, I do want to make this clearer if I can, so any feedback in making this understandable (as opposed to a "freezing quickfix") would really help.

You'll notice in my previous responses I ignored your soft insults

I didn't mean to insult - I tried to be to the point, trusting you to tell me how the message can be presented better so others could understand better with less effort. If I did insult you, I apologize deeply.

Please don't assume your users are "blaming" you for "bad software".

I'm not - I'm just frustrated you're not clear about what you want me to do:

1. I've added comments to your PR (ideally, I'd want people to understand the issue and only use the quickfix if they absolutely have to - because newer versions of listen will rely on this functionality).
2. this current issue - under which we're discussing - is about the work needed to make :ignore ignore folders. This is tricky work, and needs lots of effort, so let me know if you prefer that I commit to this issue or something else (I can't work on everything at once).
3. this issue can be fixed in Guard - if you prefer, I can focus on side-stepping this issue from there (not entirely, given your example, but it will help you avoid explicitly listing directories to watch). Let me know if you think I should work in this area instead.
4. based on your example, the -w option would work fine. If you have objections to using this - let me know why (because probably others have objectsion as well - ones I don't know about)

This isn't about "fix or revert" - there are a few ways of getting this done, and picking a way depends on your feedback.

In your specific case, you want to "ignore a whole folder", which either needs this issue fixed, or, it can also be achieved by doing the opposite - watching only what you need (which I could start implementing in Guard in one of 2 ways).

We're all just trying to help.

Yes, and I appreciate that very much, because I'm getting a sense of priorities, and sense of different perspectives people have, how they feel about this issue, how urgent it is, how satisfactory the fix is, do they find the documentation sufficient or too overwhelming, etc.

As a user, your feedback is expecially important - if you can help me reduce the confusion and shorten the discussion, I'll be able to fix this much sooner. That's why it's so important for me to know if you're using Guard, and how, and what can I focus on to help others with similar cases to yours. I need to know which solutions are acceptable in your case and why - and what information did you miss that would help you get a perfect solution (one which doesn't need the quickfix).

Again, I apologize if I have offended you. I understand your example now, and now that I do, I'd deeply appreciate any feedback on how you'd like me to go about this.

[e2]

@bradgessler, @rmm5t - I apologize for my attitude.

Listen has been a disastrous time sink because of the complexity of the subject.

Every stab I've taken at improving things (and even understanding the implications of changes) has backfired. Every improvement in concept or architecture has turned out to be a dead end (mostly because of how different the backends are).

Comparatively, people's unrealistic expectations about what Listen "should do" (sometimes formed as demands) make my experiences very unfulfilling.

Whatever breakthroughs I've made will likely never be appreciated, because they are "invisible" to users by the very nature of how Listen works.

Please understand any changes I make are still proof of my effort to move things forward - and not to sabotage your projects or ruin your day (even if that's what actually happens).

[e2]

Closed in favor of: #279 which should resolve this and other issues.

[e2]

@skizzybiz, @bradgessler, @rmm5t - Guard 2.9.0 now supports the directories statement: https://github.com/guard/guard/releases while Listen 2.8.2 skips on duplicate directories.

Let me know if there's anything else I can do mitigate the problems you are having (please open new issues).

[bradgessler]

@e2 you honestly didn't have to do that, but I thank you many times over for everything you've done.

[akerbos]

FWIW:

Even removing a useless method deprecated since 2012 caused annoyances.

I strongly feel that maintaining backwards-compatibility at any cost is wrong. See the Java API for a painful example. On the flip side, eventually removing deprecated features/API is the expected thing to happen, assuming the deprecation had been clearly communicated (release notes, maybe even warning messages during execution). So, please, don't let old baggage hold you back.

[e2]

I strongly feel that maintaining backwards-compatibility at any cost is wrong. So, please, don't let old baggage hold you back.

I agree - it's just that while keeping backward-compatibility turns even bugfixing into "advanced code-surgery", it's never appreciated. Meanwhile, even fixing a broken behavior (to fix a bug) quickly turns into "how careless are you to break everything?".

The "right" thing to do would be to release a new major version with every bugfix (but then people don't get any bugfixes, if they lock to major versions - which is like releasing an entirely new gem under a new name just to fix a bug).

So ironically, backward compatibility is a "people problem", not a technical one - it requires active support (in any form) from the people who need it most.

So check the actual overhead costs (depends on use case) and decide for yourself . You may want to look at / compare with https://github.com/smerrill/vagrant-gatling-rsync.