Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add parse_glob module and update du to use parse_glob #3754

Merged
merged 3 commits into from
Aug 10, 2022

Conversation

ackerleytng
Copy link
Contributor

Fixes #3628

The parse_glob module uses the same logic implemented in rust-lang/glob: https://github.com/rust-lang/glob/blob/master/src/lib.rs#L613 to identify negation in globs. When this pattern is identified, the ^ in the pattern is replaced with ! and glob's Pattern is built from the string.

A disadvantage of this approach is that inconsistencies may result, if this pattern identification method diverges, when rust-lang/glob's parsing of character set negation changes.

However, I think this approach avoids having to maintain or rebuild glob parsing logic, and is the best option I could think of given that rust-lang/glob's implementation is not open to extension to support multiple character set negation markers.

With the parse_glob module, fixing du is just a matter of using the new module.

GNU ls and dircolors also use fnmatch. Those two utilities will be updated to use parse_glob in separate PRs.

@ackerleytng
Copy link
Contributor Author

Any tips on how to debug the new test failure in tests/tail-2/F-headers?

I did

./uutils/util/run-gnu-test.sh tests/tail-2/F-headers

on my machine (linux) and it was fine.

@tertsdiepraam
Copy link
Member

tertsdiepraam commented Jul 31, 2022

@ackerleytng That's very curious. Maybe you could download the logs from the CI and look at those? If it's a CI only issue we might have to ignore it or fix the CI somehow.

Edit: Oh I think that test just has some intermittent failures. I'll rerun it.

@ackerleytng
Copy link
Contributor Author

I diffed the this test run with an earlier working one and nothing stood out, other than

FAIL: tests/tail-2/F-headers
============================

tail: cannot open 'b' for reading: No such file or directory
./tests/tail-2/F-headers.sh: a: unexpected delay?
tail: cannot open 'a' for reading: No such file or directory
tail: cannot open 'b' for reading: No such file or directory
==> b <==
FAIL tests/tail-2/F-headers.sh (exit status: 1)

This test

  1. Deletes files with names a, b
  2. Starts a tail -F on both those files
  3. Writes to a and b and then checks that within 12.7s a header appears for the files.

Do we know if these tests are sufficiently independent of each other?

@ackerleytng
Copy link
Contributor Author

I just noticed this: https://github.com/uutils/coreutils/blob/main/util/build-gnu.sh#L171

Since this test uses more than one process and is timing dependent, is it something to do with the hardware (performance) of the CI machine?

@tertsdiepraam
Copy link
Member

Yeah that looks like something we should fix outside this PR

@ackerleytng
Copy link
Contributor Author

Feel free to merge this if you're okay with fixing this separately!

@sylvestre
Copy link
Contributor

Yeah!

Warning: Congrats! The gnu test tests/du/slash is no longer failing!

let mut inodes: HashSet<FileInfo> = HashSet::new();
if let Some(inode) = stat.inode {
inodes.insert(inode);
if let Ok(stat) = Stat::new(path, &options) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a comment explaining what this block is doing? It isn't super obvious reading the code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see newest changes!

@@ -745,6 +745,27 @@ fn test_du_exclude_mix() {
assert!(!result.stdout_str().contains("qzerty"));
assert!(!result.stdout_str().contains("azeaze"));
assert!(result.stdout_str().contains("xcwww"));

// Negation in glob should work with both ^ and !
let result = ts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe create a new test for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

@ackerleytng ackerleytng force-pushed the main branch 3 times, most recently from de02ebb to 6aa9aa1 Compare August 8, 2022 14:40
@sylvestre sylvestre merged commit 8692301 into uutils:main Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

du --exclude='[^/]*' / behave differently
3 participants