Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: add bufferSize option to fs.opendir() #30114

Closed
wants to merge 7 commits into from

Conversation

addaleax
Copy link
Member

Add an option that controls the size of the internal
buffer.

Fixes: #29941

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

Add an option that controls the size of the internal
buffer.

Fixes: nodejs#29941
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Oct 25, 2019
@duffleit
Copy link

What a mob programming session 🙌🏻😀

@jdjuan
Copy link

jdjuan commented Oct 25, 2019

300 people watched the process of this PR during @scriptconf Great job!!!!!

doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
@drdreo
Copy link

drdreo commented Oct 25, 2019

Can approve that all tests were done

@nodejs-github-bot
Copy link
Collaborator

@Trott
Copy link
Member

Trott commented Oct 25, 2019

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

doc/api/fs.md Outdated Show resolved Hide resolved
@addaleax
Copy link
Member Author

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

@addaleax addaleax added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 25, 2019
@richardlau
Copy link
Member

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

@Trott
Copy link
Member

Trott commented Oct 25, 2019

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

And maybe one test for 32 and one other reasonable positive integer to confirm that it doesn't throw in those cases?

@addaleax
Copy link
Member Author

@Trott @richardlau I guess, yes – added tests for the error condition/that it works when using a positive integer value.

test/parallel/test-fs-opendir.js Outdated Show resolved Hide resolved
@addaleax addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 26, 2019
@nodejs-github-bot
Copy link
Collaborator

lib/internal/fs/dir.js Outdated Show resolved Hide resolved
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@addaleax
Copy link
Member Author

Landed in b35181f 🎉

@addaleax addaleax closed this Oct 28, 2019
addaleax added a commit that referenced this pull request Oct 28, 2019
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@addaleax addaleax deleted the fs-opendir-buffersize branch October 28, 2019 13:35
targos pushed a commit that referenced this pull request Nov 5, 2019
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
targos added a commit that referenced this pull request Nov 5, 2019
Notable changes:

* cli:
  * Added a new flag (`--trace-uncaught`) that makes Node.js print the
    stack trace at the time of throwing uncaught exceptions, rather than
    at the creation of the `Error` object, if there is any. This is
    disabled by default because it affects GC behavior.
    #30025
* crypto
  * Added `Hash.prototype.copy()` method. It returns a new `Hash` object
    with its internal state cloned from the original one.
    #29910
* dgram
  * Added source-specific multicast support. This adds methods to
    Datagram sockets to support RFC 4607
    (https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
    #15735
* fs
  * Added a `bufferSize` option to `fs.opendir()`. It allows to control
    the number of entries that are buffered internally when reading from
    the directory. #30114
* meta
  * Added Chengzhong Wu (https://github.com/legendecas) to
    collaborators. #30115

PR-URL: #30262
@targos targos mentioned this pull request Nov 5, 2019
targos added a commit that referenced this pull request Nov 5, 2019
Notable changes:

* cli:
  * Added a new flag (`--trace-uncaught`) that makes Node.js print the
    stack trace at the time of throwing uncaught exceptions, rather than
    at the creation of the `Error` object, if there is any. This is
    disabled by default because it affects GC behavior.
    #30025
* crypto
  * Added `Hash.prototype.copy()` method. It returns a new `Hash` object
    with its internal state cloned from the original one.
    #29910
* dgram
  * Added source-specific multicast support. This adds methods to
    Datagram sockets to support RFC 4607
    (https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
    #15735
* fs
  * Added a `bufferSize` option to `fs.opendir()`. It allows to control
    the number of entries that are buffered internally when reading from
    the directory. #30114
* meta
  * Added Chengzhong Wu (https://github.com/legendecas) to
    collaborators. #30115

PR-URL: #30262
MylesBorins pushed a commit that referenced this pull request Jan 12, 2020
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
BethGriggs pushed a commit that referenced this pull request Feb 6, 2020
Add an option that controls the size of the internal
buffer.

Fixes: #29941

PR-URL: #30114
Reviewed-By: Richard Lau <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
@MylesBorins MylesBorins mentioned this pull request Feb 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fs.Dir.read() is very slow
10 participants