refactor remote_directory provider #3837
Conversation
|
@Igorshp can you give this a test on your use case? |
|
There's no spec changes here because AFAIK this a pure refactor and is functionally identical, but has perf CPU + Mem fixes, so unsure right now of what specs to add. There's a FIXME I added with a reference to an old CHEF ticket that turned up when I was researching stuff for documentation on why we search the way we search. Fixing that was out of scope with the timebox that I want around this PR since this is a squirrel project that is distracting me from other work right now. |
|
@chef/client-core needs review |
|
This is the before and after of memory pressure (live objects after a forced full GC) on a CCR on one of my servers with the big red spike being a single remote_directory provider before this refactor was done. |
|
Also removes a 4 second delay on my cookbooks as it no longer walks my home dir at all, because I do not have purge set: �https://gist.github.com/lamont-granquist/def55172542a8c0b3200 It should also not cause the memory explosion with purge set and should be slightly faster and use less memory than @Igorshp's fix since we stop mangling large arrays in memory and just walk the filesystem dirents once and during the process skip skippable files. |
| class Chef | ||
| module Deprecation | ||
| module Provider | ||
| module RemoteDirectory |
There was a problem hiding this comment.
This is for folks who've subclassed RemoteDirectory ?
|
A few nitpicky things that I don't care deeply about, 👍 though. |
|
LGTM 👍 |
|
Okay, converted to Set and switched to @Igorshp any chance you could test this code against your case? |
|
@lamont-granquist sure thing, i'll report back with results shorlty |
|
Before: That's some impressive performance boost :) 👍 |
|
|
|
👏 🎉 |
- Huge speed and memory perf boost. In the prior code if you were copying 10 dotfiles to a home directory with a million files in it, would slurp all million files into a ruby Set object even if you were not ultimately going to purge the unmanaged files. This code inverts the logic and tracks managed files and then iterates through the filesystem without slurping a list into memory. - Only do file purging logic if purging is set. - Fixes mutation of new_resource.overwrite. - Fixes mutation of new_resource.rights (subtle). - Adds helper delegators to the new_resource properties. - Deprecates (instead of removes) now-unused methods. - Renamed a method (with deprecated alias preserved) for consistency. - Adds YARD for everything. - Changes protected to private because protected is largely useless in ruby. - Removes whyrun_supported? because the superclass sets that.
- convert to Set - use #override?
lamont-granquist
force-pushed
the
lcg/remote_directory_refactor
branch
from
2c1e8ac to
6109fb0
Compare
refactor remote_directory provider
|
👏 |
copying 10 dotfiles to a home directory with a million files in it,
would slurp all million directory entries into a ruby Set object even if you were
not ultimately going to purge the unmanaged files. This code inverts
the logic and tracks managed files and then iterates through the
filesystem without slurping a list into memory.
in ruby.