Windows have EMFILE: too many open files problem

[ausir0726]

Describe the bug
When I use 11ty develope my site it's work very well.
but when I put all my archive articles and image to 11ty and build it
it's have EMFILE: too many open files error
it's total about 7000 iamges and 4000 pages (each page have 2 layer layout with pug )
( image copy done, pug complier done, but write html fail )
I tried disable addPassthroughCopy my images, and it's work fine.
seems like addPassthroughCopy and write html will reach the open files limit at same time.

It's only happen on windows ( win10 pro installed WSL 2 docker)

To Reproduce
eleventy --input=build

Expected behavior
build success, but fail.

Screenshots

Environment:

• win10 pro installed WSL 2 docker
• tried any version of 11ty, 0.12 , 1.0.2 2.0.0-canary.16

Additional context
it's work fine on linux and mac.

[ausir0726]

I'm using a little trick for fix this at moment
it's work fine

the build folder include all my archive articles about 4000 pages and 7000 images.

const yargs = require('yargs/yargs');
const copy = require('recursive-copy');

module.exports = (eleventyConfig) => {
  eleventyConfig.setUseGitIgnore(false);
  const argv = yargs(process.argv).argv;

  if (argv.input == 'src') {
    eleventyConfig.addPassthroughCopy(`${argv.input}/images`);
    eleventyConfig.addPassthroughCopy(`${argv.input}/spotlight/**/*.jpg`);
    eleventyConfig.addPassthroughCopy(`${argv.input}/english/spotlight/**/*.jpg`);
  }
  if (argv.input == 'build') {
    copy(`${argv.input}/images`, '_site/images', { overwrite: true });
    copy(`${argv.input}/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
    copy(`${argv.input}/english/spotlight`, '_site/english/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
  }
}

[pdehaan]

Related: #2139
Wonder if copying the 7k images outside of 11ty (using rsync or cp or something) would handle EMFILE errors better.

[ausir0726]

I tried use rsync or cp it's work very well
but I don't want write my copy folders list in different place.

seem like when I use addPassthroughCopy, it will reach the limit about 7000 - 8000 images.

[pdehaan]

@ausir0726 And you get this error consistently?
I don't have a Win 10 Pro w/ WSL 2 machine/VM available (or a demo site that big), but maybe you can try a few things and see if they help.

https://www.11ty.dev/docs/copy/ says that Eleventy uses the recursive-copy package (although the recursive-copy advanced options syntax might only be supported in 2.x canary builds).

It looks like the current passthrough copy defaults are defined in:

eleventy/src/TemplatePassthrough.js

Lines 194 to 207 in 631c881

	// default options for recursive-copy
	// see https://www.npmjs.com/package/recursive-copy#arguments
	let copyOptionsDefault = {
	overwrite: true, // overwrite output. fails when input is directory (mkdir) and output is file
	dot: true, // copy dotfiles
	junk: false, // copy cache files like Thumbs.db
	results: false,
	expand: false, // follow symlinks (matches recursive-copy default)
	debug: false, // (matches recursive-copy default)
	// Note: `filter` callback function only passes in a relative path, which is unreliable
	// See https://github.com/timkendrick/recursive-copy/blob/4c9a8b8a4bf573285e9c4a649a30a2b59ccf441c/lib/copy.js#L59
	// e.g. `{ filePaths: [ './img/coolkid.jpg' ], relativePaths: [ '' ] }`
	};

Scrolling through https://github.com/timkendrick/recursive-copy#usage options, there is a potentially interesting option called options.concurrency. Although it seems like it has a default value of 255. You could see if dropping that lower helps at all.
Maybe something like:

eleventyConfig.addPassthroughCopy(`${argv.input}/images`, { concurrency: 100 });

[ausir0726]

@pdehaan
thanks for your help
seems like concurrency is work fine.
I tried to incrase to concurrency: 200 it work fine also.
but when I incrase to max 255, 11ty build fail.

seems like concurrency: 200 is the safty way out and preformance on windows.

concurrency: 100

[11ty] Copied 7883 files / Wrote 2167 files in 343.59 seconds (158.6ms each, v2.0.0-canary.16)

concurrency: 200

[11ty] Copied 7883 files / Wrote 2167 files in 333.78 seconds (154.0ms each, v2.0.0-canary.16)

concurrency: 255

[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble rendering pug template ./build/scripts/vender.js.pug (via TemplateContentRenderError)
[11ty] 2. Having trouble compiling template ./build/scripts/vender.js.pug (via TemplateContentCompileError)
[11ty] 3. EMFILE: too many open files, open 'node_modules\alpinejs\dist\cdn.js'
[11ty]     at ./build/scripts/vender.js.pug line 2 (via Error)
[11ty]
[11ty] Original error stack trace: Error: EMFILE: too many open files, open 'node_modules\alpinejs\dist\cdn.js'
[11ty]     at ./build/scripts/vender.js.pug line 2
[11ty]     at Object.openSync (node:fs:594:3)
[11ty]     at Object.readFileSync (node:fs:462:35)
[11ty]     at Function.read (C:\Users\ausir\proj\ntu\node_modules\pug-load\index.js:85:13)
[11ty]     at Object.read (C:\Users\ausir\proj\ntu\node_modules\pug\lib\index.js:164:25)
[11ty]     at C:\Users\ausir\proj\ntu\node_modules\pug-load\index.js:28:25
[11ty]     at walkAST (C:\Users\ausir\proj\ntu\node_modules\pug-walk\index.js:26:18)
[11ty]     at C:\Users\ausir\proj\ntu\node_modules\pug-walk\index.js:112:20
[11ty]     at Array.reduce (<anonymous>)
[11ty]     at walkAndMergeNodes (C:\Users\ausir\proj\ntu\node_modules\pug-walk\index.js:111:18)
[11ty]     at walkAST (C:\Users\ausir\proj\ntu\node_modules\pug-walk\index.js:40:19)
[11ty] Wrote 0 files in 10.36 seconds (v2.0.0-canary.16)
error Command failed with exit code 1.

by the way, when I use following code

if (argv.input == 'build') {
    console.time('a');
    console.time('b');
    console.time('c');
    copy(`${argv.input}/images`, '_site/images', { overwrite: true }).then(() => { console.timeEnd('a'); });
    copy(`${argv.input}/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] }).then(() => { console.timeEnd('b'); });
    copy(`${argv.input}/english/spotlight`, '_site/english/spotlight', { overwrite: true, filter: ['**/*.jpg'] }).then(() => { console.timeEnd('c'); });
  }

I got the result

a: 592.59ms
c: 2.621s
[11ty] Wrote 2167 files in 192.89 seconds (89.0ms each, v2.0.0-canary.16)
b: 3:14.173 (m:ss.mmm)

[ausir0726]

I have a profermance question about recusive-copy
why it copy 7800 images need more than 3 mins?
that build folder is merge my archive folder and src folder
I use merge-dirs
my npm scripts is merge-dirs src build && merge-dirs archive build

echo %time% && merge-dirs src build && merge-dirs archive build && echo %time%

result

14:06:10.27
14:06:10.27

it's same copy 7800 images to build folder, build only use 1s.

is recusive-copy have perfomance issue about bluk files copy ?

[ausir0726]

Unfortunately
I still got build fail on concurrency: 200
I decrease concurrency to 150 still fail.
when I decrease concurrency 100 than build success.

[pdehaan]

a: 592.59ms # `${argv.input}/images/**`
b: 3:14.173 (m:ss.mmm) # `${argv.input}/spotlight/**/*.jpg`
c: 2.621s # `${argv.input}/english/spotlight/**/*.jpg`
[11ty] Wrote 2167 files in 192.89 seconds (89.0ms each, v2.0.0-canary.16)

Where 193 second build time is 3.2 minutes or 3m:13s.
Without knowing your file structure or files, it appears that the majority of the time is spent copying the /spotlight/**/*.jpg glob.

But still, 7800 image is a lot of images, and file i/o operations. It might be worth filing an issue in the recursive-copy repo with a non-Eleventy test case and see if they have any ideas or combination of configs that might work better than the defaults.
My guess as to what's happening is that Eleventy+recursive-copy are copying 7800 images each time. Whereas rsync and merge-dirs (which I've never used) might be smart enough to say "a destination file already exists, no need to overwrite it" and it's 7800 no-ops. You might need to play around with it and clear the destination folder and try again and see if it's always 1s for merge-dirs if you do a full rebuild.

From the recursive-copy docs:

Name	Type	Required	Default	Description
options.overwrite	boolean	No	false	Whether to overwrite destination files

Interesting that they have a default overwrite: false value, but Eleventy overrides that to true. You could see if explicitly setting that back to false works for you and possibly ignores copying existing files. Although I still imagine you'd get EMFILE errors on a first build were it has to copy 7800 files the first time. But would be interesting to experiment w/ setting { overwrite: false, concurrency: 100 } on the 3 passthrough copies and see if that makes a difference on cold/warm builds.

https://www.11ty.dev/docs/debug-performance/#show-all-performance-measurements has a good section on benchmarking and if you run something like set DEBUG=Eleventy:Benchmark* & npx @11ty/eleventy it should show you aggregate benchmark data for passthrough copy.

I guess my next questions would be, if you don't do any image/asset copying and just build your pages, how long is the build time for the site? Does it still take 190s to write the 2,167 files?

[pdehaan]

Looking at merge-dirs, it looks like it has 3 options for file conflict resolution:

• 'overwrite'
• 'skip' (default?)
• 'ask' (seems suboptimal for our use case here)

As close as I can figure, recursive-copy only does 'overwrite' and seems to throw an error if you have {overwrite:false} and the target file already exists. I filed timkendrick/recursive-copy#34 which is for a 'skip'-like option when copying files (versus throwing an error).

I'd need to look a bit closer at your merge-dirs example where it was taking 1s to copy 7,800 images (with "skip"), but 3m 14s to copy the same files w/ recursive-copy (with "overwrite"). But it's hard to reproduce locally for me (on macOS) without your large images. I could try sending myself an unedited photo from my phone and renaming it 7800× and seeing how terrible that is (something tells me it'd be hard drive crushing). How big are these 7800 images? 1MB? 10KB?

[pdehaan]

OK, a bit of weekend comedy:

# Duplicate "./assets/screenshot1.png" about 8000x, for about 5.6 GB of wasted hard drive space.
for i in {2..8000}; do cp screenshot1.png "screeshot$i.png"; done

ls -lh assets/ | tail -10
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot990.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot991.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot992.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot993.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot994.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot995.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot996.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot997.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot998.png
-rw-r--r--@ 1 pdehaan  wheel   738K 15 Oct 08:45 screeshot999.png

du -sh assets # 5.6G	assets

If I manually call recursive-copy, it copies my 8003 files in ~5.2s.

[11ty] Writing www/index.html from ./src/index.liquid
[11ty] Writing www/pages/five/index.html from ./src/pages/five.liquid
[11ty] Writing www/pages/one/index.html from ./src/pages/one.liquid
[11ty] Writing www/pages/four/index.html from ./src/pages/four.liquid
[11ty] Writing www/pages/three/index.html from ./src/pages/three.liquid
[11ty] Writing www/pages/two/index.html from ./src/pages/two.liquid
[11ty] Wrote 6 files in 0.12 seconds (v2.0.0-canary.16)
asset-copy: 5.173s
Copied 8003 files

If I use passthrough copy, it copies 8002 files (and builds the 6 trivial pages) in 5.3s:

[11ty] Writing www/index.html from ./src/index.liquid
[11ty] Writing www/pages/one/index.html from ./src/pages/one.liquid
[11ty] Writing www/pages/two/index.html from ./src/pages/two.liquid
[11ty] Writing www/pages/three/index.html from ./src/pages/three.liquid
[11ty] Writing www/pages/four/index.html from ./src/pages/four.liquid
[11ty] Writing www/pages/five/index.html from ./src/pages/five.liquid
[11ty] Copied 8002 files / Wrote 6 files in 5.34 seconds (v2.0.0-canary.16)

---

const copy = require('recursive-copy');

/**
 * @typedef {import('@11ty/eleventy/src/UserConfig')} EleventyConfig
 * @typedef {ReturnType<import('@11ty/eleventy/src/defaultConfig')>} EleventyReturnValue
 * @type {(eleventyConfig: EleventyConfig) => EleventyReturnValue}
 */
module.exports = function (eleventyConfig) {
  eleventyConfig.addPassthroughCopy("assets")//, {overwrite: false, concurrency: 50});

  // console.time("asset-copy");
  // copy('assets', 'www/assets', {overwrite: true}, function(err, results) {
  //   console.timeEnd("asset-copy");
  //   if (err) {
  //     console.error(`[ERROR] Copy failed: ${err.message}`);
  //   } else {
  //     console.info(`Copied ${results.length} files`);
  //   }
  // });

  return {
    dir: {
      input: "src",
      output: "www",
    }
  };
};

Not sure how deeply nested your file structure is (mine is all just flat assets/*.png at the same depth).

merge-dirs seems to be the fastest (presumably because it skips conflicts by default):

  const mergedirs = require('merge-dirs').default;
  mergedirs('assets', 'www/assets')//, 'skip');

rm -rf www
npm run build

> [email protected] build
> time eleventy

[11ty] Writing www/pages/five/index.html from ./src/pages/five.liquid
[11ty] Writing www/pages/four/index.html from ./src/pages/four.liquid
[11ty] Writing www/index.html from ./src/index.liquid
[11ty] Writing www/pages/three/index.html from ./src/pages/three.liquid
[11ty] Writing www/pages/two/index.html from ./src/pages/two.liquid
[11ty] Writing www/pages/one/index.html from ./src/pages/one.liquid
[11ty] Wrote 6 files in 0.02 seconds (v2.0.0-canary.16)

real	0m5.771s
user	0m0.430s
sys  	0m2.542s

Then subsequent npm run build calls are much faster:

...
www/assets/screeshot997.png exists, skipping...
www/assets/screeshot998.png exists, skipping...
www/assets/screeshot999.png exists, skipping...
[11ty] Writing www/pages/five/index.html from ./src/pages/five.liquid
[11ty] Writing www/pages/one/index.html from ./src/pages/one.liquid
[11ty] Writing www/pages/three/index.html from ./src/pages/three.liquid
[11ty] Writing www/pages/two/index.html from ./src/pages/two.liquid
[11ty] Writing www/pages/four/index.html from ./src/pages/four.liquid
[11ty] Writing www/index.html from ./src/index.liquid
[11ty] Wrote 6 files in 0.02 seconds (v2.0.0-canary.16)

real	0m0.255s
user	0m0.245s
sys  	0m0.063s

So 5.77s on a fresh build without a ./www/ output folder, but only 0.25s on subsequent builds where it floods my terminal w/ "${file} exists, skipping..." output.

[ausir0726]

Dear @pdehaan

my archive folder structure
spotilight

• 2022 ( each folder include about 600 jpg images and 100 pug files )
• 2021
  ...
• 2014
  total 9962 files with 20 folders 916MB
  each image maybe about 100 - 300 kb, it's very normal images.

even use my 2015 MBP with SSD , build site need 336.91 seconds also (copied 7883 files , wrote 2168 files)

and I got build fail when concurrency: 100 with benchmark* command
but build success when concurrency: 100 without benchmark

I decrase concurrency to 50 , build success
the log as following

 set DEBUG=Eleventy:Benchmark* & npx eleventy --input=build
  Eleventy:Benchmark Benchmark    852ms   0%     4× (Aggregate) Searching the file system +0ms
  Eleventy:Benchmark Benchmark   4103ms   1%  6502× (Aggregate) Template Read +2ms
  Eleventy:Benchmark Benchmark 419783ms  99%  7883× (Aggregate) Passthrough Copy File +1ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) getOutputHref +1ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) getOutputPath +0ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) Render Permalink +1ms
  Eleventy:Benchmark Benchmark 364394ms  86% 10818× (Aggregate) Template Compile +1ms
  Eleventy:Benchmark Benchmark     51ms   0%     3× (Aggregate) > Compile > ./build/scripts/vender.js.pug +1ms
  Eleventy:Benchmark Benchmark    521ms   0%  6484× (Aggregate) Render +0ms
  ...
   Eleventy:Benchmark Benchmark 407853ms  96%  2168× (Aggregate) Template Write +0ms
  Eleventy:Benchmark Benchmark  81208ms  19%  2157× (Aggregate) > Compile > ./build/_layouts/spotlight/article.pug +1ms
  Eleventy:Benchmark Benchmark    877ms   0%     2× (Aggregate) > Compile > ./build/_layouts/about/about.pug +1ms
  Eleventy:Benchmark Benchmark    722ms   0%     1× (Aggregate) > Compile > ./build/_layouts/administration/administration.pug +1ms
  Eleventy:Benchmark Benchmark    716ms   0%     2× (Aggregate) > Compile > ./build/_layouts/index.pug +0ms
  Eleventy:Benchmark Benchmark    227ms   0%     5× (Aggregate) > Compile > ./build/_layouts/spotlight/spotlight.pug +1ms
  Eleventy:Benchmark Benchmark 275228ms  65%  2167× (Aggregate) > Compile > ./build/_layouts/default.pug +1ms
[11ty] Copied 7883 files / Wrote 2168 files in 425.59 seconds (196.3ms each, v2.0.0-canary.16)

[pdehaan]

What's your build time if you do NO passthrough copy (or manual asset copying using merge-dirs or recursive-copy)?

It seems odd that a build takes 425s and passthrough copy is taking roughly 420s; plus:

• 36s for "Template Compile"
• 408s for "Template Write"
• 81s for "./build/_layouts/spotlight/article.pug" (which I guess makes sense since I'm guessing you're using pagination which is why that gets called 2157× (given your site only has about 2168 files, so 11 files NOT using that layout)
• 275s for "default.pug" layout (which is used for every template, except one)

If removing Eleventy passthrough copy solves 99% of the performance issues, I'd say switch to another option (merge-dirs or recursive-copy, if those work). If removing passthrough copy STILL causes a build time of 400s, I'd think that the performance bottleneck is somewhere else.

But now I'm curious how quickly my relatively new [corporate] 2021 Apple M1 Max laptop could compile the site.
Although your 2015 MBP builds in 337s; versus the 425s from [presumably] Win 10 Pro installed WSL 2 Docker. Unless before you were saying your build times were roughly 343s, so maybe the +80s build times are strictly due to the lowered concurrency (from default 255 to 50).

[ausir0726]

If I remove passthrough copy
It takes 197s for build process.

each article have 2 layers layout
article-01.pug (layout article-layout.pug )
article-layout.pug (layout default.pug)

seems like copy 7000 images need another 200s.

$ set DEBUG=Eleventy:Benchmark* & npx eleventy --input=build
  Eleventy:Benchmark Benchmark     65ms   0%     2× (Aggregate) Searching the file system +0ms
  Eleventy:Benchmark Benchmark   1365ms   1%  6502× (Aggregate) Template Read +3ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) getOutputHref +1ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) getOutputPath +0ms
  Eleventy:Benchmark Benchmark      0ms   0%  4336× (Aggregate) (count) Render Permalink +1ms
  Eleventy:Benchmark Benchmark 180363ms  93% 10818× (Aggregate) Template Compile +1ms
  Eleventy:Benchmark Benchmark    153ms   0%     3× (Aggregate) > Compile > ./build/scripts/vender.js.pug +0ms
  Eleventy:Benchmark Benchmark    498ms   0%  6484× (Aggregate) Render +1ms
...
...
  Eleventy:Benchmark Benchmark 185281ms  95%  2168× (Aggregate) Template Write +1ms
  Eleventy:Benchmark Benchmark  39725ms  20%  2157× (Aggregate) > Compile > ./build/_layouts/spotlight/article.pug +0ms
  Eleventy:Benchmark Benchmark    438ms   0%     5× (Aggregate) > Compile > ./build/_layouts/spotlight/spotlight.pug +2ms
  Eleventy:Benchmark Benchmark    419ms   0%     2× (Aggregate) > Compile > ./build/_layouts/index.pug +1ms
  Eleventy:Benchmark Benchmark    150ms   0%     1× (Aggregate) > Compile > ./build/_layouts/administration/administration.pug +0ms
  Eleventy:Benchmark Benchmark    147ms   0%     2× (Aggregate) > Compile > ./build/_layouts/about/about.pug +1ms
  Eleventy:Benchmark Benchmark 135430ms  69%  2167× (Aggregate) > Compile > ./build/_layouts/default.pug +1ms
[11ty] Wrote 2168 files in 197.72 seconds (91.2ms each, v2.0.0-canary.16)

[pdehaan]

Awesome, thanks!
So... a [random] build takes 198s.
Adding passthrough copy (using 11ty) adds another 200-250s to the build time (previous build was 425.59s).

• "Template Compile": 180s
• "Template Write": 185s
• "_layouts/spotlight/article.pug": 40s
• "_layouts/default.pug": 135s

Somewhat surprising is that you're spending 135+40 = 175 seconds in your layout files (89% of your build time?).
Wonder if you have very complex layouts, or if Pug is the bottleneck (I usually prefer LiquidJS or Nunjucks, but that's just me). Might be a lot of effort to restructure your entire site to a different language just to run some benchmarks.

[ausir0726]

Seems like pug is slow than other template language
but we have use pug develope few years long
It has many benefits
but it also a little bit slow in 11ty, any way...
200s about 2000 pages maybe it acceptable....
400s.... it's a little while for waiting...

but EMFILE error on windows is really troubled us...

[ausir0726]

I tried use recursive-copy alone

benchmak.js

const copy = require('recursive-copy');


const start = async () => {
  console.time('recursive-copy');
  const copied = await copy(`build/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
  console.timeEnd('recursive-copy');
  console.log(`${copied.length} files copied`);
};

start();

result

node benchmak.js
recursive-copy: 3.674s
6080 files copied

don't undersatnd why copy 7000 files in 11ty need 400s?
maybe is possible after file copy than build the templates?

[ausir0726]

like the test before,

a: 592.59ms
c: 2.621s
[11ty] Wrote 2167 files in 192.89 seconds (89.0ms each, v2.0.0-canary.16)
b: 3:14.173 (m:ss.mmm)

seems like render pages and copy files at same time.
so it make io block

[pdehaan]

If merge-dirs works for you, that might be a good option.
In my very rough testing, it seemed to be the same speed as passthrough copy and using recursive-copy directly during the first build. But on subsequent builds, it was much faster since it ignores existing files (although that isn't without risks if an asset changes and merge-dirs ignores the new files. Plus, the library seems to be fairly old and possibly unmaintained.

Although, your ~4 second benchmark might prove that it's better to use recursive-copy rather than add a new dependency into the pipeline, plus it seems better maintained.

I'd also probably try putting the asset copy code in the eleventy.after event handler:

  eleventyConfig.on("eleventy.after", function (args) {
    // Copy files in this event?
    const outputDir = args.dir.output;
    console.log({outputDir}); // `{ outputDir: "www" }`
    // console.log("\n\n", {args}, "\n\n");
  });

If my theory is right, that eleventy.after event should fire after all the 2,168 .pug templates have been processed and written. Then you can copy the 7,883 image(?) assets over so you aren't trying to write 10,051 files all at once and flooding the pipeline/filesystem. At that point, it's just a matter of tweaking the concurrency value, if you still need that. If so, you might only need it for the spotlight/**/.jpg glob:

a: 592.59ms # `${argv.input}/images/**`
b: 3:14.173 (m:ss.mmm) # `${argv.input}/spotlight/**/*.jpg`
c: 2.621s # `${argv.input}/english/spotlight/**/*.jpg`

Although, remember that if you're doing copies on 3 folders w/ default 255 concurrency, that's potentially 765 files that the OS is trying to copy simultaneously.

[ausir0726]

I can't use merge-dirs , because I just only need copy *.jpg files.
It just help for merge src and archive folder for build

as my test

option 1 : use concurrency 50

module.exports = (eleventyConfig) => {
  const argv = yargs(process.argv).argv;
  eleventyConfig.addPassthroughCopy(`${argv.input}/images`, { concurrency: 50 });
  eleventyConfig.addPassthroughCopy(`${argv.input}/spotlight/**/*.jpg`, { concurrency: 50 });
  eleventyConfig.addPassthroughCopy(`${argv.input}/english/spotlight/**/*.jpg`, { concurrency: 50 });
  ...

result total used time : Done in 345.61s.

option 2 : use recursive-copy when build

const copy = require('recursive-copy');

module.exports = (eleventyConfig) => {
  eleventyConfig.setUseGitIgnore(false);
  const argv = yargs(process.argv).argv;

  if (argv.input == 'src') {
    eleventyConfig.addPassthroughCopy(`${argv.input}/images`);
    eleventyConfig.addPassthroughCopy(`${argv.input}/spotlight/**/*.jpg`);
    eleventyConfig.addPassthroughCopy(`${argv.input}/english/spotlight/**/*.jpg`);
  }
  if (argv.input == 'build') {
    copy(`${argv.input}/images`, '_site/images', { overwrite: true });
    copy(`${argv.input}/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
    copy(`${argv.input}/english/spotlight`, '_site/english/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
  }
  ...

result total used time : Done in 202.64s.

option 3 : use recursive-copy when eleventy.after

const copy = require('recursive-copy');

module.exports = (eleventyConfig) => {
  eleventyConfig.setUseGitIgnore(false);
  const argv = yargs(process.argv).argv;

  eleventyConfig.on("eleventy.after", function (args) {
    copy(`${argv.input}/images`, '_site/images', { overwrite: true });
    copy(`${argv.input}/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
    copy(`${argv.input}/english/spotlight`, '_site/english/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
  });

result total used time : Done in 210.24s.

seems like bottleneck is eleventyConfig.addPassthroughCopy
but I don't know why...

I will use the option 2 for fix this problem for now.

thanks.

[pdehaan]

"seems like bottleneck is eleventyConfig.addPassthroughCopy, but I don't know why..."

I think the simplest explanation is that you're trying to concurrently (1) read → (2) compile → (3) render → (4) write 2,167 *.pug files (plus calculating all the data cascade and collections, etc) while also simultaneously copying 7,883 image assets (in potentially [deeply] nested folders which need to be created before passthrough copying).

Although I'm not sure why your

  if (argv.input == 'build') {
    copy(`${argv.input}/images`, '_site/images', { overwrite: true });
    copy(`${argv.input}/spotlight`, '_site/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
    copy(`${argv.input}/english/spotlight`, '_site/english/spotlight', { overwrite: true, filter: ['**/*.jpg'] });
  }

solution is faster than doing it in an eleventy.after event, I would have expected the opposite results but maybe there is still some parallel processing going on.

[ausir0726]

@pdehaan I think addPassthroughCopy have something process will reach open files limit
it's not happen in recusive-copy async function, maybe before copy it ?, maybe some default assets process plugin ?
I tried to understand source code aobut addPassthroughCopy, but can't find way out.

[zachleat]

In retrospect here it seems unlikely that this was caused by recursive-copy (which uses graceful-fs) but more likely a duplicate of #2627 that was fixed in 3.0 with #3272! Please retest with 3.0 if this issue is still needed!