Support multiple directory trees and filename changes

[oconnore]

gulp-newer currently only supports single directory targets with direct file matches, and single file 'all' targets. For my build I need to do things like:

./public/**/*.html --> ./working/
./public/**/*.styl --> ./working/**/*.css
 --- results in ---
./public/a/b/c/index.html -> ./working/a/b/c/index.html
./public/a/b/c/index.html -> ./working/a/b/c/index.html
./public/css/stylus/main.styl -> ./working/css/stylus/main.css

I setup my build without using gulp-newer, but would like to contribute back to make this module work better for everyone (also me) in the future.

Currently, my API for these things looks like:

.pipe(newer(function(srcPath) {
  return destPath;
}))

That's really [infinitely] flexible, but maybe preserving a nicer API is a good idea as well. Lets start with:

Nested directories

If we are OK requiring that the directory structure in the target is built beforehand, we can use it to find the deepest possible match (from the right) relative to the base directory, and require no change to the gulp-newer API. This heuristic should work most of the time:

directories:
./working/a/b/c
./working/a/b/d
./working/a/b/c/e
./public/a/b/c/index.html ->./working/a/b/c/index.html

(Until nodejs/node-v0.x-archive#6662 is resolved, this may require memoizing stat calls -- I had to do that for my build, but I would want to do more profiling before I committed that assumption to a public project.)

Pathname changes

Changes of file name (extension, append, replace, etc.) is a bit more complicated, and while I can imagine some ways to integrate it with the existing single-string configuration style, that seems messy, non obvious, and prone to error. For example, attempts to correlate output files on the first build with source files, and attempts to parse a change in filename from a globbed output pattern both seem really unpleasant to implement, support, and use.

We also (I think?) would like to not require users to write a conversion function when possible, but should allow it when necessary. Here are some thoughts:

• Allow an options object to be passed as newer(destBase, { /* options */ })
• Allow rewriting the path and the filename separately. Accept a sourceBase option to change the cwd.

options = {
  destBase: /* allow the options object to be passed by itself */,
  srcBase: /* base directory of source files */,
  path: /* change path from relative to sourceBase */,
  filename: /* change filename */,
  fullpath: /* change path and filename at the same time */
}

• for path, filename, and fullpath mutators:
  • accept handlebars style string {filebase}.css with keys like:
    • {path [0-9]+(-[0-9]+)?}
    • {filebase} and {fileext}
  • accept an s/from/to/ style regex
  • accept a function that converts from source -> dest paths

Discussion

What do you think? Would you accept a pull request for something like this + feedback?

[tschaub]

Thanks for trying this out. I'd like to better understand what you're running into. I think it would be easier to talk in terms of the existing plugins your using and how they are configured.

./public/**/*.styl --> ./working/**/*.css

I assume this means you're configuring the gulp-stylus plugin with something like this:

gulp.task('stylus', function() {
  gulp.src('./public/**/*.styl')
      .pipe(stylus())
      .pipe(gulp.dest('./working'));
});

The minimum extra information gulp-newer needs to know in this case is the destination extension. So something like this could be made to work:

gulp.task('stylus', function() {
  gulp.src('./public/**/*.styl')
      .pipe(newer({dest: './working', ext: '.css'}))
      .pipe(stylus())
      .pipe(gulp.dest('./working'));
});

Does this treat the first case you mention?

Can you give examples (with code) of other cases you'd like covered?

[tschaub]

Support for the ext option added in #2 and #3, published as [email protected].

[peterbraden]

These options are nice, but I think there is one further case that needs support. I've been trying to find a way for gulp-newer to play nicely with gulp-rev (sindresorhus/gulp-rev#32)

In this scenario, the filename is changed, not just the extension.

I'm contemplating something like a mapping function passed to the options that can try and resolve a file, ie:

.pipe(newer({mapping: function(filename, dest, cb){
    //... My custom logic to look in dest for filename ...
}}));

[emgeee]

Any word on this? I'm in a similar boat whereby I have a pipe with a big ol rename() at the end of it that absolutely mucks up this or gulp-changed.

Personally, I'm thinking that gulp-newer should operate similar to gulp-rename in that 'dest' can be a string or a function that is passed the vinyl object and returns a string.

[tschaub]

See also #10 (comment). If you squint at it the right way, I think these are all the same suggestion. Something like @peterbraden's suggestion would be straightforward to implement.

If people can post more complete example tasks where they'd use a mapping option, that would be great.

[emgeee]

I'll describe a use case I'm experiencing right now:

I'm using gulp as a static site generator and I'm looking to have clean urls without extensions. Typically, a webserver will automatically serve the index.html file when a user navigates to the root, otherwise it requires the full file path, extension included. To remove having to specify the .html in the url, the final step in my pipeline is to rename <filename>.html to <filename>/index.html if filename !== 'index.html'. My rename code looks like this:

.pipe(rename(function(path) {
  if(path.basename !== 'index') {
    path.dirname += '/'+path.basename;
    path.basename = 'index';
  }
  return path;
})

In order for gulp-newer to work for me, I essentially need to be able to use the same logic. In some cases, I'm able to apply the rename step first thing in the pipeline (before gulp-newer) however it doesn't always work since the .html templates extend swig templates which are referenced relatively and so renaming the file before compiling the template would break the references.

[tschaub]

@emgeee - curious to hear if the map option added in #12 meets your needs. You provide a function that will be called with a relative path.

[emgeee]

@tschaub Yep, this solution works great!

[agborkowski]

@oconnore TC !

gulp.task('images', function() {
   return gulp.src('resources/uploads/**/*')
    //.pipe(changed('webroot/galleries'))
    //.pipe(cache('images'))
    .pipe(newer({
      dest: 'webroot/galleries',
      map: function(pathDir) { 
        pathDir = 
          path.dirname(pathDir) + '/50-x-' +
          path.basename(pathDir, path.extname(pathDir)) + '-thumb' + 
          path.extname(pathDir);
        return pathDir; 
      }
    }))
    .pipe(imagemin({ optimizationLevel: 3, progressive: true, interlaced: true }))
    .pipe(
      responsive(
      {
        '**/*' : [{
          width: 50,
          rename: {
            //path.dirname += "";
            prefix: "50-x-",
            suffix: "-thumb"
            //path.extname = ".md"
          }
        }]
      })
    )
    .pipe(gulp.dest('webroot/galleries'))
    .pipe(notify({ message: 'Images task complete' }));
});

[Sawtaytoes]

I have one other suggestion for this functionality.

I've got an ES6 file with the extension .es6. This means I build into .js, .min.js, and .map or depending on how I do this, a .js, .map, and an app.bundle.js depending on if I build development or production. If someone deletes the .map file or bundle, when I go to build, any file that hadn't had changes, such as the one w/ the deleted .map file wouldn't be built because I'm doing this:

.pipe($.newer({
    dest: dest,
    ext: '.js'
}))

What I want is:

ext: [
    '.js',
    '.min.js'
    '.map'
]

I can't figure out how to pipe in two gulp-newer operations because the first would cancel out the second if it exists. So this might be possible that way as well.

[tschaub]

@Saturn2888 - could you open a separate issue with your suggestion? This ticket has migrated pretty far from the original title/description. I'm going to close this as addressed by #2, #3, and #12.