Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zlib: Make the finish flush flag configurable #6069

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions doc/api/zlib.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,31 @@ http.createServer((request, response) => {
}).listen(1337);
```

By default, the zlib methods with throw an error when decompressing
truncated data. However, if it is known that the data is incomplete, or
the desire is to inspect only the beginning of a compressed file, it is
possible to suppress the default error handling by changing the flushing
method that is used to compressed the last chunk of input data:

```js
// This is a truncated version of the buffer from the above examples
const buffer = new Buffer('eJzT0yMA', 'base64');

zlib.unzip(buffer, { finishFlush: zlib.Z_SYNC_FLUSH }, (err, buffer) => {
if (!err) {
console.log(buffer.toString());
} else {
// handle error
}
});
```

This will not change the behavior in other error-throwing situations, e.g.
when the input data has an invalid format. Using this method, it will not be
possible to determine whether the input ended prematurely or lacks the
integrity checks, making it necessary to manually check that the
decompressed result is valid.

## Memory Usage Tuning

<!--type=misc-->
Expand Down Expand Up @@ -231,6 +256,7 @@ Note that some options are only relevant when compressing, and are
ignored by the decompression classes.

* flush (default: `zlib.Z_NO_FLUSH`)
* finishFlush (default: `zlib.Z_FINISH`)
* chunkSize (default: 16*1024)
* windowBits
* level (compression only)
Expand Down
32 changes: 20 additions & 12 deletions lib/zlib.js
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ function zlibBufferSync(engine, buffer) {
if (!(buffer instanceof Buffer))
throw new TypeError('Not a string or buffer');

var flushFlag = binding.Z_FINISH;
var flushFlag = engine._finishFlushFlag;

return engine._processChunk(buffer, flushFlag);
}
Expand Down Expand Up @@ -282,6 +282,14 @@ function Unzip(opts) {
Zlib.call(this, opts, binding.UNZIP);
}

function isValidFlushFlag(flag) {
return flag === binding.Z_NO_FLUSH ||
flag === binding.Z_PARTIAL_FLUSH ||
flag === binding.Z_SYNC_FLUSH ||
flag === binding.Z_FULL_FLUSH ||
flag === binding.Z_FINISH ||
flag === binding.Z_BLOCK;
}

// the Zlib class they all inherit from
// This thing manages the queue of requests, and returns
Expand All @@ -294,17 +302,16 @@ function Zlib(opts, mode) {

Transform.call(this, opts);

if (opts.flush) {
if (opts.flush !== binding.Z_NO_FLUSH &&
opts.flush !== binding.Z_PARTIAL_FLUSH &&
opts.flush !== binding.Z_SYNC_FLUSH &&
opts.flush !== binding.Z_FULL_FLUSH &&
opts.flush !== binding.Z_FINISH &&
opts.flush !== binding.Z_BLOCK) {
throw new Error('Invalid flush flag: ' + opts.flush);
}
if (opts.flush && !isValidFlushFlag(opts.flush)) {
throw new Error('Invalid flush flag: ' + opts.flush);
}
if (opts.finishFlush && !isValidFlushFlag(opts.finishFlush)) {
throw new Error('Invalid flush flag: ' + opts.finishFlush);
}

this._flushFlag = opts.flush || binding.Z_NO_FLUSH;
this._finishFlushFlag = typeof opts.finishFlush !== 'undefined' ?
opts.finishFlush : binding.Z_FINISH;

if (opts.chunkSize) {
if (opts.chunkSize < exports.Z_MIN_CHUNK ||
Expand Down Expand Up @@ -486,12 +493,13 @@ Zlib.prototype._transform = function(chunk, encoding, cb) {
if (this._closed)
return cb(new Error('zlib binding closed'));

// If it's the last chunk, or a final flush, we use the Z_FINISH flush flag.
// If it's the last chunk, or a final flush, we use the Z_FINISH flush flag
// (or whatever flag was provided using opts.finishFlush).
// If it's explicitly flushing at some other time, then we use
// Z_FULL_FLUSH. Otherwise, use Z_NO_FLUSH for maximum compression
// goodness.
if (last)
flushFlag = binding.Z_FINISH;
flushFlag = this._finishFlushFlag;
else {
flushFlag = this._flushFlag;
// once we've flushed the last of the queue, stop flushing and
Expand Down
28 changes: 28 additions & 0 deletions test/parallel/test-zlib-flush-flags.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
'use strict';
require('../common');
const assert = require('assert');
const zlib = require('zlib');

assert.doesNotThrow(() => {
zlib.createGzip({ flush: zlib.Z_SYNC_FLUSH });
});

assert.throws(() => {
zlib.createGzip({ flush: 'foobar' });
}, /Invalid flush flag: foobar/);

assert.throws(() => {
zlib.createGzip({ flush: 10000 });
}, /Invalid flush flag: 10000/);

assert.doesNotThrow(() => {
zlib.createGzip({ finishFlush: zlib.Z_SYNC_FLUSH });
});

assert.throws(() => {
zlib.createGzip({ finishFlush: 'foobar' });
}, /Invalid flush flag: foobar/);

assert.throws(() => {
zlib.createGzip({ finishFlush: 10000 });
}, /Invalid flush flag: 10000/);
17 changes: 17 additions & 0 deletions test/parallel/test-zlib-truncated.js
Original file line number Diff line number Diff line change
Expand Up @@ -45,5 +45,22 @@ const inputString = 'ΩΩLorem ipsum dolor sit amet, consectetur adipiscing el'
zlib[methods.decomp](truncated, function(err, result) {
assert(/unexpected end of file/.test(err.message));
});

const syncFlushOpt = { finishFlush: zlib.Z_SYNC_FLUSH };

// sync truncated input test, finishFlush = Z_SYNC_FLUSH
assert.doesNotThrow(function() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just a suggestion/ idea): Maybe you could add a case that triggers the new error if the flush flag is not valid? (Just to make sure that the flag was caught somewhere)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Updated the PR with some tests for that.

const result = zlib[methods.decompSync](truncated, syncFlushOpt)
.toString();
assert.equal(result, inputString.substr(0, result.length));
});

// async truncated input test, finishFlush = Z_SYNC_FLUSH
zlib[methods.decomp](truncated, syncFlushOpt, function(err, decompressed) {
assert.ifError(err);

const result = decompressed.toString();
assert.equal(result, inputString.substr(0, result.length));
});
});
});