Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer) #2974

Merged
merged 13 commits into from
Jan 27, 2022
Merged

Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer) #2974

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
5684bae
minor refactoring
Cyan4973 Jan 1, 2022
c0c5ffa
streaming compression : lazy parameter adaptation with stable input
Cyan4973 Jan 1, 2022
37b87ad
make stableSrc compatible with regular streaming API
Cyan4973 Jan 2, 2022
27d336b
minor behavior refinements
Cyan4973 Jan 3, 2022
4b9d1dd
fixed incorrect comment
Cyan4973 Jan 5, 2022
8296be4
pretend consuming input to provide a sense of forward progress
Cyan4973 Jan 6, 2022
270f9bf
better consistency in accessing @input
Cyan4973 Jan 24, 2022
c1668a0
fix extended case combining stableInBuffer with continue() and flush(…
Cyan4973 Jan 25, 2022
af3d9c5
added streaming test starting from non-0 pos
Cyan4973 Jan 26, 2022
b99ece9
converted checks into user validation generating error codes
Cyan4973 Jan 26, 2022
cbff372
added helper function inBuffer_forEndFlush()
Cyan4973 Jan 26, 2022
dda4c10
added ZSTD_compressStream2() + ZSTD_c_stableInBuffer test
Cyan4973 Jan 26, 2022
f2d9652
more usage of new error code stabilityCondition_notRespected
Cyan4973 Jan 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/common/error_private.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ const char* ERR_getErrorString(ERR_enum code)
case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
case PREFIX(stabilityCondition_notRespected): return "pledged buffer stability condition is not respected";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ZSTD_checkBufferStability() is currently using srcBuffer_wrong and dstBuffer_wrong. If you're going to add this code, can you please switch them over too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good idea

case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
case PREFIX(dictionary_wrong): return "Dictionary mismatch";
case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
Expand Down
4 changes: 2 additions & 2 deletions lib/compress/zstd_compress.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5678,8 +5678,8 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
&& (totalInputSize < ZSTD_BLOCKSIZE_MAX) ) { /* not even reached one block yet */
if (cctx->stableIn_notConsumed) { /* not the first time */
/* check stable source guarantees */
assert(input->src == cctx->expectedInBuffer.src);
assert(input->pos == cctx->expectedInBuffer.size);
RETURN_ERROR_IF(input->src != cctx->expectedInBuffer.src, stabilityCondition_notRespected, "stableInBuffer condition not respected: wrong src pointer");
RETURN_ERROR_IF(input->pos != cctx->expectedInBuffer.size, stabilityCondition_notRespected, "stableInBuffer condition not respected: externally modified pos");
}
/* pretend input was consumed, to give a sense forward progress */
input->pos = input->size;
Expand Down
1 change: 1 addition & 0 deletions lib/zstd_errors.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ typedef enum {
ZSTD_error_tableLog_tooLarge = 44,
ZSTD_error_maxSymbolValue_tooLarge = 46,
ZSTD_error_maxSymbolValue_tooSmall = 48,
ZSTD_error_stabilityCondition_notRespected = 50,
ZSTD_error_stage_wrong = 60,
ZSTD_error_init_missing = 62,
ZSTD_error_memory_allocation = 64,
Expand Down