split command not working right #3790
Labels
Comments
ioperations
changed the title
resistor
added a commit
to resistor/coreutils
that referenced
this issue
Aug 10, 2022
happened to be an exact divisor of the buffer size used to read the input stream. The issue here was that file was being split byte-wise in chunks of 1G. The input stream was being read in chunks of 8KB, which evenly divides the chunk size. Because the check to allocate the next output chunk was done at the bottom of the loop previously, it would never occur because the current input chunk was fully consumed at that point. By moving the check to the top of the loop (but still late enough that we know we have bytes to write) we resolve this issue. This scenario is unfortunately hard to write a test for, since we don't explicitly control the input chunk size. Fixes uutils#3790
resistor
added a commit
to resistor/coreutils
that referenced
this issue
Aug 16, 2022
happened to be an exact divisor of the buffer size used to read the input stream. The issue here was that file was being split byte-wise in chunks of 1G. The input stream was being read in chunks of 8KB, which evenly divides the chunk size. Because the check to allocate the next output chunk was done at the bottom of the loop previously, it would never occur because the current input chunk was fully consumed at that point. By moving the check to the top of the loop (but still late enough that we know we have bytes to write) we resolve this issue. This scenario is unfortunately hard to write a test for, since we don't explicitly control the input chunk size. Fixes uutils#3790
sylvestre
pushed a commit
that referenced
this issue
Aug 16, 2022
#3800) * Fix a bug in split where chunking would be skipped when the chunk size happened to be an exact divisor of the buffer size used to read the input stream. The issue here was that file was being split byte-wise in chunks of 1G. The input stream was being read in chunks of 8KB, which evenly divides the chunk size. Because the check to allocate the next output chunk was done at the bottom of the loop previously, it would never occur because the current input chunk was fully consumed at that point. By moving the check to the top of the loop (but still late enough that we know we have bytes to write) we resolve this issue. This scenario is unfortunately hard to write a test for, since we don't explicitly control the input chunk size. Fixes #3790
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
$ split -b 1G big2.4Gfile && ll 2.4GB big2.4Gfile 1GB xaa 1GB xab 460MB xac$ ./target/debug/coreutils split -b 1G big2.4Gfile && ll 2.4GB big2.4Gfile 2.4GB xaaThe text was updated successfully, but these errors were encountered: