Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessively large filesize reported #315

Open
Jacoby1218 opened this issue Dec 5, 2022 · 4 comments
Open

Excessively large filesize reported #315

Jacoby1218 opened this issue Dec 5, 2022 · 4 comments

Comments

@Jacoby1218
Copy link

This happened when i was trying to compress my Microsoft Flight Simulator packages folder. Not sure why or if there's something i can provide to you to help.
image

@TomB654
Copy link

TomB654 commented Dec 18, 2022

I've had the same thing happen with SW:KotOR, just with less extreme numbers. However, the folder properties show that it has actually been compressed properly.

Screenshot CompactGUI

@Freaky
Copy link

Freaky commented Dec 31, 2022

I note there's no error handling in the GetCompressedFileSize call:

    Function GetFileSizeOnDisk(file As String) As Long
        Dim hosize As UInteger
        Dim losize As UInteger = GetCompressedFileSizeW(file, hosize)
        Return CLng(hosize) << 32 Or losize
    End Function

Per Microsoft documentation:

If the return value is INVALID_FILE_SIZE and lpFileSizeHigh is non-NULL, an application must call GetLastError to determine whether the function has succeeded (value is NO_ERROR) or failed (value is other than NO_ERROR).

INVALID_FILE_SIZE happens to be (2^32)-1. If these calls are failing for some reason, CompactGUI will think each one is 4 GiB.

@sonicviz
Copy link

sonicviz commented Jan 1, 2023

I just tried this on my MSFS add ons folder and got the same issue.

What's the advantage of using this vs the native Win 11 folder compression option in the folder properties dialog?
Afaik by using the windows compression option it will automatically compress new files added to that compress folder (and/or subfolders) using the same API methods, so what's the advantage of using this app over native Win 11 functionality?

@Freaky
Copy link

Freaky commented Jan 1, 2023

The file/folder property you're describing is the old NTFS LZNT1 compression system introduced in 1995 with Windows NT 3.51.

It works in 64k chunks, and saves space by using sparse cluster fragments - i.e. if you get 64k down to 44k, you only need to allocate 11 x 4k pages instead of the full 16 that 64k of data would have otherwise taken, plus the metadata to store that fragment.

Windows Overlay Filter file compression - which is what CompactGUI and my own tool is using - was introduced in Windows 10. It uses the same family of compressors (LZ77), but more modern variants which compress better, and a simpler storage mechanism where compressed data is simply written to a contiguous alternate data stream with a packed list of block offsets.

The new approach makes better use of its more advanced compressors, as it isn't only limited to saving in multiples of 4k per chunk, and has less filesystem overhead due to its simpler metadata compared with storing fragments. It also doesn't share the same file size limitations - NTFS only supports a limited number of fragments per file. The disadvantage, of course, is it's write-once.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants