v1.3.0

This is the third minor release of sqsh-tools and the first release after the rename from libsqsh. Also a lot of effort was put into speed improvements. This should be instantly noticable when listing files. There are a lot of changes in this release. Most notable are:

sqsh-ls

Speed! sqsh-ls uses new functionality mentioned below that results in much faster listing of files. In that regard we are consistently faster than unsquashfs -lls and rdsquashfs -l.

Also, sqsh-ls gained the --utc flag which allows to ignore the timezone and just report UTC.

libsqsh

Tree Traversal API

During performance analytics we noticed, that a lot of time was wasted in letting the libc manage memory when doing tree traversals in user code. To give the programmer a way to reduce malloc/free cycles within their code, we're introducing SqshTreeTraversal. This API allows to recursively traverse a file tree at a given start directory or file. It is already used in sqsh-ls and one of the most significant speed improvements during this release. Currently we only provide the zero-copy lowlevel API, which has some rough edges, but we provide condensed examples with this library.

New easy APIs

This release introduces the sqsh_easy_directory_list_path() function that works like sqsh_easy_directory_list() with the difference, that the _path function does return the fully qualified path instead of just the content names.

Deprecation

This release deprecates most of the non-null-terminated getter/size-getter function compinations like sqsh_directory_iterator_name() and sqsh_directory_iterator_name_size() and replaces them with a ...2() variant that is able to report size and value back to the caller in one function call. This make it hopefully harder to misuse these API as with these new functions there is actually a syntactical difference between null terminated and non-terminated strings.

We also deprecated some functions that have wrong return types. As replacement, we introduces new ...2() variants of these functions.

Bugs

A long outstanding bug has been fixed: When iterating over big directories, libsqsh could wrongly report a corrupted file, even if it was valid.

Internals

libsqsh had gone through a lot of internal refactoring, making the SqshFile code more modular. The different data types are handled in their own module now, instead of having switch-cases everywhere in SqshFile.

We also began to implement write support for libsqsh. This is far from finished and none of the APIs are exposed or even compiled in the library. Nonetheless, we already use those APIs to make writing unit test easier for us.

We also fixed some issues when using libsqsh-lzo with libsqsh when it was build with -O2.

Also we enabled -Wconversation for libsqsh. This allows us to uncover potential bugs when handling large files on 32 bit platforms.

Outlook

This was one of the bigger releases we did. From here, we'll deprecate more of the getter/size-getter and provide alternatives following the new pattern.

Also sqsh-unpack will be migrated the the tree traverse API, keeping it more in line with sqsh-ls.

We also will wrap functionality of the tree traverse API in new sqsh_easy_ functions.

Currently libsqsh is already able to work in multithreaded contexts, it's functionality is part of the tests. Nonetheless, there are no simple APIs in place that would actually allow multithreaded decompression for example. We're aiming to support multithreaded file extraction without letting the user of this library take care about threading themselves.

We will continiously add more functionality to the write support in libsqsh, but don't expect too much. Write support is quite a beast. libsqsh-1.x will continue to be readonly.

What has Changed

• extract: lookup extractor implementation once by @Gottox in #158
• extract: fix doxygen by @Gottox in #159
• tools: move header files to include directory by @Gottox in #160
• easy: add a function to list directory paths by @Gottox in #161
• chores: rewrite for sqsh-tools renaming by @Gottox in #162
• sqsh-ls: add --utc flag by @Gottox in #163
• ls: avoid newfstatat syscalls incritical path by @Gottox in #164
• ls: refactor PRINT_MODE macro into static functions by @Gottox in #165
• file_iterator: dynamically allocate zero block by @Gottox in #166
• file_iterator: Correctly set the sparse_size. by @Gottox in #168
• extract: fix tests when building with -O2 by @Gottox in #169
• include: fix include by @Gottox in #170
• subprojects: update cextras by @Gottox in #171
• subprojects: update cextras, make download on demand by @Gottox in #172
• .gitignore: fix typo by @Gottox in #173
• archive: fix memory leak in error handling by @Gottox in #174
• Fix/fuzzer by @Gottox in #175
• .github: run a build test for 32bit compilation by @Gottox in #177
• file: add implementation table instead of checking the type every time. by @Gottox in #176
• Fix/xattr iterator remove strlen by @Gottox in #180
• chores: add and fix conversion warnings in libsqsh by @Gottox in #178
• meson.build: print warning when no compression is enabled by @Gottox in #181
• introduce superblock_builder by @Gottox in #179
• file: add default implementations for inode types by @Gottox in #182
• file: use own getter for the symlink size by @Gottox in #183
• fuzzer: fix leak when calling xattr_iterator by @Gottox in #184
• data: fix macos build by @Gottox in #185
• superblock: move log2 to common module by @Gottox in #186
• Add/tree traversal by @Gottox in #187
• Add/tree traversal by @Gottox in #188
• traversal: fix macos build. by @Gottox in #189
• test: fix race condition in tools tests by @Gottox in #190
• directory_iterator: fix errors when doing fast filename lookup by @Gottox in #191
• directory: replace _size getters with a combined _name setter. by @Gottox in #192
• reader: cache iterator size inside of reader. by @Gottox in #193
• directory_iterator: use memchr for consistency checking by @Gottox in #194
• inode_map: remove already processed TODOs by @Gottox in #196

Full Changelog: v1.2.0...v1.3.0