v1.5.1

This is the first patch release of sqsh-tools for the 1.5 branch. This release adds sqsh_file_iterator_mt() to the header file.

Full Changelog: v1.5.0...v1.5.1

v1.5.0

Welcome to the fifth minor release of sqsh-tools! This release again focuses on speed improvements and better reliability. We added FreeBSD to the test suite. Again thanks to the sqsh-rs project, libsqsh got some kinks worked out. Especially 32 bit support should now be more stable. Also libsqsh finally gained multithreading support. While being pretty basic, it already is able to increase the decompression speed for sqsh-unpack.

sqsh-ls

sqsh-ls does not only escape files if stdout is a tty.

• -R, --raw - Do not escape filenames, even if the output is a terminal.
• -e, --escape - Do escape filenames, even if the output is a not terminal.

sqsh-unpack

The unpack util now extracts files using multiple threads.

libsqsh

Version functions

libsqsh now contains header macros and functions to determine the version that the binary is compiled against and actual current versions:

• const char* sqsh_version(void) Returns a string containing the library version, e.g. 1.5.0
• uint16_t sqsh_version_major(void) Returns the major version of the current library version, e.g. 1
• uint16_t sqsh_version_minor(void) Returns the minor version of the current library version, e.g. 5
• uint16_t sqsh_version_patch(void) Returns the patch version of the current library version, e.g. 0

Multithreading

While libsqsh was threadsafe since ages, it didn't support any multithreaded functionality. With this version this finally has been implemented. The current implementation is pretty inefficient, but still faster than single core performance.

New Functions

• struct SqshThreadpool* sqsh_threadpool_new(size_t, int*) - creates a new threadpool

• int sqsh_threadpool_wait(struct SqshThreadpool*) - waits until all tasks in the threadpool are done

• int sqsh_threadpool_free(struct SqshThreadpool*) - frees a threadpool. Unfinished tasks are given an opportunity to finish, but no further tasks are being processed. It is advised to call _wait before freeing the pool

• int sqsh_file_to_stream_mt(const SqshFile*, struct SqshThreadpool*, FILE*, sqsh_file_to_stream_mt_cb, void*) - extracts a file to a given file stream. The stream must be randomly accessible, so it wont work for pipes.

• void sqsh_file_iterator_mt(const SqshFile*, struct SqshThreadpool*, sqsh_file_iterator_mt_cb, void*) - creates a struct FileIterator and calls cb for every iteration.

• bool sqsh_file_iterator_is_zero_block(const SqshFileIterator*) - Returns true if the current iteration is a zero block.

Deprecated functions

• time_t sqsh_easy_file_mtime() has been replaced by uint32_t sqsh_easy_file_mtime2()
• size_t sqsh_easy_file_size() has been replaced by uint64_t sqsh_easy_file_size2()
• int sqsh_export_table_resolve_inode() has been replaced by int sqsh_export_table_resolve_inode2()
• uint32_t sqsh_file_block_count() has been replaced by uint64_t sqsh_file_block_count2()
• int sqsh_file_iterator_skip() has been replaced by int sqsh_file_iterator_skip2()
• int sqsh_file_reader_advance() has been replaced by int sqsh_file_reader_advance2()
• size_t sqsh_mapper_size() has been replaced by uint64_t sqsh_mapper_size2()

What's Changed

• common: add jq to docker container by @Gottox in #258
• test: make fuzzer-repro generator be a native target by @Gottox in #260
• utils: add functions to get the current version of the library by @Gottox in #261
• Chores/add usages by @Gottox in #263
• file: use the inode reference for the parent directory. by @Gottox in #262
• Improve/ls better escape by @Gottox in #259
• mapper: add a file mapper that freads file chunks by @Gottox in #144
• file: remove unkown file type by @Gottox in #264
• .Mk.yaml: use build modes by @Gottox in #265
• README.md: split readme by @Gottox in #266
• doc: fix doxygen by @Gottox in #267
• Update README.md by @Gottox in #268
• Doc updates for SqshTreeTraversal by @Dr-Emann in #269
• Allow max_depth=0 for SqshTreeTraversal by @Dr-Emann in #270
• Optimize sqsh_tree_traversal_path_dup by @Dr-Emann in #271
• chores: update cextras by @Gottox in #272
• easy/xattr: Return proper error code when malloc fails by @Gottox in #273
• Recycle stack items in SqshTreeTraversal on pop by @Dr-Emann in #274
• test: add doctest support by @Gottox in #275
• README.md: check result of archive_open. by @Gottox in #276
• file: sqsh__file_parent_inode_ref should be hidden by @Gottox in #277
• config: split .compression_lru_size for data and metablocks by @Gottox in #278
• examples: remove warnings when compiling with clang-17 by @Gottox in #279
• chores: fix 32bit build with -Wconversion by @Gottox in #281
• extract_view: remove field offset. Was never used. by @Gottox in #283
• Add/multithread read by @Gottox in #282
• unpack: use cextras for semaphore by @Gottox in #285
• README.md: mention sqsh-rs by @Gottox in #287
• README.md: add libsqsh link to sqsh-rs by @Gottox in #289
• Fix/sqsh config oob by @Gottox in #291
• mapper: Replace CxRcMap with CxRcRadixTree by @Gottox in #290
• Run FreeBSD tests on Cirrus CI by @Gottox in #292
• v1.5.0 by @Gottox in #288

Full Changelog: v1.4.0...v1.5.0

v1.4.0

This is the fourth minor release of sqsh-tools. This release does not change anything about the command line utils, never the less, there are lots of changes behind the scene. There are new user facing APIs in libsqsh and various refactorings throughout all of the application. Most notable changes are:

sqsh-ls

If a file contains non-printable characters, they will get escaped and printed as \n, \r, \t, ...

sqsh-unpack

sqsh-unpack was moved to the traversal API. This harmonizes the source with sqsh-ls and allows to increase the test coverage of the traversal API.

Also a potential TOCTOU issue has been fixed.

libsqsh

easy traversal API

This releases exposes sqsh_easy_tree_traversal() that returns an array of all files within this path.

support for custom mappers

libsqsh was always quite flexible when in comes to the transport. We support mounting files via http for example. With the new release we expose this API to the users, allowing them to implement their own memory efficient transport. This was implemented to integrate well with radare2

rewrite of SqshTreeTraversal

This release bring a new a rewrite of SqshTreeTraversal. This should make the source more read- and maintainable. The rewrite also brings a new methods:

• sqsh_tree_traversal_set_max_depth() allows to set a maximal traversal depth when creating a tree.

new features of the SqshFile API

Based on the learnings from integrating libsqsh into radare2, we revised the SqshFile API and added a few more functions:

• sqsh_lopen() opens a path without trying to resolve the symlink if it encounters one.
• sqsh_file_symlink_resolve() and sqsh_file_symlink_resolve_all() if the file is a symlink, these function allow to resolve them in one level (sqsh_file_symlink_resolve()) or recursively (sqsh_file_symlink_resolve_all()). libsqsh performs recursion detection, so sqsh_file_symlink_resolve_all() should be safe to be called even on untrusted squashfs archives.

Deprecation

No functionality was deprecated in this release.

Bugs

A bug with the traversal API that sometimes misses to report trailing SQSH_TREE_TRAVERSAL_STATE_DIRECTORY_END events has been fixed.

Internals

This release contains a few refactorings. Notably when an archive is missing an export table, libsqsh tracks the inode number to inode reference (that represents the physical location of an inode inside of the archive) with its own data structures. Before this release, this was a flat array which allowed malicious archives to allocate up to 4GB of data which could lead to crashes on 32bit systems. After this release, the collector uses a radix map as a data backend, which allocates data segments as they are written.

Outlook

The next releases will again focus on the write support of libsqsh. This work has mostly stalled throughout the implementation of this release.

Furthermore I plan to move libcurl support from the library to a subproject and move it into its own repository. Starting with the next major release, curl support won't be a feature of libsqsh itself any more, but provided as a third party library.

libsqsh currently uses hash maps to track the different segments of the archive. This has the downside of unpredictable runtime characteristics and can be used to slow libsqsh down and increase the memory consumption with well crafted archives. To prevent this, libsqsh will replace these hash maps with radix trees, which have predicable memory layouts.

What's Changed

• traversal: rewrite by @Gottox in #198
• Improve/macros by @Gottox in #199
• inode_map: use a vtable instead of branching. by @Gottox in #200
• inode_map: initialise atomics by @Gottox in #201
• inode_map: replace atomics with a mutex. by @Gottox in #202
• ls: escape output filename paths by @Gottox in #203
• write support for uncompressed metablocks by @Gottox in #204
• Chores/license 2024 by @Gottox in #205
• subprojects: update cextras by @Gottox in #206
• fix build on darwin by @Gottox in #207
• utils: fix _log2 return types for platforms where int isn't 32bit by @Gottox in #208
• compression_options: fix build with gcc-13 by @Gottox in #210
• common: the build artifact was renamed to give more context by @Gottox in #212
• introduce utest.h as a replacement for cextras testlib by @Gottox in #213
• test: fix wextra-semi when using utest by @Gottox in #214
• replace testlib with utest by @Gottox in #215
• test: fix -Wextra-semi by @Gottox in #216
• test: move cpp-test to utest.h by @Gottox in #217
• fix clang build on macos by @Gottox in #218
• Fix/alpine container by @Gottox in #220
• unpack: fix dangling file stream by @Gottox in #221
• directory_iterator: fix sonarqube reports. by @Gottox in #222
• extract: disconnect zlib return value from out value by @Gottox in #219
• Improve/remove testlib by @Gottox in #223
• chores: replace Makefile by .Mk.yaml by @Gottox in #225
• Improve/tree traversal documentation by @Gottox in #227
• common: update Dockerfile to Alpine 3.20 by @Gottox in #229
• Improve/release pipeline by @Gottox in #230
• meson.build: do not print warning when sqsh-tools is built with zstd … by @Gottox in #231
• README.md: add reference to backhand. by @Gottox in #232
• mapper: add official support for custom mappers by @Gottox in #228
• mmap_mapper: fix mac/openbsd build by @Gottox in #233
• mapper: make SqshMemoryMapperImpl a public API. by @Gottox in #234
• file: use the generic constructurs/destructors for open and close by @Gottox in #235
• file: add support for sqsh_file_symlink_resolve by @Gottox in #236
• common: do not export sqsh__log2 by @Gottox in #238
• .Mk.yaml: configure to be compatible with abi-dumper by @Gottox in #239
• Improve/path resolution cleanup by @Gottox in #237
• Add/easy traversal by @Gottox in #226
• Fix/resolve all export by @Gottox in #240
• test: reactivate xattr integration tests by @Gottox in #241
• Fix/obsd test by @Gottox in #243
• Better testing of mmap, move posix specific functions to own module. by @Gottox in #242
• traversal: rewrite implementation by @Gottox in #244
• chores: update cextras to fix MacOS build issue by @Gottox in #245
• Fix/openbsd build by @Gottox in #246
• .github: upload failing inputs from fuzzer runs by @Gottox in #248
• common: use CX_*_IMPL macros instead of defining our own. by @Gottox in #249
• chores: update cextras to fix mac/openbsd build by @Gottox in #250
• file: remove auto search of directory inodes by @Gottox in #251
• test: move inputs to their original names. by @Gottox in #252
• inode_map: fix deadlock when the inode map encounters inconsistencies by @Gottox in #253
• Improve/remove magic dir node number by @Gottox in #247
• traversal: push element to the stack as soon as possible by @Gottox in #254
• v1.4.0 by @Gottox in #255
• file: Do not e...

v1.3.1

This is the first micro release to the v1.3 branch of sqsh-tools. It fixes compile warnings with gcc-13. There are no API changes.

Full Changelog: v1.3.0...v1.3.1

v1.3.0

This is the third minor release of sqsh-tools and the first release after the rename from libsqsh. Also a lot of effort was put into speed improvements. This should be instantly noticable when listing files. There are a lot of changes in this release. Most notable are:

sqsh-ls

Speed! sqsh-ls uses new functionality mentioned below that results in much faster listing of files. In that regard we are consistently faster than unsquashfs -lls and rdsquashfs -l.

Also, sqsh-ls gained the --utc flag which allows to ignore the timezone and just report UTC.

libsqsh

Tree Traversal API

During performance analytics we noticed, that a lot of time was wasted in letting the libc manage memory when doing tree traversals in user code. To give the programmer a way to reduce malloc/free cycles within their code, we're introducing SqshTreeTraversal. This API allows to recursively traverse a file tree at a given start directory or file. It is already used in sqsh-ls and one of the most significant speed improvements during this release. Currently we only provide the zero-copy lowlevel API, which has some rough edges, but we provide condensed examples with this library.

New easy APIs

This release introduces the sqsh_easy_directory_list_path() function that works like sqsh_easy_directory_list() with the difference, that the _path function does return the fully qualified path instead of just the content names.

Deprecation

This release deprecates most of the non-null-terminated getter/size-getter function compinations like sqsh_directory_iterator_name() and sqsh_directory_iterator_name_size() and replaces them with a ...2() variant that is able to report size and value back to the caller in one function call. This make it hopefully harder to misuse these API as with these new functions there is actually a syntactical difference between null terminated and non-terminated strings.

We also deprecated some functions that have wrong return types. As replacement, we introduces new ...2() variants of these functions.

Bugs

A long outstanding bug has been fixed: When iterating over big directories, libsqsh could wrongly report a corrupted file, even if it was valid.

Internals

libsqsh had gone through a lot of internal refactoring, making the SqshFile code more modular. The different data types are handled in their own module now, instead of having switch-cases everywhere in SqshFile.

We also began to implement write support for libsqsh. This is far from finished and none of the APIs are exposed or even compiled in the library. Nonetheless, we already use those APIs to make writing unit test easier for us.

We also fixed some issues when using libsqsh-lzo with libsqsh when it was build with -O2.

Also we enabled -Wconversation for libsqsh. This allows us to uncover potential bugs when handling large files on 32 bit platforms.

Outlook

This was one of the bigger releases we did. From here, we'll deprecate more of the getter/size-getter and provide alternatives following the new pattern.

Also sqsh-unpack will be migrated the the tree traverse API, keeping it more in line with sqsh-ls.

We also will wrap functionality of the tree traverse API in new sqsh_easy_ functions.

Currently libsqsh is already able to work in multithreaded contexts, it's functionality is part of the tests. Nonetheless, there are no simple APIs in place that would actually allow multithreaded decompression for example. We're aiming to support multithreaded file extraction without letting the user of this library take care about threading themselves.

We will continiously add more functionality to the write support in libsqsh, but don't expect too much. Write support is quite a beast. libsqsh-1.x will continue to be readonly.

What has Changed

• extract: lookup extractor implementation once by @Gottox in #158
• extract: fix doxygen by @Gottox in #159
• tools: move header files to include directory by @Gottox in #160
• easy: add a function to list directory paths by @Gottox in #161
• chores: rewrite for sqsh-tools renaming by @Gottox in #162
• sqsh-ls: add --utc flag by @Gottox in #163
• ls: avoid newfstatat syscalls incritical path by @Gottox in #164
• ls: refactor PRINT_MODE macro into static functions by @Gottox in #165
• file_iterator: dynamically allocate zero block by @Gottox in #166
• file_iterator: Correctly set the sparse_size. by @Gottox in #168
• extract: fix tests when building with -O2 by @Gottox in #169
• include: fix include by @Gottox in #170
• subprojects: update cextras by @Gottox in #171
• subprojects: update cextras, make download on demand by @Gottox in #172
• .gitignore: fix typo by @Gottox in #173
• archive: fix memory leak in error handling by @Gottox in #174
• Fix/fuzzer by @Gottox in #175
• .github: run a build test for 32bit compilation by @Gottox in #177
• file: add implementation table instead of checking the type every time. by @Gottox in #176
• Fix/xattr iterator remove strlen by @Gottox in #180
• chores: add and fix conversion warnings in libsqsh by @Gottox in #178
• meson.build: print warning when no compression is enabled by @Gottox in #181
• introduce superblock_builder by @Gottox in #179
• file: add default implementations for inode types by @Gottox in #182
• file: use own getter for the symlink size by @Gottox in #183
• fuzzer: fix leak when calling xattr_iterator by @Gottox in #184
• data: fix macos build by @Gottox in #185
• superblock: move log2 to common module by @Gottox in #186
• Add/tree traversal by @Gottox in #187
• Add/tree traversal by @Gottox in #188
• traversal: fix macos build. by @Gottox in #189
• test: fix race condition in tools tests by @Gottox in #190
• directory_iterator: fix errors when doing fast filename lookup by @Gottox in #191
• directory: replace _size getters with a combined _name setter. by @Gottox in #192
• reader: cache iterator size inside of reader. by @Gottox in #193
• directory_iterator: use memchr for consistency checking by @Gottox in #194
• inode_map: remove already processed TODOs by @Gottox in #196

Full Changelog: v1.2.0...v1.3.0

v1.2.1

This is a maintainance release that fixes an issue with sparse files.

What's Changed

• file_iterator: Correctly set the sparse_size. @Gottox in #168

Full Changelog: v1.2.0...v1.2.1

v1.2.0

This release is centered around stabilization and API consistency. It also restructured the source code to prepare further changes. Most notable changes are:

• tools: tools (sqsh-ls, sqsh-cat, sqsh-stat, sqsh-xattr, sqshfs) support long-options now.
• the SqshTreeWalker module has been deprecated. It was replaced by SqshPathResolver. This solves possible misunderstandings about the purpose (and therefore lacking features) of this module.
• sqsh_inode_map_get() and sqsh_inode_map_set() have been deprecated and were replaced by sqsh_inode_map_get2() and sqsh_inode_map_set2() this allows the user to properly catch errors when inserting malicious data into the map.
• sqsh_directory_iterator_inode_number() has been deprecated and was replaced by sqsh_directory_iterator_inode(). functionally both functions are equivalent, it's just a cosmetical change.

What's Changed

• extract: make sure return codes are propagated by @Gottox in #109
• file_iterator: return correct error when iterator is not a file by @Gottox in #110
• .github: allow patch to not met coverage goals by @Gottox in #111
• metablock_iterator: _skip function is no longer exported by @Gottox in #113
• directory_iterator: replace _inode_number with _inode by @Gottox in #112
• directory_iterator: un-inline _inode_number by @Gottox in #114
• utils: move memdup from libsqsh to cextras by @Gottox in #115
• test: add tests for easy file functions by @Gottox in #116
• tree_walker: fix _next function. by @Gottox in #118
• tree_walker: introduce iterator-like _next2 function by @Gottox in #119
• tree_walker: fix typos and c&p errors in header documentation by @Gottox in #120
• inode_map: Replace _map_get() with _map_get2() by @Gottox in #122
• inode_map: add new test cases. by @Gottox in #123
• inode_map: Replace _map_set() with _map_set2() by @Gottox in #125
• README.md: rephrase the easy-to-use API description by @Gottox in #127
• tree_walker: document the path base for tree_walker by @Gottox in #128
• Makefile: run with ccache by default by @Gottox in #130
• rearrange data definitions by @Gottox in #129
• directory_iterator: fix potential over-/underflow by @Gottox in #131
• data: replace htoleXX calls by leXXtoh by @Gottox in #132
• directory_iterator: improve documentation by @Gottox in #135
• Update README.md by @Gottox in #136
• curl_mapper: enforce at least TLS v1.2 by @Gottox in #137
• chores: put _new and _free functions into a macro by @Gottox in #138
• tree: rename walker to path resolver. by @Gottox in #139
• .github: publish to doxygen subdirectory by @Gottox in #140
• metablock_iterator: fix memory leak by @Gottox in #142
• .github: check for meson format correctness by @Gottox in #145
• Improve/rearrange source by @Gottox in #146
• libsqsh: move utils.h to common library by @Gottox in #148
• sqsh_tree_walker: correct deprecation messages. by @Gottox in #150
• tools: support longopts by @Gottox in #153
• Improve/rm alloca by @Gottox in #154
• Improve/modularize utils by @Gottox in #155
• test: fix build with muon. by @Gottox in #156
• v1.2.0 by @Gottox in #157

Full Changelog: v1.1.0...v1.2.0

v1.1.2

This is a maintenance release fixing the following issues:

Cherry Picked PRs

• metablock_iterator: fix memory leak by @Gottox in #142

Full Changelog: v1.1.1...v1.1.2

v1.1.1

This is a maintenance release fixing the following issues:

Cherry Picked PRs

• file_iterator: return correct error when iterator is not a file by @Gottox in #110
• extract: make sure return codes are propagated by @Gottox in #109
• tree_walker: fix _next function. by @Gottox in #118

Full Changelog: v1.1.0...v1.1.1

v1.1.0

This is the second stable release of libsqsh. It introduces new userfacing APIs to work with xattrs and hardens the directory and path handling against corrupt archives. Also the tooling was improved as not PRs are checked for format errors and code coverage (using codecov.io). There were also preparations for chunked decompression, a feature that will be finished in later versions and eliminate a few edge cases where data copies still happen.

All in all this release had a good developing pace and I'm happy with the result. The library is stabilizing and maturing fine.

Merged PRs

• .github: run doxygen step even if openbsd is not by @Gottox in #72
• Create codeql.yml by @Gottox in #73
• directory_iterator: consistency check on open by @Gottox in #74
• Makefile: fix doc target by @Gottox in #76
• Fix build on MacOS by @Gottox in #78
• README.md: document better what libsqsh actually is by @Gottox in #80
• README.md: fix typo by @Gottox in #81
• tree: add test for recursively alternating symlinks by @Gottox in #79
• easy: use generic collector for easy directory by @Gottox in #82
• easy: add xattr API. by @Gottox in #83
• .github: add CodeCov report by @Gottox in #84
• README.md: add CodeCov badge by @Gottox in #85
• Fix spelling of "unknown" by @Dr-Emann in #86
• include: make field with typo deprecated by @Gottox in #87
• .github: add codecov.yaml configuration by @Gottox in #88
• meson.build: add version constraint to libcurl by @Gottox in #89
• test: add in depth tests for compression options by @Gottox in #90
• .github: disable curl build on mac by @Gottox in #91
• README.md: add license badge by @Gottox in #92
• .github: configure code-coverage by @Gottox in #93
• .github: run CI on pull requests to main by @Gottox in #95
• Make sqsh_error_str thread safe, and document that the value may be overwritten by @Dr-Emann in #94
• clang-format by @Gottox in #96
• .github: run format check before running actual tests. by @Gottox in #97
• easy: fix wrong key length in xattr. by @Gottox in #98
• Improve directory handling by @Gottox in #100
• test: add test for easy file exists by @Gottox in #102
• lib: sync README.md with reality by @Gottox in #101
• README.md: typo fix. by @Gottox in #103
• Reimplement extractor by @Gottox in #104
• v1.1.0 by @Gottox in #105
• include: fix documentation for sqsh_inode_map_get() by @Gottox in #107

New Contributors

• @Dr-Emann made their first contribution in #86

Full Changelog: v1.0.0...v1.1.0