Skip to content

Latest commit

 

History

History
394 lines (346 loc) · 14.6 KB

README.adoc

File metadata and controls

394 lines (346 loc) · 14.6 KB

Rust U4Pak

Build status Release License

More or less re-implementation of Python U4Pak for fun, ease of use (standalone binary), and speed (multi-threading).

This is a tool to pack, unpack, check, and list the contents of Unreal Engine 4 packages. Under Linux it can also be used to read-only FUSE-mount archives. Note that only a limited number of pak versions are supported.

Table 1. Supported Versions

Pak Version

Unreal Engine Version

Reading

Writing

1

< 4.0

✔️

✔️

2

4.0 - 4.2

✔️

✔️

3

4.3 - 4.15

✔️

✔️

4

4.16 - 4.19

✔️

5

4.20

✔️

6

-

✔️

7

4.21

✔️

8

4.22 - 4.24

✔️

9

4.25

✔️

10

-

✔️

11

4.26 - 4.27

✔️

Table 2. Supported Compression

Name

Supported

None

✔️

Zlib

✔️

Gzip

LZ4

Instead of passing arguments you can also put the arguments in a file with the extension .u4pak and pass the path to that instead. This is useful for Windows users that aren’t used to a terminal. You can even associate the extension with u4pak.exe so that it will be automatically opened with it when you double click it. File paths in a .u4pak file are relative to the directory containing the file. The syntax of these files is not shell syntax. If you don’t have any white space, double quotes ("), or hash marks () in your file names you don’t have to worry about anything. is used to start a comment line (only if it doesn’t touch any non-white space on it’s left) and " is used to quote arguments containing white space, #, or ". In order to write a " in a quoted argument you simply need to double it, meaning an argument that contains nothing but a single " is written as """". Newlines are ignored like any other white space. An example .u4pak file whould be:

# This is packing my project:
pack
--version=4
--mount-point=../../..

"C:\Users\Alice\My Documents\U4Project\NewArchive.pak"

":rename=/:C:\Users\Alice\My Documents\U4Project\Some Files"
":zlib,rename=/:Some Other Files"

If u4pak.exe is run by double clicking or by dropping a .u4pak file on it it won’t immediately close the terminal window, but will instead ask you to press ENTER. It does this so you have a chance to read the output. Since I don’t use Windows (I cross compile on Linux and test with Wine) I could not test this particular feature. If it doesn’t work please report a bug. In order to force the "Press ENTER to continue…​" message you can pass the argument --pause-on-exit=always (Windows-only).

1. Usage

u4pak [--pause-on-exit=<always|never|auto>] [SUBCOMMAND]

Or:

u4pak "C:\Path\to\arguments.u4pak"

2. Sub-Commands

Sub-Command

Description

check

Check consistency of a package

help

Prints general help message or the help of the given subcommand(s)

info

Show summarized information of a package

list

List content of a package

mount

Mount package as read-only filesystem (Linux-only)

pack

Create a new package

unpack

Unpack content of a package

For help to the various sub-commands run u4pak help SUBCOMMAND.

3. File Format

Byte order is little endian and the character encoding of file names seems to be ASCII (or ISO-8859-1/UTF-8 that coincidentally only uses ASCII compatiple characters).

Offsets and sizes seem to be 64bit or at least unsigned 32bit integers. If interpreted as 32bit integers all sizes (except the size of file names) and offsets are followed by another 32bit integer of the value 0, which makes me guess these are 64bit values. Also some values exceed the range of signed 32bit integers, so they have to be at least unsigned 32bit integers.

This information was reverse engineered from the Elemental Demo for Linux (which contains a 2.5 GB .pak file), the Unreal Engine 4 - Five Tech Demos, version 7 was reverse engineered from the Supraland Demo, and the Conan Exiles variant was reverse egineered from the SandstormFix_EXP workshop item. Reverse engineering was done by poking around the pak files with a hex editor, no kind of decompilation was done by me.

Versions >= 8 where reversed from an empty unreal engine project.

Basic layout
  • Data Records

  • Index

    • Index Header

    • Index Records

  • Footer

In order to parse a file you need to read the footer first. The footer contains an offset pointer to the start of the index records. The index records then contain offset pointers to the data records.

3.1. Encoded Record

Offset  Size  Type         Description
     0     4  uint32_t     bitfield containing record information
                              0-5  : Compression block size
                              6-21 : Compression blocks count
                              22   : Encrypted
                              23-28: Compression method
                              29   : Size 32-bit safe?
                              30   : Uncompressed size 32-bit safe?
                              31   : Offset 32-bit safe?
if offset 32 bit
     4     4  uint32_t     offset
else
     4     8  uint64_t     offset
end
if uncompressed size 32 bit
     ?     4  uint32_t     uncompressed size
else
     ?     8  uint64_t     uncompressed size
end
if compression method != 0x00
  if size 32 bit
     ?     4  uint32_t     size
  else
     ?     4  uint32_t     size
  end
end
if compression block count > 0 && (encrypted || compression block count != 1)
  for _ in 0..compression block count
     ?     4  uint32_t     block size
  end
end

3.2. Record

Note
This structure, while still present in version >= 10 is not used anymore by default. See Encoded Record for the current record information.
Offset  Size  Type         Description
     0     8  uint64_t     offset
     8     8  uint64_t     size (N)
    16     8  uint64_t     uncompressed size
    24     4  uint32_t     compression method:
                              0x00 ... none
                              0x01 ... zlib
                              0x10 ... bias memory
                              0x20 ... bias speed
if version <= 1
    28     8  uint64_t     timestamp
end
     ?    20  uint8_t[20]  data sha1 hash
if version >= 3
 if compression_method != 0x00
  ?+20     4  uint32_t     block count (M)
  ?+24  M*16  CB[M]        compression blocks
 end
     ?     1  uint8_t      is encrypted
   ?+1     4  uint32_t     The uncompressed size of each compression block.
end                        The last block can be smaller, of course.
if variant == "Conan Exiles"
     ?     4  uint32_t     Unknown field. For Conan Exiles index record only
                           seen it to have the value 0.
end

3.3. Compression Block (CB)

Size

16 bytes

Offset  Size  Type         Description
     0     8  uint64_t     compressed data block start offset.
                           version <= 4: offset is absolute to the file
                           version 7: offset is relative to the offset
                                      field in the corresponding Record
     8     8  uint64_t     compressed data block end offset.
                           There may or may not be a gap between blocks.
                           version <= 4: offset is absolute to the file
                           version 7: offset is relative to the offset
                                      field in the corresponding Record

3.4. Data Record

Offset  Size  Type            Description
     0     ?  Record          file metadata (offset field is 0, N = compressed_size)
if variant == "Conan Exiles"
     ?    20  ?               Unknown. Maybe another SHA-1 sum of something?
                              The first 4 bytes have values other than the extra
                              4 bytes in the index record, which is why I didn't
                              put those into the general record structure.
else if version >= 4 and compression_method != 0x00
     ?     4  uint32_t        Unknown.
end
     ?     N  uint8_t[N]      file data
Note

Starting with version 4 there is an additional 4 bytes in the repeated data record copy (the record that precedes the actual file date, not the record in the index). I don’t know what that is. It is not always the same value. E.g. it is the same for some files, but different for others. The first 2 bytes seem to be always the same (0x78 0x9c in a v4 and a v7 pak that I saw), so maybe its 2 16 bit fields?

This is why I’ve deactivated packing for versions > 3.

3.5. Full Directory Index

The data structure is a map<DirectoryName, map<FileName, offset>>.

Offset  Size  Type            Description
     0     4  uint32_t        directory count (D)
for i in 0..D
     ?     4  int32_t         directory name size (DS)
                              For some games a negative value means it’s a UTF-16
                              string in 2 * -S bytes.
   ?+4    DS  char[DS]        directory name (includes terminating null byte)
?+4+DS     4  uint32_t        file count (F)
  for j in 0..F
     ?     4  int32_t         file name size (FS)
                              For some games a negative value means it’s a UTF-16
                              string in 2 * -S bytes.
   ?+4    FS  char[FS]        file name (includes terminating null byte)
?+4+FS     4  uint32_t        offset in encoded entry info
  end
end

3.6. Index Record

Note
This structure, while still present in version >= 10 is not used anymore by default. See Full Directory Index for record information.
Offset  Size  Type            Description
     0     4  int32_t         file name size (S)
                              For some games a negative value means it's a UTF-16
                              string in 2 * -S bytes.
     4     S  char[S]         file name (includes terminating null byte)
   4+S     ?  Record          file metadata
if variant == "Conan Exiles"
     ?     4  ?               Unknown. Only saw all 0 so far.
end

3.7. Index

Version >= 10
Offset  Size  Type            Description
     0     4  int32_t         mount point size (S)
                              For some games a negative value means it’s a UTF-16
                              string in 2 * -S bytes.
     4     S  char[S]         mount point (includes terminating null byte)
   S+4     4  int32_t         entry count
   S+8     8  uint64_t        path hash seed (1)
  S+16     4  uint32_t        has path hash index
if has path index != 0
  S+20     8  int64_t         path hash index offset
  S+28     8  int64_t         path hash index size
  S+36    20  uint8_t[20]     path hash index hash
end
     ?     4  uint32_t        has full directory index
if has full directory index != 0
   ?+4     8  int64_t         full directory index offset
  ?+12     8                  full directory index size
  ?+20    20  uint8_t[20]     full directory index hash
end
     ?     4  int32_t         encoded entry info size (P)
   ?+4     P  uint8_t[P]      encoded entry info 
   ?+P     4  uint32_t        file count, probably unused / 0 (N)
 ?+P+4     ?  IndexRecord[N]  records
  1. Needs clarification

Legacy (Version < 10)
Offset  Size  Type            Description
     0     4  int32_t         mount point size (S)
                              For some games a negative value means it's a UTF-16
                              string in 2 * -S bytes.
     4     S  char[S]         mount point (includes terminating null byte)
   S+4     4  uint32_t        record count (N)
   S+8     ?  IndexRecord[N]  records
Table 3. Size and index features

Versions

Size

Index Encryption

Encryption Key Guid

Compression method name

Frozen Index

v1 - v3

44 bytes

v4 - v6

45 bytes

✔️

v7

65 bytes

✔️

✔️

v8

193 bytes

✔️

✔️

✔️(Max. 4)

v9

226 bytes

✔️

✔️

✔️(Max. 5)

✔️

v10 - v11

225 bytes

✔️

✔️

✔️(Max. 5)

Offset  Size  Type         Description
if version >= 7
     0    20  uint8_t[20]  encryption key Guid
end
if version >= 4
     ?     1  uint8_t      encrypted index
end
     ?     4  uint32_t     magic: 0x5A6F12E1
   ?+4     4  uint32_t     version: 1 - 11
   ?+8     8  uint64_t     index offset
  ?+16     8  uint64_t     index size
  ?+24    20  uint8_t[20]  index sha1 hash
if version == 9
  ?+44     1  uint8_t      frozen index
end
if version == 8
  ?+44   128  uint8_t[128] compression methods (4 * 32 chars)
else if version > 8
     ?   160  uint8_t[160] compression methods (5 * 32 chars)
end
  • fezpak: pack, unpack, list and mount FEZ .pak archives

  • psypkg: pack, unpack, list and mount Psychonauts .pkg archives

  • bgebf: unpack, list and mount Beyond Good and Evil .bf archives

  • unvpk: extract, list, check and mount Valve .vpk archives (C++)

  • rust-vpk: Rust rewrite of the above (Rust)

  • t2fbq: unpack, list and mount Trine 2 .fbq archives

  • u4pak: old Python version of this program

5. MPLv2 License

Rust U4Pak - pack, unpack, check, list and mount Unreal Engine 4 packages

Copyright © 2024 Mathias Panzenböck, L. Sprengel

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.