Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Differential cmap compression #4790

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified external/bcmaps/78-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78ms-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/78ms-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/83pv-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90ms-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90ms-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90msp-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90msp-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90pv-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/90pv-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Add-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Add-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Add-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Add-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-0.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-1.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-3.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-4.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-5.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-6.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-CNS1-UCS2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-0.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-1.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-3.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-4.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-5.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-GB1-UCS2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-0.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-1.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-3.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-4.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-5.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-6.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Japan1-UCS2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Korea1-0.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Korea1-1.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Korea1-2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Adobe-Korea1-UCS2.bcmap
Binary file not shown.
Binary file modified external/bcmaps/B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/B5pc-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/B5pc-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS1-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS1-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS2-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/CNS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETHK-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETHK-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETen-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETen-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETenms-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/ETenms-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Ext-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Ext-RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Ext-RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Ext-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GB-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GB-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GB-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GB-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBK-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBK-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBK2K-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBK2K-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBKp-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBKp-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBT-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBT-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBT-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBT-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBTpc-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBTpc-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBpc-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/GBpc-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKdla-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKdla-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKdlb-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKdlb-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKgccs-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKgccs-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKm314-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKm314-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKm471-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKm471-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKscs-B5-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/HKscs-B5-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Hankaku.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Hiragana.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-Johab-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-Johab-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCms-UHC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCms-UHC-HW-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCms-UHC-HW-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCms-UHC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCpc-EUC-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/KSCpc-EUC-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Katakana.bcmap
Binary file not shown.
Binary file modified external/bcmaps/NWP-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/NWP-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/RKSJ-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/RKSJ-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/Roman.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UCS2-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UCS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF16-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF16-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF8-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniCNS-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UCS2-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UCS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF16-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF16-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF8-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniGB-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UCS2-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UCS2-HW-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UCS2-HW-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UCS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF16-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF16-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF8-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF16-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF16-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF8-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJIS2004-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISPro-UCS2-HW-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISPro-UCS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISPro-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISX0213-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISX0213-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISX02132004-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniJISX02132004-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UCS2-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UCS2-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF16-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF16-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF32-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF32-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF8-H.bcmap
Binary file not shown.
Binary file modified external/bcmaps/UniKS-UTF8-V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/V.bcmap
Binary file not shown.
Binary file modified external/bcmaps/WP-Symbol.bcmap
Binary file not shown.
1 change: 1 addition & 0 deletions external/bcmaps_temp/savings.json

Large diffs are not rendered by default.

47 changes: 46 additions & 1 deletion external/cmapscompress/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Quick notes about binary CMap format (bcmap)

The format is designed to package some information from the CMap files located at external/cmap. Please notice for size optimization reasons, the original information blocks can be changed (split or joined) and items in the blocks can be swaped.
The format is designed to package some information from the CMap files located at external/cmap. Please notice for size optimization reasons, the original information blocks can be changed (split or joined) and items in the blocks can be swapped.

The data stored in binary format in network byte order (big-endian).

Expand All @@ -15,6 +15,51 @@ The following primitives used during encoding of the file:
- signed fixed number (SB[n]) – similar to the SN, but it represents a signed number that is stored in B[n]
- string (S) – the string is encoded as sequence of bytes. First comes length is characters encoded as UN, when UTF16 characters encoded as UN.

# Differential compression

The contents of each CMap file is either stored normally or differentially. In the latter case, a second CMap file (the 'base file') is needed for file decoding.

The first record in each file indicates if the file is stored normally or differentially.
It is a string (S) – let's call it *baseFileName* – which
- is empty ('') if the file is stored normally, or
- contains the file name of the base file (without path or extension) if it is stored differentially.

In either case, it is followed by the (possibly decoded) file contents which are structured as described in the [file structure](#file-structure) section.

### Decoding differential data

If a CMap file (let's name it *A*) is stored differentially, file contents are to be constructed from the contents of *A* and from the base file (which we shall call *B*).
The records to follow are alternately of the following type, starting with *copy*.

A **copy**-type instruction specified by
- startDelta as UN
- length as UN

which instructs to read *length* bytes from *B*, where startDelta specifies the start position as an offset from the previously used array end. (The previous array end is initialized with the position of the start of content, i.e., after *baseFileName* and *contentSize* in *B*).

An **insert**-type instruction is specified by
- length as UN

and instructs to read append the following *length* bytes from *A* and append it to the contents.

It may happen that file *B* itself is stored differentially and depends on a further file. In this case, *B* has to be restored before restoring *A*. The following pseudocode accomplishes the decoding
```
var contents = '';
var previousEnd = 0; // position after *baseFileName* in baseFile
for (var copy = true; contents.length < contentSize; copy = !copy) {
if (copy) {
var start = previousEnd + A.readUN();
var length = A.readUN();
contents.append(B.subarray(start, start + length));
previousEnd = start + length;
} else {
var length = A.readUN();
contents.append(A.readBytes(length));
}
}
```

<a name="file-structure"></a>
# File structure

The first byte is a header:
Expand Down
Loading