MAINT: Text extraction improvements

[MartinThoma]

Credits to pubpub-zz, see
#1118 (comment)

Co-authored-by: pubpub-zz [email protected]

[codecov]

Codecov Report

Merging #1126 (40925ed) into main (0b693e1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1126   +/-   ##
=======================================
  Coverage   92.02%   92.02%           
=======================================
  Files          24       24           
  Lines        4667     4667           
  Branches      964      964           
=======================================
  Hits         4295     4295           
  Misses        227      227           
  Partials      145      145           

Impacted Files	Coverage Δ
PyPDF2/_page.py	92.60% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b693e1...40925ed. Read the comment docs.

[dkg]

Note that the modification to parse_to_unicode here attempts to clean up some part of ae0ff49, but doesn't seem to account for the earlier modification in that commit, where the null dstString was mapped to .. If you are going to go with this approach, you should avoid mapping the null dstString to . as well (that is, revert the first hunk of ae0ff49).

But see also my comment over on #1118 about why i think this approach is less correct than the approach that you've already merged.

[dkg]

I've opened py-pdf/sample-files#13 to put habibi.pdf in the sample-files repo. i recommend including a test for it before merging this.

[MartinThoma]

With

        elif process_char:
            lst = [x for x in l.split(b" ") if x]
            map_dict[-1] = len(lst[0]) // 2
            if len(lst) == 1:       # some case where the 2nd param is empty (seems not IAW pdfspec)
                map_dict[
                    unhexlify(lst[0]).decode(
                        "charmap" if map_dict[-1] == 1 else "utf-16-be", "surrogatepass"
                    )
                ] = ""
            else:
                while len(lst) > 0:
                    map_dict[
                        unhexlify(lst[0]).decode(
                            "charmap" if map_dict[-1] == 1 else "utf-16-be", "surrogatepass"
                        )
                    ] = unhexlify(lst[1]).decode(
                        "utf-16-be", "surrogatepass"
                    )  # join is here as some cases where the code was split
                    int_entry.append(int(lst[0], 16))
                    lst = lst[2:]

I get

>                       ] = unhexlify(lst[1]).decode(
                            "utf-16-be", "surrogatepass"
                        )  # join is here as some cases where the code was split
E                       binascii.Error: Odd-length string