Feat/chipper v2

[benjats07]

This PR adds chipper v2 to the library, leveraging inference speedup and support for hierarchy (assign a parent element to the elements)

The main library changes are

• New model, chipperv2 is available. There are significant changes that necessitate code changes in pre and post-processing beyond just changes to the weights.
• a schema change to the TextRegion object, which now has a bbox property, rather than being a subclass of Rectangle
• LayoutElement now has a parent property that can be populated with another LayoutElement to represent inferred hierarchy.
• LocationlessLayoutElement is removed. Elements produced by Chipper now have bounding boxes, and if we need to handle elements without bounding boxes in the future, we can make the bbox property optional.

Note that because of the schema changes, ingest tests aren't working properly due to the incompatibility with the current unstructured version.

In order to test this new model you can use

from unstructured_inference.constants import OCRMode
from unstructured_inference.inference import layout
from unstructured_inference.models.base import get_model

file = "sample-docs/layout-parser-paper-fast.pdf"
model = get_model("chipperv2")
doc = layout.DocumentLayout.from_file(
    file,
    model,
    ocr_mode=OCRMode.FULL_PAGE.value,
    supplement_with_ocr_elements=True,
    ocr_strategy="never",
)

print(doc)

Also, you can try with an image:

file = "sample-docs/example_table.jpg"
model = get_model("chipperv2")
doc = layout.DocumentLayout.from_image_file(
    file,
    model,
    ocr_mode=OCRMode.FULL_PAGE.value,
    supplement_with_ocr_elements=True,
    ocr_strategy="never",
)

print(doc)

[LaverdeS]

I haven't looked though the code but I threw some 15 pdf samples (mostly part of sample-docs and ingestion) using chipperv2 to see if the hierarchical relationships were present and could not find any.

The import from unstructured_inference.constants import OCRMode suggested in the PR description throws a ImportError: cannot import name 'OCRMode' from 'unstructured_inference.constants', so the snippet I used is the one suggested by @ajjimeno in his last comment.

I acknowledge the presence of List elements and the parent attribute of the LayoutElements but its value is always None, even when is certain that a hierarchy is present.

From the 15 pdf samples, 94 elements were detected in total but no List-item was present, which is also odd, considering chipperv2 returns this category in the original XML output. I suspect this is related to the token2json transform from DonutProcessor @ajjimeno. This adds to the lack of parent attribute diff than None, a makes most of the returned documents to miss a considerable amount of the pdf content.

I tested more on a sample russian.pdf.

Some observations:

This is the output of chipperv2 in XML using it directly from HF:

This shows chipperv2 successfully parses the document returning List and List-item elements in a hierarchical structure.

Now, this is the output of chipperv2 being called through unstructured-inference=='0.6.1':

All the content is present, but there is no chance for hierarchical info to fit in the element class data and the List elements are ignored.

With the last changes introduced in this PR, the output is:

Which is missing a lot of the content (all from which were Listor List-items in the XML), and has a suspicious Image element from 'source': <Source.PDFMINER: 'pdfminer'>.

I am testing to plot to returned element coords but I have not clear what is the reference image for them. Are the coords returned considering the resolution needed for chipper when doing pdf2image? @ajjimeno.

[badGarnet]

I see many requests for constants. Could you provide an example of what is required in those requests? Is it a nice to have or a compulsory requirement? Thanks!

So here is an example of using a settings constant to represent this parameters: 0871489 credit to @qued
There are a few reasons:

• some values like "<s_" are used in multiple places so it is easier to maintain and less error prune if we define it once and use the variable name instead of typing the actual value
• some values may be a specific value that is really part of a configuration of the model like rank=128. Define it explicitly makes the code easier to read later on (so it is more clear on what knobs are present in the model and what values they are; otherwise someone would have to read through the code to find all those places and values)
• technically they are not required to make the model run but they would make maintenance much easier down the road so I guess you can call them nice to have. Hence I used question mark as a suggestion instead of requiring change on those.