Klaijan/xlsx sub tables #1585

Klaijan · 2023-09-29T21:38:41Z

Executive Summary
Unstructured is now able to capture subtables, along with other text element types within the .xlsx sheet.

Technical Details

The function now reads the excel without header as default
Leverages the connected components search to find subtables within the sheet. This search is based on dfs search
It also handle the overlapping table or text cases
Row with only single cell of data is considered not a table, and therefore passed on the determine the element type as text
In connected elements, it is possible to have table title, header, or footer. We run the count for the first non-single empty rows from top and bottom to determine those text

Result
This table now reads as:

[
    {
        "type": "Title",
        "element_id": "3315afd97f7f2ebcd450e7c939878429",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "3315afd97f7f2ebcd450e7c939878429",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Quarterly revenue</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>Group financial performance</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>Segmental results</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>Segmental analysis</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <td>Cash flow</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>5</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "Financial performance"
    },
    {
        "type": "Table",
        "element_id": "17f5d512705be6f8812e5dbb801ba727",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "3315afd97f7f2ebcd450e7c939878429",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Quarterly revenue</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>Group financial performance</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>Segmental results</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>Segmental analysis</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>4</td>\n    </tr>\n    <tr>\n      <td>Cash flow</td>\n      <td>FY 22</td>\n      <td>FY 23</td>\n      <td></td>\n      <td>5</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nQuarterly revenue\nNine quarters to 30 June 2023\n\n\n1\n\n\nGroup financial performance\nFY 22\nFY 23\n\n2\n\n\nSegmental results\nFY 22\nFY 23\n\n3\n\n\nSegmental analysis\nFY 22\nFY 23\n\n4\n\n\nCash flow\nFY 22\nFY 23\n\n5\n\n\n"
    },
    {
        "type": "Title",
        "element_id": "8a9db7161a02b427f8fda883656036e1",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "8a9db7161a02b427f8fda883656036e1",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Mobile customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <td>Fixed broadband customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>7</td>\n    </tr>\n    <tr>\n      <td>Marketable homes passed</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>TV customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>9</td>\n    </tr>\n    <tr>\n      <td>Converged customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <td>Mobile churn</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>Mobile data usage</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>Mobile ARPU</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>13</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "Operational metrics"
    },
    {
        "type": "Table",
        "element_id": "d5d16f7bf9c7950cd45fae06e12e5847",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "8a9db7161a02b427f8fda883656036e1",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Mobile customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>6</td>\n    </tr>\n    <tr>\n      <td>Fixed broadband customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>7</td>\n    </tr>\n    <tr>\n      <td>Marketable homes passed</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>TV customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>9</td>\n    </tr>\n    <tr>\n      <td>Converged customers</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <td>Mobile churn</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>Mobile data usage</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>Mobile ARPU</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>13</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nMobile customers\nNine quarters to 30 June 2023\n\n\n6\n\n\nFixed broadband customers\nNine quarters to 30 June 2023\n\n\n7\n\n\nMarketable homes passed\nNine quarters to 30 June 2023\n\n\n8\n\n\nTV customers\nNine quarters to 30 June 2023\n\n\n9\n\n\nConverged customers\nNine quarters to 30 June 2023\n\n\n10\n\n\nMobile churn\nNine quarters to 30 June 2023\n\n\n11\n\n\nMobile data usage\nNine quarters to 30 June 2023\n\n\n12\n\n\nMobile ARPU\nNine quarters to 30 June 2023\n\n\n13\n\n\n"
    },
    {
        "type": "Title",
        "element_id": "f97e9da0e3b879f0a9df979ae260a5f7",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "f97e9da0e3b879f0a9df979ae260a5f7",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Average foreign exchange rates</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>14</td>\n    </tr>\n    <tr>\n      <td>Guidance rates</td>\n      <td>FY 23/24</td>\n      <td></td>\n      <td></td>\n      <td>14</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "Other"
    },
    {
        "type": "Table",
        "element_id": "080e1a745a2a3f2df22b6a08d33d59bb",
        "metadata": {
            "filename": "vodafone.xlsx",
            "file_directory": "example-docs",
            "last_modified": "2023-10-03T17:51:34",
            "filetype": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
            "parent_id": "f97e9da0e3b879f0a9df979ae260a5f7",
            "languages": [
                "spa",
                "ita"
            ],
            "page_number": 1,
            "page_name": "Index",
            "text_as_html": "<table border=\"1\" class=\"dataframe\">\n  <tbody>\n    <tr>\n      <td>Topic</td>\n      <td>Period</td>\n      <td></td>\n      <td></td>\n      <td>Page</td>\n    </tr>\n    <tr>\n      <td>Average foreign exchange rates</td>\n      <td>Nine quarters to 30 June 2023</td>\n      <td></td>\n      <td></td>\n      <td>14</td>\n    </tr>\n    <tr>\n      <td>Guidance rates</td>\n      <td>FY 23/24</td>\n      <td></td>\n      <td></td>\n      <td>14</td>\n    </tr>\n  </tbody>\n</table>"
        },
        "text": "\n\n\nTopic\nPeriod\n\n\nPage\n\n\nAverage foreign exchange rates\nNine quarters to 30 June 2023\n\n\n14\n\n\nGuidance rates\nFY 23/24\n\n\n14\n\n\n"
    }
]

-ingest order

This pull request includes updated ingest test fixtures. Please review and merge if appropriate. Co-authored-by: Klaijan <[email protected]>

cragwolfe · 2023-10-03T17:38:20Z

also adds languages param -- worth calling out in the description?

cragwolfe

Please add a few .xlsx test docs to test different layout patterns.

…ured-IO/unstructured into klaijan/xlsx-sub-tables

This pull request includes updated ingest test fixtures. Please review and merge if appropriate. Co-authored-by: Klaijan <[email protected]>

test_unstructured_ingest/test-ingest.sh

cragwolfe

LGTM pending nit on the test-ingest.sh.
🎉

Klaijan added 7 commits September 28, 2023 17:58

find subtables

a948837

fix bug

2cdbbce

add EXPECTED_TITLE to test cases

c9d65c5

big fixed

1c6f199

Merge branch 'main' into klaijan/xlsx-sub-tables

91c2ac7

update changelog

c54a9f5

update changelog

907b689

Klaijan marked this pull request as draft September 29, 2023 21:57

Klaijan added 2 commits September 29, 2023 19:07

add language detection

e14c9ac

add language test

109f196

Klaijan marked this pull request as ready for review September 29, 2023 23:29

Klaijan and others added 16 commits October 1, 2023 15:38

make pip-compile

de5d710

attempt to fix elastic search error in test fixture

0fb799c

request timeout 60s

9419e9e

request timeout 30s

3ee486f

Merge branch 'main' into klaijan/xlsx-sub-tables

e3979ce

reorder es test

d8a51fb

-ingest order

Merge branch 'main' into klaijan/xlsx-sub-tables

0efd7fe

Merge branch 'main' into klaijan/xlsx-sub-tables

3bb3c29

Merge branch 'main' into klaijan/xlsx-sub-tables

68d6db2

Merge branch 'main' into klaijan/xlsx-sub-tables

27e499e

Update CHANGELOG.md

5c742fa

Klaijan/xlsx sub tables <- Ingest test fixtures update (#1626)

c7bf892

This pull request includes updated ingest test fixtures. Please review and merge if appropriate. Co-authored-by: Klaijan <[email protected]>

Update create_and_fill_es.py

f4aa793

Update xlsx.py

476b3cf

Merge branch 'main' into klaijan/xlsx-sub-tables

f9e3bb0

Update test-ingest.sh

224ed98

Klaijan requested a review from cragwolfe October 3, 2023 16:36

add languages params description in CHANGELOG

17c2ee2

Klaijan added 3 commits October 3, 2023 14:07

Merge branch 'main' into klaijan/xlsx-sub-tables

a5f5a0b

linting

833c0fc

Merge branch 'main' into klaijan/xlsx-sub-tables

a1b9be6

cragwolfe requested changes Oct 3, 2023

View reviewed changes

Klaijan and others added 12 commits October 3, 2023 20:13

add test. edit filter function. add _get_metadata for cleanliness

4bf6805

linting

e4ae06c

Merge branch 'klaijan/xlsx-sub-tables' of https://github.com/Unstruct…

e84ab88

…ured-IO/unstructured into klaijan/xlsx-sub-tables

add example doc

69a73e9

Merge branch 'main' into klaijan/xlsx-sub-tables

efcfa51

update test_auto

6b9d97f

Merge branch 'klaijan/xlsx-sub-tables' of https://github.com/Unstruct…

c027f20

…ured-IO/unstructured into klaijan/xlsx-sub-tables

add test and example docs

3a0a210

Klaijan/xlsx sub tables <- Ingest test fixtures update (#1638)

3f75489

This pull request includes updated ingest test fixtures. Please review and merge if appropriate. Co-authored-by: Klaijan <[email protected]>

Merge branch 'main' into klaijan/xlsx-sub-tables

4187edd

edit test

ae52401

Klaijan/xlsx sub tables <- Ingest test fixtures update (#1639)

4a942ca

This pull request includes updated ingest test fixtures. Please review and merge if appropriate. Co-authored-by: Klaijan <[email protected]>

Klaijan requested a review from cragwolfe October 4, 2023 14:27

cragwolfe reviewed Oct 4, 2023

View reviewed changes

test_unstructured_ingest/test-ingest.sh Outdated Show resolved Hide resolved

cragwolfe approved these changes Oct 4, 2023

View reviewed changes

Update test-ingest.sh

451cfab

Klaijan merged commit 0a65fc2 into main Oct 4, 2023
39 checks passed

Klaijan deleted the klaijan/xlsx-sub-tables branch October 4, 2023 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Klaijan/xlsx sub tables #1585

Klaijan/xlsx sub tables #1585

Klaijan commented Sep 29, 2023 •

edited

Loading

cragwolfe commented Oct 3, 2023

cragwolfe left a comment

cragwolfe left a comment

Klaijan/xlsx sub tables #1585

Klaijan/xlsx sub tables #1585

Conversation

Klaijan commented Sep 29, 2023 • edited Loading

cragwolfe commented Oct 3, 2023

cragwolfe left a comment

Choose a reason for hiding this comment

cragwolfe left a comment

Choose a reason for hiding this comment

Klaijan commented Sep 29, 2023 •

edited

Loading