-
Notifications
You must be signed in to change notification settings - Fork 809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: Table chunking #1540
Merged
Merged
chore: Table chunking #1540
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amanda103
force-pushed
the
amanda/chunk-by-chars-table
branch
from
September 27, 2023 20:38
bd3a13b
to
7e76610
Compare
badGarnet
reviewed
Sep 27, 2023
badGarnet
reviewed
Sep 27, 2023
badGarnet
reviewed
Sep 27, 2023
amanda103
force-pushed
the
amanda/chunk-by-chars-table
branch
2 times, most recently
from
September 29, 2023 00:46
c70ba59
to
cb4cb42
Compare
cragwolfe
reviewed
Sep 29, 2023
newelh
suggested changes
Sep 29, 2023
amanda103
force-pushed
the
amanda/chunk-by-chars-table
branch
from
September 29, 2023 21:49
6544681
to
229199e
Compare
newelh
approved these changes
Sep 30, 2023
badGarnet
reviewed
Oct 2, 2023
combine_text_under_n_chars=5, | ||
) | ||
elements = partition_docx(filename) | ||
chunks = chunk_by_title(elements, max_characters=9, combine_text_under_n_chars=5) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we also have assert to check for this example that
- without combining text under n chars set to 5 there are tiny chunks
- with the setting we have text with minimum 5 chars
badGarnet
reviewed
Oct 2, 2023
badGarnet
reviewed
Oct 2, 2023
Co-authored-by: Yao You <[email protected]>
amanda103
force-pushed
the
amanda/chunk-by-chars-table
branch
from
October 3, 2023 00:37
f93de18
to
ac2e78a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change is adding to our
add_chunking_strategy
logic so that we are able to chunk Table elements'text
andtext_as_html
params. In order to keep the functionality under the sameby_title
chunking strategy we have renamed thecombine_under_n_chars
tomax_characters
. It functions the same way for the combining elements under Title's, as well as specifying a chunk size (in chars) for TableChunk elements.*renaming the variable to
max_characters
will also reflect the 'hard max' we will implement for large elements in followup PRsAdditionally -> some lint changes snuck in when I ran
make tidy
hence the minor changes in unrelated files :)TODO:
✅ add unit tests
--> note: added where I could to unit tests! Some unit tests I just clarified that the chunking strategy was now 'by_title' because we don't have a file example that has Table elements to test the 'by_num_characters' chunking strategy
✅ update changelog
To manually test: