Skip to content

Commit

Permalink
avoid java doc change
Browse files Browse the repository at this point in the history
Signed-off-by: xinyual <[email protected]>
  • Loading branch information
xinyual committed Mar 11, 2024
1 parent eb4f36b commit 2f8557a
Showing 1 changed file with 5 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,14 @@ public FixedTokenLengthChunker(Map<String, Object> parameters) {
}

/**
* Validate the chunked passages for fixed token length algorithm,
* Validate and parse the parameters for fixed token length algorithm,
* will throw IllegalArgumentException when parameters are invalid
*
* @param parameters a map containing parameters, containing the following parameters:
* 1. tokenizer the analyzer tokenizer in opensearch, please check https://opensearch.org/docs/latest/analyzers/tokenizers/index/
* 2. token_limit the token limit for each chunked passage
* 3. overlap_rate the overlapping degree for each chunked passage, indicating how many token comes from the previous passage
* 4. max_token_count the max token limit for the tokenizer
* 1. tokenizer: the <a href="https://opensearch.org/docs/latest/analyzers/tokenizers/index/">analyzer tokenizer</a> in opensearch
* 2. token_limit: the token limit for each chunked passage
* 3. overlap_rate: the overlapping degree for each chunked passage, indicating how many token comes from the previous passage
* 4. max_token_count: the max token limit for the tokenizer
* Here are requirements for parameters:
* max_token_count and token_limit should be a positive integer
* overlap_rate should be within range [0, 0.5]
Expand Down

0 comments on commit 2f8557a

Please sign in to comment.