~~Nesting Sentences Support~~ → Add Context for nested sentence

[azu]

He said "This is a pen. That is not a pen.".

Currently, sentence-splitter parse the text as follows:

We want to support nesting sentences.

• Sentence A
  • Sentence B
  • Sentence C

PairMarker is related
https://github.com/azu/sentence-splitter/blob/master/src/parser/PairMaker.ts

AST Design

• It is not perfect

[
  {
    "type": "Sentence",
    "raw": "He said 「This is a pen. That is not a pen」.",
    "children": [
      {
        "type": "Str",
        "raw": "He said ",
        "value": "He said "
      },
      {
        "type": "PairMark",
        "pairType": "start",
        "raw": "「 ",
        "value": "「 "
      },
      {
        "type": "Sentence",
        "raw": "This is a pen.",
        "children": [
          {
            "type": "Str",
            "raw": "This is a pen.",
            "value": "This is a pen. "
          }
        ]
      },
      {
        "type": "Sentence",
        "raw": "That is not a pen.",
        "children": [
          {
            "type": "Str",
            "raw": "That is not a pen",
            "value": "That is not a pen"
          }
        ]
      },
      {
        "type": "PairMark",
        "pairType": "end",
        "raw": "」",
        "value": "」 "
      },
      {
        "type": "Punctuation",
        "raw": ".",
        "value": "."
      }
    ]
}

Related

• 括弧で囲まれた文章が一文として判定される textlint-rule-sentence-length#18
  • UseCase: Count Sentence Length
  • It is different type from others
  • Probably, We can not resolve it in this library
• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-max-ten#17
  • UseCase: Count 、 in a sentence
• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-no-doubled-conjunctive-particle-ga#19
  • UseCase: Count が in a sentence
• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-no-doubled-joshi#40
  • UseCase: Count 助詞 in a sentence

[azu]

I noticed that some case is difficult.

彼は「コレ」と読んだ

I think that 「コレ」 is not a sentence.
sentence-splitter can not detect it.

[azu]

It should be opt-in feature.

We can not detect which is better.

[azu]

• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-no-doubled-joshi#40
  • UseCase: Count 助詞 in a sentence
• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-no-doubled-conjunctive-particle-ga#19
  • UseCase: Count が in a sentence
• 括弧で囲まれた文章が一文として判定される textlint-ja/textlint-rule-max-ten#17

Probably, These are rule implemetation bug. Not this library

[azu]

We are talking about pens.
He said "This is a pen. I like it".
I could relate to that statement.

Current parser parse it following sentences.

Second sentence has "This is a pen. I like it", but we can not split it into new sentence.

The conversation text is just Str node.
HTML does not have suitable semantics for conversation.

• html - Most semantic way to markup a conversation (or interview)? - Stack Overflow

As a result, sentence-splitter can not support nesting sentence.
Probably, rule implementation should handble the quote text after parsing sentences by sentence-splitter.

We will close this issue by adding current behavior.