Extra space within doc.text method

[rulai-huajunzeng]

An unexpected space was added within the doc.text.
code as below:

>>> nlp=spacy.load('en', add_vectors=False)
>>> nlp(u"[email protected]\nThank you!").text
u'[email protected] \nThank you!'

Note that there is a space before \n which was not there in the input. This caused incorrect position of tokens when I tried to map to original text

Your Environment

• Operating System: Ubuntu
• Python Version Used: Python 2.7.12 :: Anaconda 4.1.1 (64-bit)
• spaCy Version Used: 1.6.0
• Environment Information:

[honnibal]

Thanks for the report -- bad bug. Will try to get this fixed shortly. In the meantime I think the error relates to the recent URL regex changes.

[oroszgy]

Recently \n keeps being a token which is a bit strange for me.
See this:

>>>repr(nlp(u"Hi\nThank you!")[1])
'\n'

[honnibal]

@oroszgy Non-trivial whitespace has always been a token in spaCy -- I figured you'd have noticed this before. Is something new here that I'm not understanding?

[oroszgy]

Hm, interesting, somehow I did not noticed this behaviour before. I guess my previous comment is useless in this case.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.