adding per field tokenizer

[saliksyed]

Added a per field tokenizer and changed the lunr variable to be explicitly set for window. (This prevents errors when running in strict mode)

[olivernn]

How will this affect using lunr as an npm module? Might need a check around whether window exists before assigning lunr to it.

[olivernn]

What is the use case for having individual tokenizers per field?

How does this affect searching too? Search queries are passed through the main lunr.tokenizer rather than the individual field tokenizers, would this not cause problems if the per field tokenizers are different? It'd be good to see some tests covering this specifically as well as how the per field tokenizers work.

I like the change for explicitly assigning lunr to window if it prevents warnings in strict mode, as I mentioned in the line comment it would be good to do a check so lunr can still work in environments where the global object is not window (node.js). I do think this should be a separate change though so if you could split this into two pull requests that'd be great.

[saliksyed]

Okay -- I can split into two pull requests and I see why you would want to
check for window before assigning. The use case is that we have some fields
that have strange formats (tokens seperated by "." notation) and want to
use custom logic to split the strings before then passing results to
default lunr tokenizer.

On Tue, Aug 20, 2013 at 9:27 AM, Oliver Nightingale <
[email protected]> wrote:

What is the use case for having individual tokenizers per field?

How does this affect searching too? Search queries are passed through the
main lunr.tokenizer rather than the individual field tokenizers, would
this not cause problems if the per field tokenizers are different? It'd be
good to see some tests covering this specifically as well as how the per
field tokenizers work.

I like the change for explicitly assigning lunr to window if it prevents
warnings in strict mode, as I mentioned in the line comment it would be
good to do a check so lunr can still work in environments where the global
object is not window (node.js). I do think this should be a separate
change though so if you could split this into two pull requests that'd be
great.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/46#issuecomment-22957841
.