TUTORIAL_1_BASICS.md

Tutorial 1: NLP Base Types

This is part 1 of the tutorial, in which we look into some of the base types used in this library.

Creating a Sentence

There are two types of objects that are central to this library, namely the Sentence and Token objects. A Sentence holds a textual sentence and is essentially a list of Token.

Let's start by making a Sentence object for an example sentence.

# The sentence objects holds a sentence that we may want to embed or tag
from flair.data import Sentence

# Make a sentence object by passing a whitespace tokenized string
sentence = Sentence('The grass is green .')

# Print the object to see what's in there
print(sentence)

This should print:

Sentence: "The grass is green ." - 5 Tokens

The print-out tells us that the sentence consists of 5 tokens. You can access the tokens of a sentence via their token id or with their index:

# using the token id
print(sentence.get_token(4))
# using the index itself
print(sentence[3])

which should print in both cases

Token: 4 green

This print-out includes the token id (4) and the lexical value of the token ("green"). You can also iterate over all tokens in a sentence.

for token in sentence:
    print(token)

This should print:

Token: 1 The
Token: 2 grass
Token: 3 is
Token: 4 green
Token: 5 .

Tokenization

In some use cases, you might not have your text already tokenized. For this case, we added a simple tokenizer using the lightweight segtok library.

If you want to use this tokenizer, simply set the use_tokenizer flag when instantiating your Sentence with an untokenized string:

from flair.data import Sentence

# Make a sentence object by passing an untokenized string and the 'use_tokenizer' flag
sentence = Sentence('The grass is green.', use_tokenizer=True)

# Print the object to see what's in there
print(sentence)

Adding Custom Tokenizers

You can also pass custom tokenizers to the initialization method. Instead of passing a boolean True value to the use_tokenizer parameter, you can pass a tokenization method, like this:

from flair.data import Sentence, segtok_tokenizer


# Make a sentence object by passing an untokenized string and a tokenizer
sentence = Sentence('The grass is green.', use_tokenizer=segtok_tokenizer)

# Print the object to see what's in there
print(sentence)

This should print:

Sentence: "The grass is green ." - 5 Tokens

The second way allows you to write your own wrapper around the tokenizer you want to use. The wrapper is a function which has the same signature as flair.data.segtok_tokenizer (take a string and return List[Token]). Check the code of flair.data.space_tokenizer (which is very simple) to have an idea of how to implement such wrapper.

Adding Tags to Tokens

A Token has fields for linguistic annotation, such as lemmas, part-of-speech tags or named entity tags. You can add a tag by specifying the tag type and the tag value. In this example, we're adding an NER tag of type 'color' to the word 'green'. This means that we've tagged this word as an entity of type color.

# add a tag to a word in the sentence
sentence[3].add_tag('ner', 'color')

# print the sentence with all tags of this type
print(sentence.to_tagged_string())

This should print:

The grass is green <color> .

Each tag is of class Label which next to the value has a score indicating confidence. Print like this:

# get token 3 in the sentence 
token = sentence[3]

# get the 'ner' tag of the token
tag = token.get_tag('ner')

# print token
print(f'"{token}" is tagged as "{tag.value}" with confidence score "{tag.score}"')

This should print:

"Token: 4 green" is tagged as "color" with confidence score "1.0"

Our color tag has a score of 1.0 since we manually added it. If a tag is predicted by our sequence labeler, the score value will indicate classifier confidence.

Adding Labels to Sentences

A Sentence can have one or multiple labels that can for example be used in text classification tasks. For instance, the example below shows how we add the label 'sports' to a sentence, thereby labeling it as belonging to the sports category.

sentence = Sentence('France is the current world cup winner.')

# add a label to a sentence
sentence.add_label('sports')

# a sentence can also belong to multiple classes
sentence.add_labels(['sports', 'world cup'])

# you can also set the labels while initializing the sentence
sentence = Sentence('France is the current world cup winner.', labels=['sports', 'world cup'])

Labels are also of the Label class. So, you can print a sentence's labels like this:

sentence = Sentence('France is the current world cup winner.', labels=['sports', 'world cup'])

print(sentence)
for label in sentence.labels:
    print(label)

This should print:

sports (1.0)
world cup (1.0)

This indicates that the sentence belongs to these two classes, each with confidence score 1.0.

Next

Now, let us look at how to use pre-trained models to tag your text.