README.md contains non-functional code

[ahenkes1]

The README.md (and the corresponding landing page of the documentation) contains non-functional code. Specifically the following:

Loading a pretrained tokenizer from the Hub

use tokenizers::tokenizer::{Result, Tokenizer};

fn main() -> Result<()> {
    # #[cfg(feature = "http")]
    # {
        let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None)?;

        let encoding = tokenizer.encode("Hey there!", false)?;
        println!("{:?}", encoding.get_tokens());
    # }
    Ok(())
}

Here, the function to load the tokenizer from a pretrained model is not available in 'Tokenizer':

error[E0599]: no function or associated item named `from_pretrained` found for struct `Tokenizer` in the current scope
   --> src/main.rs:30:32
    |
30  |     let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None);
    |                                ^^^^^^^^^^^^^^^ function or associated item not found in `Tokenizer`

[ArthurZucker]

Hey! You have to enbale the https features for that 🤗 Should we clarify the doc about this?

[ahenkes1]

Hey! That makes sense! A small hint wouldn't hurt ;)