v1.7.0: New 50 MB model, CLI, better downloads and lots of bug fixes

✨ Major features and improvements

• NEW: Improved English model.
• NEW: Additional smaller English model (50 MB, only 2% less accurate than larger model).
• NEW: Command line interface to download and link models, view debugging info and print Markdown info for easy copy-pasting to GitHub.
• NEW: Alpha support for Finnish and Bengali.
• Updated language data for Swedish and French.
• Simplified import of lemmatizer data to make it easier to add lemmatization for other languages.

Improved model download and installation

To increase transparency and make it easier to use spaCy with your own models, all data is now available as direct downloads, organised in individual releases. spaCy v1.7 also supports installing and loading models as Python packages. You can now choose how and where you want to keep the data files, and set up "shortcut links" to load models by name from within spaCy. For more info on this, see the new models documentation.

# out-of-the-box: download best-matching default model
python -m spacy download en

# download best-matching version of specific model for your spaCy installation
python -m spacy download en_core_web_md

# pip install .tar.gz archive from path or URL
pip install /Users/you/en_core_web_md-1.2.0.tar.gz
pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-1.2.0/en_core_web_md-1.2.0.tar.gz

# set up shortcut link to load installed package as "en_default"
python -m spacy link en_core_web_md en_default

# set up shortcut link to load local model as "my_amazing_model"
python -m spacy link /Users/you/data my_amazing_model

nlp1 = spacy.load('en')
nlp2 = spacy.load('en_core_web_md')
nlp3 = spacy.load('my_amazing_model')

⚠️ Backwards incompatibilities

• IMPORTANT: Due to fixes to the lemmatizer, the previous English model (v1.1.0) is not compatible with spaCy v1.7.0. When upgrading to this version, you need to download the new model (en_core_web_md v1.2.0). The German model is still valid and will be linked to the de shortcut automatically.
• spaCy's package manger sputnik is now deprecated. For now, we will keep maintaining our download server to support the python -m spacy.{en|de}.download all command in older versions, but it will soon re-route to download the models from GitHub instead.
• English lemmatizer data is now stored in Python files in spacy/en and the WordNet data previously stored in corpora/en has been removed. This should not affect your code, unless you have added functionality that relies on these data files.

This will be the last major release before v2.0, which will introduce a few breaking changes to allow native deep learning integration. If you're using spaCy in production, don't forget to pin your dependencies:

# requirements.txt
spacy>=1.7.0,<2.0.0

# setup.py
install_requires=['spacy>=1.7.0,<2.0.0']

🔴 Bug fixes

• Fix issue #401: Contractions with 's are now lemmatized correctly.
• Fix issue #507, #711, #798: Models are now available as direct downloads.
• Fix issue #669: Span class now has lower_ and upper_ properties.
• Fix issue #686: Pronouns now lemmatize to -PRON-.
• Fix issue #704: Sentence boundary detection improved with new English model.
• Fix issue #717: Contracted verbs now have the correct lemma.
• Fix issue #730, #763, #880, #890: A smaller English model (en_core_web_sm) is now available.
• Fix issue #755: Add missing import to prevent exception using --force.
• Fix issue #759: All available NUM_WORDS are now recognised correctly as like_number.
• Fix issue #766: Add operator to matcher and make sure open patterns are closed at doc end.
• Fix issue #768: Allow zero-width infix tokens in tokenizer exceptions.
• Fix issue #771: Version numbers for ujson and plac are now specified correctly.
• Fix issue #775: "Shell" and "shell" are now excluded from English tokenizer exceptions.
• Fix issue #778: spaCy is now available on conda via conda-forge.
• Fix issue #781: Lemmatizer is now correctly applied to OOV words.
• Fix issue #791: Environment variables are now passed to subprocess calls in cythonize.
• Fix issue #792: Trailing whitespace is now handled correctly by the tokenizer.
• Fix issue #801: Update global infix rules to prevent attached punctuation in complex cases.
• Fix issue #805: Swedish tokenizer exceptions are now imported correctly.
• Fix issue #834: load_vectors() now accepts arbitrary space characters as word tokens.
• Fix issue #840: Use better regex for matching URL patterns in tokenizer exceptions.
• Fix issue #847: "Shed" and "shed" are now excluded from English tokenizer exceptions.
• Fix issue #856: Vocab now adds missing words when importing vectors.
• Fix issue #859: Prevent extra spaces from being added after applying token_match regex.
• Fix issue #868: Model data can now be downloaded to any directory.
• Fix issue #886: token.idx now matches original index when text contains newlines.

📖 Documentation and examples

• NEW: Models usage documentation.
• NEW: Command line interface documentation.
• NEW: spacy-models, including the latest model releases.
• NEW: Linear model feature scheme API docs.
• Add help wanted (easy) issue label for contribution requests suitable for beginners.
• Update installation docs with more details on model download and building spaCy from source.
• Fix various typos and inconsistencies.

👥 Contributors

This release is brought to you by @honnibal and @ines. Thanks to @magnusburton, @jktong, @JasonKessler, @sudowork, @oiwah, @raphael0202, @latkins, @ematvey, @Tpt, @wallinm1, @knub, @wehlutyk, @vaulttech, @nycmonkey, @jondoughty, @aniruddha-adhikary, @badbye, @shuvanon, @rappdw, @ericzhao28, @juanmirocks and @rominf for the pull requests!

v1.6.0: Improvements to tokenizer and tests

✨ Major features and improvements

• Updated token exception handling mechanism to allow the usage of arbitrary functions as token exception matchers.
• Improve how tokenizer exceptions for English contractions and punctuations are generated.
• Update language data for Hungarian and Swedish tokenization.
• Update to use Thinc v6 to prepare for spaCy v2.0.

🔴 Bug fixes

• Fix issue #326: Tokenizer is now more consistent and handles abbreviations correctly.
• Fix issue #344: Tokenizer now handles URLs correctly.
• Fix issue #483: Period after two or more uppercase letters is split off in tokenizer exceptions.
• Fix issue #631: Add richcmp method to Token.
• Fix issue #718: Contractions with She are now handled correctly.
• Fix issue #736: Times are now tokenized with correct string values.
• Fix issue #743: Token is now hashable.
• Fix issue #744: were and Were are now excluded correctly from contractions.

📋 Tests

• Modernise and reorganise all tests and remove model dependencies where possible.
• Improve test speed to ~20s for basic tests (from previously >80s) and ~100s including models (from previously >200s).
• Add fixtures for spaCy components and test utilities, e.g. to create Doc object manually.
• Add documentation for tests to explain conventions and organisation.

👥 Contributors

Thanks to @oroszgy, @magnusburton, @guyrosin and @danielhers and for the pull requests!

v1.5.0: Alpha support for Swedish and Hungarian

✨ Major features and improvements

• NEW: Alpha support for Swedish tokenization.
• NEW: Alpha support for Hungarian tokenization.
• Update language data for Spanish tokenization.
• Speed up tokenization when no data is preloaded by caching the first 10,000 vocabulary items seen.

🔴 Bug fixes

• List the language_data package in the setup.py.
• Fix missing vec_path declaration that was failing if add_vectors was set.
• Allow Vocab to load without serializer_freqs.

📖 Documentation and examples

• NEW: spaCy Jupyter notebooks repo: ongoing collection of easy-to-run spaCy examples and tutorials.
• Fix issue #657: Generalise dependency parsing annotation specs beyond English.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @oroszgy, @magnusburton, @jmizgajski, @aikramer2, @fnorf and @bhargavvader for the pull requests!

v1.4.0: Improved language data and alpha Dutch support

✨ Major features and improvements

• NEW: Alpha support for Dutch tokenization.
• Reorganise and improve format of language data.
• Add shared tag map, entity rules, emoticons and punctuation to language data.
• Convert entity rules, morphological rules and lemmatization rules from JSON to Python.
• Update language data for English, German, Spanish, French, Italian and Portuguese.

🔴 Bug fixes

• Fix issue #649: Update and reorganise stop lists.
• Fix issue #672: Make token.ent_iob_ return unicode.
• Fix issue #674: Add missing lemmas for contracted forms of "be" to TOKENIZER_EXCEPTIONS.
• Fix issue #683: Morphology class now supplies tag map value for the special space tag if it's missing.
• Fix issue #684: Ensure spacy.en.English() loads the Glove vector data if available. Previously was inconsistent with behaviour of spacy.load('en').
• Fix issue #685: Expand TOKENIZER_EXCEPTIONS with unicode apostrophe (’).
• Fix issue #689: Correct typo in STOP_WORDS.
• Fix issue #691: Add tokenizer exceptions for "gonna" and "Gonna".

⚠️ Backwards incompatibilities

No changes to the public, documented API, but the previously undocumented language data and model initialisation processes have been refactored and reorganised. If you were relying on the bin/init_model.py script, see the new spaCy Developer Resources repo. Code that references internals of the spacy.en or spacy.de packages should also be reviewed before updating to this version.

📖 Documentation and examples

• NEW: "Adding languages" workflow.
• NEW: "Part-of-speech tagging" workflow.
• NEW: spaCy Developer Resources repo – scripts, tools and resources for developing spaCy.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @dafnevk, @jvdzwaan, @RvanNieuwpoort, @wrvhage, @jaspb, @savvopoulos and @davedwards for the pull requests!

v1.3.0: Improve API consistency

✨ Major features and improvements

• Add Span.sentiment attribute.
• #658: Add Span.noun_chunks iterator (thanks @pokey).
• #642: Let --data-path be specified when running download.py scripts (thanks @ExplodingCabbage).
• #638: Add German stopwords (thanks @souravsingh).
• #614: Fix PhraseMatcher to work with new Matcher (thanks @sadovnychyi).

🔴 Bug fixes

• Fix issue #605: accept argument to Matcher now rejects matches as expected.
• Fix issue #617: Vocab.load() now works with string paths, as well as Path objects.
• Fix issue #639: Stop words in Language class now used as expected.
• Fix issues #656, #624: Tokenizer special-case rules now support arbitrary token attributes.

📖 Documentation and examples

• Add "Customizing the tokenizer" workflow.
• Add "Training the tagger, parser and entity recognizer" workflow.
• Add "Entity recognition" workflow.
• Fix various typos and inconsistencies.

👥 Contributors

Thanks to @pokey, @ExplodingCabbage, @souravsingh, @sadovnychyi, @manojsakhwar, @TiagoMRodrigues, @savkov, @pspiegelhalter, @chenb67, @kylepjohnson, @YanhaoYang, @tjrileywisc, @dechov, @wjt, @jsmootiv and @blarghmatey for the pull requests!

v1.2.0: Alpha tokenizers for Chinese, French, Spanish, Italian and Portuguese

✨ Major features and improvements

• NEW: Support Chinese tokenization, via Jieba.
• NEW: Alpha support for French, Spanish, Italian and Portuguese tokenization.

🔴 Bug fixes

• Fix issue #376: POS tags for "and/or" are now correct.
• Fix issue #578: --force argument on download command now operates correctly.
• Fix issue #595: Lemmatization corrected for some base forms.
• Fix issue #588: Matcher now rejects empty patterns.
• Fix issue #592: Added exception rule for tokenization of "Ph.D."
• Fix issue #599: Empty documents now considered tagged and parsed.
• Fix issue #600: Add missing token.tag and token.tag_ setters.
• Fix issue #596: Added missing unicode import when compiling regexes that led to incorrect tokenization.
• Fix issue #587: Resolved bug that caused Matcher to sometimes segfault.
• Fix issue #429: Ensure missing entity types are added to the entity recognizer.

v1.1.0: Bug fixes and adjustments

✨ Major features and improvements

• Rename new pipeline keyword argument of spacy.load() to create_pipeline.
• Rename new vectors keyword argument of spacy.load() to add_vectors.

🔴 Bug fixes

• Fix issue #544: Add vocab.resize_vectors() method, to support changing to vectors of different dimensionality.
• Fix issue #536: Default probability was incorrect for OOV words.
• Fix issue #539: Unspecified encoding when opening some JSON files.
• Fix issue #541: GloVe vectors were being loaded incorrectly.
• Fix issue #522: Similarities and vector norms were calculated incorrectly.
• Fix issue #461: ent_iob attribute was incorrect after setting entities via doc.ents
• Fix issue #459: Deserialiser failed on empty doc
• Fix issue #514: Serialization failed after adding a new entity label.

v1.0.0: Support for deep learning workflows and entity-aware rule matcher

✨ Major features and improvements

• NEW: custom processing pipelines, to support deep learning workflows
• NEW: Rule matcher now supports entity IDs and attributes
• NEW: Official/documented training APIs and GoldParse class
• Download and use GloVe vectors by default
• Make it easier to load and unload word vectors
• Improved rule matching functionality
• Move basic data into the code, rather than the json files. This makes it simpler to use the tokenizer without the models installed, and makes adding new languages much easier.
• Replace file-system strings with Path objects. You can now load resources over your network, or do similar trickery, by passing any object that supports the Path protocol.

⚠️ Backwards incompatibilities

• The data_dir keyword argument of Language.__init__ (and its subclasses English.__init__ and German.__init__) has been renamed to path.
• Details of how the Language base-class and its sub-classes are loaded, and how defaults are accessed, have been heavily changed. If you have your own subclasses, you should review the changes.
• The deprecated token.repvec name has been removed.
• The .train() method of Tagger and Parser has been renamed to .update()
• The previously undocumented GoldParse class has a new __init__() method. The old method has been preserved in GoldParse.from_annot_tuples().
• Previously undocumented details of the Parser class have changed.
• The previously undocumented get_package and get_package_by_name helper functions have been moved into a new module, spacy.deprecated, in case you still need them while you update.

🔴 Bug fixes

• Fix get_lang_class bug when GloVe vectors are used.
• Fix Issue #411: doc.sents raised IndexError on empty string.
• Fix Issue #455: Correct lemmatization logic
• Fix Issue #371: Make Lexeme objects hashable
• Fix Issue #469: Make noun_chunks detect root NPs

👥 Contributors

Thanks to @daylen, @RahulKulhari, @stared, @adamhadani, @izeye and @crawfordcomeaux for the pull requests!