Assertions to exceptions (#13692)

* Raise exceptions instead of using assertions for control flow #12789 * # coding=utf-8 * Raise exceptions instead of using assertions for control flow * Raise exceptions instead of using assertions for control flow * Update src/transformers/tokenization_utils.py Raise exceptions instead of using assertions for control flow Co-authored-by: Suraj Patil <[email protected]> * Update src/transformers/tokenization_utils.py Raise exceptions instead of using assertions for control flow Co-authored-by: Sylvain Gugger <[email protected]> * Raise exceptions instead of using assertions for control flow * test * Raise exceptions instead of using assertions for control flow Co-authored-by: MocktaiLEngineer <[email protected]> Co-authored-by: Suraj Patil <[email protected]> Co-authored-by: Sylvain Gugger <[email protected]>
huggingface · Sep 22, 2021 · a0c08aa · a0c08aa
1 parent 27d4639
commit a0c08aa
Showing 1 changed file with 2 additions and 1 deletion.
diff --git a/src/transformers/tokenization_utils.py b/src/transformers/tokenization_utils.py
@@ -362,7 +362,8 @@ def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_to
 
         tokens_to_add = []
         for token in new_tokens:
-            assert isinstance(token, str)
+            if not isinstance(token, str):
+                raise TypeError(f"Token {token} is not a string but a {type(token)}.")
             if not special_tokens and hasattr(self, "do_lower_case") and self.do_lower_case:
                 token = token.lower()
             if (