[Bug] Davlan/bert fails in demo #116
Labels
bug
Something isn't working
Comments
|
Good catch - thanks! I thought I tested all the demos after migrating the demo-site, but looks like I missed this one :) Fixing now. |
|
Demo back up and running - just need to fix the decoding for the numbers (should be 2016, not 2 0 1 6; unrelated to this issue) |
|
Fixed the number tokenization (which will be updated in v2.0.1; release coming soon) |
|
Thanks xenova -- looking forward to it! |
|
Live in v2.0.1 👍 |
DimQ1
added a commit
to DimQ1/transformers.js
that referenced
this issue
Jun 1, 2023
* [demo] Fix token-classification (Closes huggingface#116) * Fix Bert tokenizer regex for numbers * Update Bert pretokenizer regex Should match the rust implementation: https://github.com/huggingface/tokenizers/blob/b4fcc9ce6e4ad5806e82826f816acfdfdc4fcc67/tokenizers/src/pre_tokenizers/bert.rs#L11 - Removes whitespace - Splits on unicode punctuation and certain ascii characters * Add tokenizer test cases with numbers * Build demo website after release * [package.json] Update keywords * [version] Update to 2.0.1 * Freeze onnxruntime dependencies (huggingface#124) Their latest version has a few issues, particularly with webgpu, and also uses .wasm files which are incompatible with their previous versions. So, while those issues are sorted out, it's best to freeze their packages to the latest stable version. * Use versioned links (#Closes huggingface#114) Prevents issues where users copy-paste the import code, and then a future update breaks it. Also ensures that the default wasm files match the target version * [version] Update to 2.0.2 * Update package-lock.json * Update README.md * Replace `Math.max` with custom `max` function * Add `sentence-transformers` models to supported models/tasks * Correctly use default module if present * Use error mapping instead of switch block * [docs] Fix numbering --------- Co-authored-by: Joshua Lochner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
uncaught promise error when using https://xenova.github.io/transformers.js/
and selecting the Token classification w/ Davlan/bert-base-multilingual-cased-ner-hrl
Task:
How to reproduce
Steps or a minimal working example to reproduce the behavior
go to https://xenova.github.io/transformers.js/
Select the Token classification w/ Davlan/bert-base-multilingual-cased-ner-hrl option from the drop down
Select "Generate" button
Nothing appears to happen and developer window reports:
caught (in promise) Error: token_ids must be a non-empty array of integers.
at Function.decode (worker-ed2ef37e.js:1790:46739)
at worker-ed2ef37e.js:1791:944
at Array.map ()
at token_classification (worker-ed2ef37e.js:1791:906)
at async worker-ed2ef37e.js:1790:127237
Expected behavior
A clear and concise description of what you expected to happen.
I was hoping to see the resulting token ids for the demo.
Logs/screenshots
If applicable, add logs/screenshots to help explain your problem.
Environment
Additional context
Add any other context about the problem here.
The translation demo does work, so other aspects of this awesome library are working.
The text was updated successfully, but these errors were encountered: