Decode stream python #1678
Decode stream python #1678
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| /// | ||
| #[pyclass(module = "tokenizers.decoders", name = "DecodeStream")] | ||
| #[derive(Clone)] | ||
| pub struct PyDecodeStream { |
There was a problem hiding this comment.
intersting, so we don't use the rust DecodeStream object. I am guessing it's for ownership reasons? Otherwise we needs to wrap the PyDecodeStream {stream: DecodeStream} with arc?
There was a problem hiding this comment.
Arc doesn't save us. The issue is with the borrow of the tokenizer's lifetime.
You can technically reborrow in every call in rust too, it's just not very "rusty".
For Python, we need to get access to the tokenizer on every call, and cloning into an Arc feels super wasteful (and breaks every update you might do on the tokenizer afterwards).
This seems innocuous enough since currently users have to hold already a reference to the tokenizer anyway.
| } | ||
| } | ||
|
|
||
| /// Internal function exposed only to bypass python limitations |
There was a problem hiding this comment.
what were the limitations?
Lifetimes cannot cross the boundary. |
Different API because Python cannot handle lifetimes properly.
Narsil
force-pushed
the
decode_stream_python
branch
from
e16eaac
to
70c4f6b
Compare
Follow up of #1677