Absorb ssplit-cpp and clean headers towards packaging #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jerinphilip
changed the title
jerinphilip
added a commit
that referenced
this pull request
Oct 4, 2023
Change simplifies submodule dependencies towards packaging for
distribution. Path being taken is:
1. `google/sentencepiece` instead of `browsermt/sentencepiece`, which
should correspond to `libsentencepiece-dev`. There are some disparities
in training code, but should be okay for inference. It appears it is not
possible to rely on `libsentencepiece-dev` or equivalents because these
do no export `SentencePieceText` (see relevant SentencePieceAPI docs).
2. `browsermt/ssplit-cpp` is absorbed here into
`{Splitter,Regex}.{hh,cc}`.
3. 8-bit GEMM provider is enabled using `mozilla/gemmology`, which using
CMake is pointed to for include-directory. Since this is header-only and
used only in `QMM.cc`, gemmology headers end up private and is not
needed to distribute. This adds an `xsimd` dependency, which appears to
be available via repositories.
Links
- SentencePiece API
https://github.com/google/sentencepiece/blob/8cbdf13794284c30877936f91c6f31e2c1d5aef7/src/sentencepiece_processor.h#L387-L401
- google/sentencepiece https://github.com/google/sentencepiece
- browsermt/sentencepiece https://github.com/browsermt/sentencepiece
- browsermt/ssplit-cpp https://github.com/browsermt/ssplit-cpp
- mozilla/gemmology https://github.com/mozilla/gemmology
- xsimd https://github.com/xtensor-stack/xsimd
- libsentencepiece-dev
https://packages.debian.org/stable/libdevel/libsentencepiece-dev
Pull-Request: #13
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change simplifies submodule dependencies towards packaging for distribution. Path being taken is:
google/sentencepieceinstead ofbrowsermt/sentencepiece, which should correspond tolibsentencepiece-dev. There are some disparities in training code, but should be okay for inference. It appears it is not possible to rely onlibsentencepiece-devor equivalents because these do no exportSentencePieceText(see relevant SentencePieceAPI docs).browsermt/ssplit-cppis absorbed here into{Splitter,Regex}.{hh,cc}.mozilla/gemmology, which using CMake is pointed to for include-directory. Since this is header-only and used only inQMM.cc, gemmology headers end up private and is not needed to distribute. This adds anxsimddependency, which appears to be available via repositories.Links