Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generalize convert scripts #3838

Merged
merged 43 commits into from
Nov 9, 2023
Merged

Generalize convert scripts #3838

Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
4823b9b
Initial generic convert script
Galunid Oct 26, 2023
2220124
Remove comments
Galunid Oct 27, 2023
0ff2371
Make gguf_writer member of Model, rework tokenizer export
Galunid Oct 28, 2023
8618b4e
Add [UNTESTED] Baichuan support
Galunid Oct 28, 2023
989db34
Missing variable
Galunid Oct 29, 2023
550b925
Missing variable
Galunid Oct 29, 2023
443f7d5
Call add_tensor before write_* functions
Galunid Oct 29, 2023
08918b7
MPT conversion fix
Galunid Oct 31, 2023
3bb9844
Get rid of dumb print
Galunid Oct 31, 2023
0afa75a
Add Falcon support
Galunid Oct 31, 2023
94ba1db
Add Starcoder and Refact
Galunid Oct 31, 2023
6f6856c
[Untested] Initial Persimmon support
Galunid Oct 31, 2023
b9c664a
Woops
Galunid Oct 31, 2023
0743f7a
Fix variable
Galunid Oct 31, 2023
dc3115f
Add another alias to n_layers
Galunid Oct 31, 2023
b2ba44e
Flake8 fixes
Galunid Oct 31, 2023
c94df09
Rework tokenizer handling
Galunid Oct 31, 2023
235acc1
Small refactor
Galunid Oct 31, 2023
f4b9a7e
Remove 'old' conversion scripts
Galunid Oct 31, 2023
3ec89dc
Use 'IntEnum' instead of 'Enum'
Galunid Oct 31, 2023
4fdd7cd
Review fixes, persimmon fixes
Galunid Nov 1, 2023
8f31dc5
fix mypy errors
cebtenzzre Nov 2, 2023
66ccd62
sort imports
cebtenzzre Nov 2, 2023
e9abcc9
fix linter complaints
cebtenzzre Nov 2, 2023
007be85
model.py : add missing future import
cebtenzzre Nov 2, 2023
03c9683
Restore support for RWForCausalLM
Galunid Nov 4, 2023
fd30850
Add big endian support
Galunid Nov 4, 2023
e64f4de
Revert "Remove 'old' conversion scripts" - needed for testing
Galunid Nov 4, 2023
2120195
Yarn rope for baichuan
Galunid Nov 4, 2023
087f88c
Rename convert-generic -> convert-hf-to-gguf
Galunid Nov 4, 2023
f7de892
Move util to gguf-py/gguf
Galunid Nov 4, 2023
781bc54
Move everything to convert-hf-to-gguf.py
Galunid Nov 5, 2023
fefc3db
address review comments
cebtenzzre Nov 5, 2023
648252e
Fix flake8 complaints
Galunid Nov 6, 2023
05fb6f4
sort imports
cebtenzzre Nov 6, 2023
7a3433b
store_true defaults to False, not None
Galunid Nov 7, 2023
73780f5
Change ftype from int value to str value
Galunid Nov 7, 2023
88b0d9e
Review fixes
Galunid Nov 7, 2023
b714883
Rename variable
Galunid Nov 7, 2023
6a83bce
parse_args improvements
Galunid Nov 7, 2023
2862d16
Remove outdated scripts
Galunid Nov 8, 2023
c79c358
Merge branch 'master' into generic-convert
Galunid Nov 8, 2023
2789ad9
Fix import path
Galunid Nov 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
247 changes: 0 additions & 247 deletions convert-bloom-hf-to-gguf.py
View file
Open in desktop

This file was deleted.

Loading