4K context doesn't work?

[Zoraman26]

Describe the bug

Not sure if i'm doing things wrong but i downloaded a 4K context size model and any time i try to make an API request i still get errors bout too large acontext size

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

Download TheBloke_Llama-2-13B-GPTQ set max lenth to 4096, it no worky

Screenshot

Logs

N/A

System Info

NVIDIA GeForce RTX 3060
AMD Ryzen 5 7600X
64GB Ram

[RandomInternetPreson]

Try changing the compress_pos_emb value to 2 instead of 1.

[matatonic]

See #3153, (after changing compress_pos_emb to 2), it still wont work.

[CrazyShipOne]

I created settings.yaml as copy of settings-template.yaml, and changed truncation_length to fit model's context size, like 4096. Hopefully in following versions this parameter could be changed automatically when loading models.

[matatonic]

I created settings.yaml as copy of settings-template.yaml, and changed truncation_length to fit model's context size, like 4096. Hopefully in following versions this parameter could be changed automatically when loading models.

You can also update truncation_length on a per-model basis using the models/user-config.yaml file, which may better suit your needs. Also, if you haven't updated it, you should update your models/config.yaml and characters/instruction-following/*.yaml files.

I also just realized that this was a llama-2 model, you don't need to use compress_pos_emb to get 4k with those. Start by updating your models/config.yaml and you may find it just works.

[missionfloyd]

I created settings.yaml as copy of settings-template.yaml, and changed truncation_length to fit model's context size, like 4096. Hopefully in following versions this parameter could be changed automatically when loading models.

Set it how you want it and click "Save settings" on the models tab. The settings will be loaded automatically when you load the model.

[epolewski]

Did you download the model when it first dropped? Looks like the old config.json has 2048 set and it was updated to 4096 a couple days later.

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.