Absolutely amazing. Waiting for mistral large 2

[AlexBefest]

Guys, I am very grateful to you for such an incredible job. You just made a revolution... We are waiting for the support of mistral large 2

[Cobalt1103]

Yes, I completely agree. I also think that ktransformers should primarily support mistral large 2; it's such a great model... I've seen how it works in other projects, and it truly delivers excellent results. Firstly, it is very fast, which is crucial for our tasks where data processing speed is critical. Secondly, it has high accuracy, allowing us to obtain more quality results.

Moreover, mistral large 2 has excellent support for various languages, making it versatile for our international projects. This is particularly important as we often work with data in different languages.

Additionally, I've heard that mistral large 2 has good compatibility with other libraries and frameworks, which will make integration into ktransformers easier.

And finally, I've seen that mistral large 2 has an active developer community, which means we can get support and updates, which is important for long-term use.

Not to mention the fact that he is at the level of 4o... A must-have model, absolutely

[james0zan]

Currently, the main speedup of KTransformers over llama.cpp is specialized for MoE models, which will also benefit the Mixtral series of models. Thus, we plan to support them soon.

For dense models like mistral-large, llama3.1, and Qwen2-70B, we are still working on some special optimizations that may take some time to finish. Before that, llama.cpp with layer offload would be a good choice to inference them in heterogeneous environment.