chore: add llama_multinode example with nccl

[b0xtch]

Very barebones implementation of Llama multinode for distributed inference

• Adds support for running Llama model inference across multiple nodes and GPUs.
• a simple tcp server to exchange nccl_id

Blocked:

• I am blocked on the flash_attn on cuda >12.X so I haven't been able to finish this
• Flash Attention not working on CUDA 12.1 #1936
• flash attention is broken for cuda-12.x version  Dao-AILab/flash-attention#1004

[LaurentMazare]

• I am blocked on the flash_attn on cuda >12.X so I haven't been able to finish this

Are you sure it's blocked on these versions? I think it's only on 12.1, fwiw I'm using it very frequently with 12.2 and 12.4 and I don't see any such issue.

[b0xtch]

• I am blocked on the flash_attn on cuda >12.X so I haven't been able to finish this

Are you sure it's blocked on these versions? I think it's only on 12.1, fwiw I'm using it very frequently with 12.2 and 12.4 and I don't see any such issue.

I tested with the versions below - {11.8}:

There is a similar report with the problem for v12.5 as well

• Compilation failure with --features flash-attn EricLBuehler/mistral.rs#485

I could be missing something but I am simply starting a new instances, cloning and running