New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Varying experts per layer #43

Open

pglorio wants to merge 85 commits into main from pyramid_moe

pglorio commented Dec 7, 2023

No description provided.

pglorio added 30 commits

December 4, 2023 23:22


          Update switch_mlp.py


          Update switch_mlp.py

f2ecc71


          Update switch_mlp.py

c7ea711


          Update switch_mlp.py

d5f7af4


          Update transformer_block.py

35d3ac4


          Update switch_mlp.py

0e7eb23


          Update transformer_block.py

4ad505a


          Update transformer_block.py

3af0059


          Update transformer_block.py

13c54d9


          Update arguments.py

df2eeb2


          Update initialize.py

1f07db5


          Update initialize.py

b9d6475


          Update initialize.py

1c20628


          Update initialize.py

413bf43


          Update initialize.py

9237b34


          Update arguments.py

01dc1e9


          Update arguments.py

68b99a0


          Update pretrain_gpt.py

28bc3a3


          Update arguments.py

b819330


          Update transformer_block.py

56ad120


          Update switch_mlp.py

8ce18aa


          Update arguments.py

0875fc5


          Update transformer_block.py

4989c58


          Update transformer_block.py

ef743f4


          Update switch_mlp.py

25056d5


          Update switch_mlp.py

7dad158


          Update switch_mlp.py

2aee703


          Update switch_mlp.py

babbebb


          Update switch_mlp.py

179d2a4


          Update transformer_block.py

253c87c

pglorio added 30 commits

December 20, 2023 11:20


          print statement of data shape

74b0c7b


          print statement of data shape

8f7bb1c


          Update transformer_layer.py

714cd8d


          Update gpt_model.py

5fd134f


          Update training.py

d95239c


          Update gpt_model.py

8ac39c9


          Update gpt_model.py

687687e


          Update gpt_model.py

fff6470


          Update gpt_model.py

fac5f79


          Update gpt_model.py

b157390


          Update pretrain_gpt.py

214e0d0


          Update gpt_model.py

0597d75


          Update transformer_layer.py

2b017d1


          Update switch_mlp.py

09b1228


          Update switch_mlp.py

3800c0e


          Update gpt_model.py

5a583a2


          Update fused_softmax.py


          Update fused_softmax.py

0d6f7c9


          Update fused_softmax.py

f8c8de9


          Update fused_softmax.py

2bfb7c0


          Update fused_softmax.py

36a2376


          Update switch_mlp.py

7048fc1


          Update switch_mlp.py

431fa63


          Update switch_mlp.py

fc5390c


          Update switch_mlp.py

5791a87


          Update switch_mlp.py

03aecf6


          Update switch_mlp.py

0e5b34f


          Update switch_mlp.py

704726d


          Update switch_mlp.py

277afeb


          Update switch_mlp.py

4a31d0f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet