Code

MoE#639

Code

Open

Muennighoff wants to merge 144 commits intomainallenai/OLMo:mainfrom Muennighoff/MoEallenai/OLMo:Muennighoff/MoECopy head branch name to clipboard

+1,896-15

Commits

Commits on Jun 20, 2024

• Clean MoE implementation

  Muennighoff

  committed
  e725eb9

• Add conf

  Muennighoff

  committed
  db24750

• Fix return args

  Muennighoff

  committed
  18450de

• Rmv outdated kwarg

  Muennighoff

  committed
  4ab7f77

• Rmv legacy kwarg

  Muennighoff

  committed
  dba42fd

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  6c5f8a3

• Add distributed_strategy

  Muennighoff

  committed
  6a8e089

• Allow w/o weight attr

  Muennighoff

  committed
  1a9a317

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  ddf6fd4

• Allow w/o weight attr

  Muennighoff

  committed
  ab55e07

• Add MoE params

  Muennighoff

  committed
  7aeefd4

• Rmv kwarg

  Muennighoff

  committed
  3eab45c

Commits on Jun 21, 2024

• Reduce lb & moe losses

  Muennighoff

  committed
  6d736da

• LN & Emb Dec

  Muennighoff

  committed
  d07c638

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  cdb592f

• Do not decay emb

  Muennighoff

  committed
  1399841

• Tmp - debug throughput

  Muennighoff

  committed
  a13b5b8

• Fix

  Muennighoff

  committed
  935167e

• Fix

  Muennighoff

  committed
  b96972d

• maintain init order

  Muennighoff

  committed
  0079490

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  8b1c441

• Decay emb

  Muennighoff

  committed
  e2c7286

• Keep EA on CPU

  Muennighoff

  committed
  d39a37c

• Do not decay emb

  Muennighoff

  committed
  3acfc04

• Change norm

  Muennighoff

  committed
  4432261

Commits on Jun 22, 2024

• Confs

  Muennighoff

  committed
  cef7707

• Adapt wrap

  Muennighoff

  committed
  2a6df33

Commits on Jun 23, 2024

• Add conf

  Muennighoff

  committed
  7421890

Commits on Jun 24, 2024

• decemb conf

  Muennighoff

  committed
  021974e

• Updates

  Muennighoff

  committed
  d5a0626

Commits on Jun 25, 2024

• up

  Muennighoff

  committed
  fce086f

• Update weka

  Muennighoff

  committed
  daa7c91

• Rev weka

  Muennighoff

  committed
  3a40b7f

• Adapt

  Muennighoff

  committed
  2676d03

• Add fine

  Muennighoff

  committed
  2fb4c96

Commits on Jun 26, 2024

• Adapt

  Muennighoff

  committed
  448a9a8

• Adapt

  Muennighoff

  committed
  e361918

• Add eps

  Muennighoff

  committed
  9377aa5

Commits on Jun 27, 2024

• Update confs

  Muennighoff

  committed
  bad3a34

• Change eps

  Muennighoff

  committed
  3dbd156

Commits on Jun 28, 2024

• Add conf

  Muennighoff

  committed
  a6496d6

• Confs

  Muennighoff

  committed
  5dd6135

Commits on Jun 29, 2024

• Add conf

  Muennighoff

  committed
  195c77d

• Fix path

  Muennighoff

  committed
  4ea9f0a

• Fix path

  Muennighoff

  committed
  44fa5ae

Commits on Jun 30, 2024

• Adapt confs

  Muennighoff

  committed
  b7658eb

• add s3 conf

  Muennighoff

  committed
  16edece

• fix paths

  Muennighoff

  committed
  f8e061f

Commits on Jul 1, 2024

• Update confs

  Muennighoff

  committed
  c8a51cf

• Increase bs

  Muennighoff

  committed
  b95a05f

• Adjust paths

  Muennighoff

  committed
  20103ec

• conf

  Muennighoff

  committed
  459ef27

• confs

  Muennighoff

  committed
  84f21db

• confs

  Muennighoff

  committed
  08d3253

• Update confs

  Muennighoff

  committed
  4b07140

• Update conf

  Muennighoff

  committed
  cf6fa33

• Conf

  Muennighoff

  committed
  d95f978

Commits on Jul 2, 2024

• Finegrained

  Muennighoff

  committed
  3c15abf

• Cx5

  Muennighoff

  committed
  4163e70

• Add conf

  Muennighoff

  committed
  aae0e0b

• Add cx5

  Muennighoff

  committed
  76e6e6d

• Up confs

  Muennighoff

  committed
  31e387b

• Up confs

  Muennighoff

  committed
  f631d8e

Commits on Jul 4, 2024

• Dfix

  Muennighoff

  committed
  142720b

• reddit

  Muennighoff

  committed
  412a55e

• flan

  Muennighoff

  committed
  669bba5

• fine

  Muennighoff

  committed
  7d90908

• indent

  Muennighoff

  committed
  3c97519

• Add conf

  Muennighoff

  committed
  6e08b09

Commits on Jul 5, 2024

• Conf

  Muennighoff

  committed
  1a6a2e2

• Add shared

  Muennighoff

  committed
  103e450

• Add

  Muennighoff

  committed
  48b6c14

• Add conf

  Muennighoff

  committed
  8a387fa

Commits on Jul 6, 2024

• Add conf

  Muennighoff

  committed
  3203724

• Add conf

  Muennighoff

  committed
  5309137

• Addqk

  Muennighoff

  committed
  bf66e68

• Make QK Norm parametric

  Muennighoff

  committed
  63c12e1

Commits on Jul 7, 2024

• dense comp

  Muennighoff

  committed
  1c4aa8d

• add conf

  Muennighoff

  committed
  6a94263

• conf

  Muennighoff

  committed
  b58c316

• conf

  Muennighoff

  committed
  3bbfaed

Commits on Jul 18, 2024

• Change torch version

  Muennighoff

  committed
  364659f

Commits on Jul 20, 2024

• Add anneal

  Muennighoff

  committed
  15f5503

• Resets

  Muennighoff

  committed
  6c516d8

Commits on Jul 21, 2024

• add conf

  Muennighoff

  committed
  8a0758e

• conf

  Muennighoff

  committed
  c23b048

• Adjust mlp hidden

  Muennighoff

  committed
  285ff10

• Change ratio

  Muennighoff

  committed
  99aec31

• Add conf

  Muennighoff

  committed
  a43eae8

• Conf

  Muennighoff

  committed
  aaefc58

• Adjust

  Muennighoff

  committed
  270271f

• add conf

  Muennighoff

  committed
  c084d34

• Fix conf

  Muennighoff

  committed
  6789ee2

• Adjust

  Muennighoff

  committed
  f084fa0

• Add alt

  Muennighoff

  committed
  14ee7e4

Commits on Jul 23, 2024

• Fix typo; update script

  Muennighoff

  committed
  f831adf

• merge main

  Muennighoff

  committed
  26eb3f3

• add moe reorder

  Muennighoff

  committed
  0a3b076

• Add datafix

  Muennighoff

  committed
  d6ccbf0

• Max doc len MoE

  Muennighoff

  committed
  6a2c17e

• add fa varlen

  Muennighoff

  committed
  9ccb2f1

• Add conf

  Muennighoff

  committed
  f5291ec

Commits on Jul 24, 2024

• add conf

  Muennighoff

  committed
  ed571a8

• fix conf

  Muennighoff

  committed
  71d2d2a

• fixconf

  Muennighoff

  committed
  8b72521

• fixonf

  Muennighoff

  committed
  fc822a0

• fix

  Muennighoff

  committed
  1ccaf9a

Commits on Jul 25, 2024

• Add 8k

  Muennighoff

  committed
  c157441

Commits on Jul 26, 2024

• add conf

  Muennighoff

  committed
  06b8010

• conf

  Muennighoff

  committed
  cf31e53

Commits on Jul 31, 2024

• Adapt

  Muennighoff

  committed
  a750fde

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  committed
  c853a43

• Sort

  Muennighoff

  committed
  bea80ec

• Simplify configs for merge

  Muennighoff

  committed
  dfdcfc5

Commits on Aug 1, 2024

• Update megablocks installation

  Muennighoff

  committed
  4606598

• Simplify

  Muennighoff

  committed
  f6c707d

• Simplify

  Muennighoff

  committed
  76f0376

• typing & changes

  Muennighoff

  committed
  9631c80

• Format

  Muennighoff

  committed
  2fa2acb

• Change torch.tensor to Any so type check passes?

  Muennighoff

  committed
  c517703

• Simplify

  Muennighoff

  committed
  b4eb33f

• Simplify

  Muennighoff

  committed
  ae6f16a

• Sort

  Muennighoff

  committed
  02781be

• Fix type checks

  Muennighoff

  committed
  a62a1ee

• Format

  Muennighoff

  committed
  0ecd4b8

Commits on Aug 3, 2024

• Fix typo; MoEArgs func

  Muennighoff

  committed
  d8452a0

• Format

  Muennighoff

  committed
  8a28ced

• Check for act ckpt strategy & moe; fix typo

  Muennighoff

  committed
  91f5553

• fix import

  Muennighoff

  committed
  61ac104

Commits on Aug 20, 2024

• Sort impot

  Muennighoff

  authored
  f4faf8a

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  authored
  fdc1021

Commits on Aug 21, 2024

• Fix typo

  Show description for ed82181
  

  Muennighoff

  and

  2015aroras

  authored
  ed82181

• Simplify isinstance

  Show description for b0cc754
  

  Muennighoff

  and

  2015aroras

  authored
  b0cc754

• Clean conf & move constructor

  Muennighoff

  committed
  ca9b41f

Commits on Sep 4, 2024

• Add ref

  Muennighoff

  committed
  215c0f5

• Merge main

  Muennighoff

  committed
  43baf74

• Sort imports

  Muennighoff

  committed
  775e514

• Format

  Muennighoff

  committed
  cd0004b

Commits on Sep 12, 2024

• No exp ass

  Muennighoff

  committed
  acb23dd

• Revert

  Muennighoff

  committed
  a143469

• Simplify

  Muennighoff

  committed
  1a4bdae

Commits on Oct 3, 2024

• Rmv interleave

  Muennighoff

  committed
  410064a

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  committed
  671bc8e

Commits on Oct 4, 2024

• expert_assignments on gpu

  Muennighoff

  committed
  04a2da5