Code

WIP: Shared t5 code#286

Code

Open

thomasw21 wants to merge 25 commits intomainbigscience-workshop/Megatron-DeepSpeed:mainfrom thomas/add_shared_t5bigscience-workshop/Megatron-DeepSpeed:thomas/add_shared_t5Copy head branch name to clipboard

+590-32

Commits

Commits on Jun 27, 2022

• WIP

  thomasw21

  committed
  cc6dd1d

  Copy full SHA for cc6dd1d

• Fix shared T5

  thomasw21

  committed
  fbc1d04

  Copy full SHA for fbc1d04

• WIP

  thomasw21

  committed
  91a20aa

  Copy full SHA for 91a20aa

• Shared t5 tests

  thomasw21

  committed
  e55b603

  Copy full SHA for e55b603

• Woops

  thomasw21

  committed
  b5568fb

  Copy full SHA for b5568fb

• Woops

  thomasw21

  committed
  50cf6d9

  Copy full SHA for 50cf6d9

• Hack my way into fix attn_mask

  thomasw21

  committed
  dcb2d61

  Copy full SHA for dcb2d61

• DS has poor default

  thomasw21

  committed
  04cdc2e

  Copy full SHA for 04cdc2e

• Maybe this is better

  thomasw21

  committed
  ffbe3bf

  Copy full SHA for ffbe3bf

• Maybe this is better

  thomasw21

  committed
  90ee34c

  Copy full SHA for 90ee34c

• SEP is only defined for HFTokenizer

  thomasw21

  committed
  ed3dbea

  Copy full SHA for ed3dbea

• add --vocab-extra-ids 100

  thomasw21

  committed
  c93b563

  Copy full SHA for c93b563

• Hopefully this fixes MLM

  thomasw21

  committed
  ef205dd

  Copy full SHA for ef205dd

• Hopefully this fixes MLM

  thomasw21

  committed
  c71b379

  Copy full SHA for c71b379

• Fix MLM datasets length

  thomasw21

  committed
  9da7588

  Copy full SHA for 9da7588

• I think GPT2 tokenizer might be missing <sep> token

  thomasw21

  committed
  07f0ce5

  Copy full SHA for 07f0ce5

• I think GPT2 tokenizer might be missing <sep> token

  thomasw21

  committed
  12626f6

  Copy full SHA for 12626f6

• Flatten to pass tuples of tensor instead

  thomasw21

  committed
  52c42b2

  Copy full SHA for 52c42b2

• Lambda function can't unpack tuples

  thomasw21

  committed
  b25803b

  Copy full SHA for b25803b

• Woops

  thomasw21

  committed
  26d9dce

  Copy full SHA for 26d9dce

• More dirty fixes

  thomasw21

  committed
  fc0560a

  Copy full SHA for fc0560a

• Woops

  thomasw21

  committed
  8578ab3

  Copy full SHA for 8578ab3

• Removing erroneous trailing layers

  thomasw21

  committed
  216a3f5

  Copy full SHA for 216a3f5

Commits on Jun 30, 2022

• Add TODO for loading previous checkpoint

  thomasw21

  committed
  f557523

  Copy full SHA for f557523

Commits on Jul 4, 2022

• WIP: conversion script:

  thomasw21

  committed
  e73aa58

  Copy full SHA for e73aa58