NeuralSP: Neural network based Speech Processing

How to install

cd tools
make KALDI=/path/to/kaldi TOOL=/path/to/save/tools

Key features

Corpus

• ASR

  • AISHELL-1
  • AISHELL-2
  • AMI
  • CSJ
  • LaboroTVSpeech
  • Librispeech
  • Switchboard (+Fisher)
  • TEDLIUM2/TEDLIUM3
  • TIMIT
  • WSJ

• LM

  • Penn Tree Bank
  • WikiText2

Front-end

• Frame stacking
• Sequence summary network [link]
• SpecAugment [link]
• Adaptive SpecAugment [link]

Encoder

• RNN encoder
  • (CNN-)BLSTM, (CNN-)LSTM, (CNN-)BLGRU, (CNN-)LGRU
  • Latency-controlled BRNN [link]
  • Random state passing (RSP) [link]
• Transformer encoder [link]
  • Chunk hopping mechanism [link]
  • Relative positional encoding [link]
  • Causal mask
• Conformer encoder [link]
• Time-depth separable (TDS) convolution encoder [link] [line]
• Gated CNN encoder (GLU) [link]

Connectionist Temporal Classification (CTC) decoder

• Beam search
• Shallow fusion
• Forced alignment

RNN-Transducer (RNN-T) decoder [link]

• Beam search
• Shallow fusion

Attention-based decoder

• RNN decoder
  • Shallow fusion
  • Cold fusion [link]
  • Deep fusion [link]
  • Forward-backward attention decoding [link]
  • Ensemble decoding
  • internal LM estimation [link]
• Attention type
  • location-based
  • content-based
  • dot-product
  • GMM attention
• Streaming RNN decoder specific
  • Hard monotonic attention [link]
  • Monotonic chunkwise attention (MoChA) [link]
  • Delay constrained training (DeCoT) [link]
  • Minimum latency training (MinLT) [link]
  • CTC-synchronous training (CTC-ST) [link]
• Transformer decoder [link]
• Streaming Transformer decoder specific
  • Monotonic Multihead Attention [link] [link]

Language model (LM)

• RNNLM (recurrent neural network language model)
• Gated convolutional LM [link]
• Transformer LM
• Transformer-XL LM [link]
• Adaptive softmax [link]

Output units

• Phoneme
• Grapheme
• Wordpiece (BPE, sentencepiece)
• Word
• Word-char mix

Multi-task learning (MTL)

Multi-task learning (MTL) with different units are supported to alleviate data sparseness.

• Hybrid CTC/attention [link]
• Hierarchical Attention (e.g., word attention + character attention) [link]
• Hierarchical CTC (e.g., word CTC + character CTC) [link]
• Hierarchical CTC+Attention (e.g., word attention + character CTC) [link]
• Forward-backward attention [link]
• LM objective

ASR Performance

AISHELL-1 (CER)

Model	dev	test
Conformer LAS	4.1	4.5
Transformer	5.0	5.4
Streaming MMA	5.5	6.1

AISHELL-2 (CER)

Model	test_android	test_ios	test_mic
Conformer LAS	6.1	5.5	5.9

CSJ (WER)

Model	eval1	eval2	eval3
Conformer LAS	5.7	4.4	4.9
BLSTM LAS	6.5	5.1	5.6
LC-BLSTM MoChA	7.4	5.6	6.4

Switchboard 300h (WER)

Model	SWB	CH
BLSTM LAS	9.1	18.8

Switchboard+Fisher 2000h (WER)

Model	SWB	CH
BLSTM LAS	7.8	13.8

LaboroTVSpeech (CER)

Model	dev_4k	dev	tedx-jp-10k
Conformer LAS	7.8	10.1	12.4

Librispeech (WER)

Model	dev-clean	dev-other	test-clean	test-other
Conformer LAS	1.9	4.6	2.1	4.9
Transformer	2.1	5.3	2.4	5.7
BLSTM LAS	2.5	7.2	2.6	7.5
BLSTM RNN-T	2.9	8.5	3.2	9.0
UniLSTM RNN-T	3.7	11.7	4.0	11.6
UniLSTM MoChA	4.1	11.0	4.2	11.2
LC-BLSTM RNN-T	3.3	9.8	3.5	10.2
LC-BLSTM MoChA	3.3	8.8	3.5	9.1
Streaming MMA	2.5	6.9	2.7	7.1

TEDLIUM2 (WER)

Model	dev	test
Conformer LAS	7.0	6.8
BLSTM LAS	8.1	7.5
LC-BLSTM RNN-T	8.0	7.7
LC-BLSTM MoChA	10.3	8.6
UniLSTM RNN-T	10.7	10.7
UniLSTM MoChA	13.5	11.6

WSJ (WER)

Model	test_dev93	test_eval92
BLSTM LAS	8.8	6.2

LM Performance

Penn Tree Bank (PPL)

Model	valid	test
RNNLM	87.99	86.06
+ cache=100	79.58	79.12
+ cache=500	77.36	76.94

WikiText2 (PPL)

Model	valid	test
RNNLM	104.53	98.73
+ cache=100	90.86	85.87
+ cache=2000	76.10	72.77

Reference

• https://github.com/kaldi-asr/kaldi
• https://github.com/espnet/espnet
• https://github.com/awni/speech
• https://github.com/HawkAaron/E2E-ASR

Dependency

• https://github.com/SeanNaren/warp-ctc
• https://github.com/HawkAaron/warp-transducer
• https://github.com/1ytic/warp-rnnt