README.md

CS224 Natural Language Processing 2022

Course notes and code: including formula derivation, course knowledge summary, homework and model implementation code.

Lesson Link：CS224N 2022 Schedule 👈 You Can get Slides from here.

Lesson Video: Bilibili | YouTube 💡

Lecture and Papers

• Word Vectors
  • Efficient Estimation of Word Representations in Vector Space (original word2vec paper)
  • Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)
• Word Vectors 2 and Word Window Classification
  • GloVe: Global Vectors for Word Representation (original GloVe paper)
    • Official Code
  • Improving Distributional Similarity with Lessons Learned from Word Embeddings
  • Evaluation methods for unsupervised word embeddings
  • A Latent Variable Model Approach to PMI-based Word Embeddings
  • Linear Algebraic Structure of Word Senses, with Applications to Polysemy
  • On the Dimensionality of Word Embedding
  • Appendix: VSM，LSA，PMI，N-garm，NNLM，RNNLM，SVD
• Backprop and Neural Networks
  • Learning Representations by Backpropagating Errors (seminal Rumelhart et al. backpropagation paper)
  • Natural Language Processing (Almost) from Scratch
• Dependency Parsing
  • Incrementality in Deterministic Dependency Parsing
  • A Fast and Accurate Dependency Parser using Neural Networks
    • Unofficial Code
  • Dependency Parsing (a book need to buy)
  • Globally Normalized Transition-Based Neural Networks
  • Universal Stanford Dependencies: A cross-linguistic typology
  • UD Standard: Universal Dependencies
  • Appendix: t-SNE，Understand Stanford Universal Dependencies，Beam Search，All NLP Task Evaluation
• Recurrent Neural Networks and Language Models
  • N-gram Language Models (textbook chapter)
  • Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.1 and 10.2)
• Vanishing Gradients, Fancy RNNs, Seq2Seq
  • Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
  • On the difficulty of training Recurrent Neural Networks (proof of vanishing gradient problem)
  • Vanishing Gradient Solutions: Multi-Level Hierarchy, Long Short-Term Memory(LSTM)/Residual Neural Network, Rectified Linear Unit (ReLU) Activation Function
• Machine Translation, Attention, Subword Models
  • BLEU (original paper)
  • Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)
  • Sequence Transduction with Recurrent Neural Networks (early seq2seq speech recognition paper)
  • Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)
  • Massive Exploration of Neural Machine Translation Architectures (practical advice for hyperparameter choices)
  • Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
  • Revisiting Character-Based Neural Machine Translation with Capacity and Compression
• Final Projects: Custom and Default; Practical Tips
  • Practical Methodology (Deep Learning book chapter)
  • Addition:
    • Model Pruning (Movement Pruning: Adaptive Sparsity by Fine-Tuning)
    • Model Quantization (TRAINING WITH QUANTIZATION NOISE FOR EXTREME MODEL COMPRESSION)
    • BabyAI (BabyAI 1.1)
    • gSCAN (A Benchmark for Systematic Generalization in rounded Language Understanding)
  • Appendix: NeurlPS2020 Pruning
• Transformers
  • Attention Is All You Need
  • Layer Normalization
  • Image Transformer
  • Music Transformer: Generating music with long-term structure
• More about Transformers and Pretraining
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • Contextual Word Representations: A Contextual Introduction
  • Appendix:
    • GLUE：Explanation，SQuAD v1.1-v2，Situations With Adversarial Generations(SWAG)
    • Pretrain and Finetune: Pre training refers to a model of pre training or the process of pre training the model; Fine tuning refers to the process of applying the pre trained model to its own data set and adapting the parameters to its own data set.
• Question Answering
  • SQuAD: 100,000+ Questions for Machine Comprehension of Text
  • Bidirectional Attention Flow for Machine Comprehension
  • Reading Wikipedia to Answer Open-Domain Questions
  • Latent Retrieval for Weakly Supervised Open Domain Question Answering
  • Dense Passage Retrieval for Open-Domain Question Answering
  • Learning Dense Representations of Phrases at Scale
    • Code
  • Appendix: TF-IDF，BM25
• Natural Language Generation
  • The Curious Case of Neural Text Degeneration
  • Get To The Point: Summarization with Pointer-Generator Networks
  • Hierarchical Neural Story Generation
  • How NOT To Evaluate Your Dialogue System
  • Appendix: METEOR，ROUGE
• Reference in Language and Coreference Resolution
  • Coreference Resolution chapter of Jurafsky and Martin
  • End-to-end Neural Coreference Resolution.
  • Appendix: OntoNotes Benchmark
• T5 and large language models: The good, the bad, and the ugly
  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  • Addition:
    • mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
    • How Much Knowledge Can You Pack Into the Parameters of a Language Model?
    • Extracting Training Data from Large Language Models
    • Do Transformer Modifications Transfer Across Implementations and Applications?
• Integrating knowledge in language models
  • ERNIE: Enhanced Language Representation with Informative Entities
  • Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
  • Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
  • Language Models as Knowledge Bases?
  • Appendix:
    • Baselines：DrAQ，R3，DSQA，Evidence Aggregation，BERTserini，ORQA
• Social & Ethical Considerations in NLP Systems
• Model Analysis and Explanation
• Future of NLP + Deep Learning

Tutorial

Code

• Python
• PyTorch
• Gensim

Theory

• Singular Value Decomposition

Assignment

• Assignment 1
• Assignment 2
• Assignment 3
• Assignment 4
• Assignment 5
• Final Project