project_topics.md

Project topics

1. Transformers:

   • Auto-regressive models:
     • Attention Is All You Need
     • Let's build GPT: from scratch, in code, spelled out
   • Masked autoencoders:
     • Masked Autoencoders Are Scalable Vision Learners
     • Generative Pretraining From Pixels
   • Vision Transformers:
     • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
     • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
   • Rotary position embeddings:
     • RoFormer: Enhanced Transformer with Rotary Position Embedding
     • How Rotary Position Embedding Supercharges Modern LLMs
   • Applications to Neural PDE solvers:
     • Poseidon: Efficient Foundation Models for PDEs
     • Multiple Physics Pretraining for Physical Surrogate Models
     • DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training

2. Generative models:

   • Diffusion models:
     • Step-by-Step Diffusion: An Elementary Tutorial
     • What are Diffusion Models?
     • How I Understand Diffusion Models
     • Diffusion is spectral autoregression
   • Flow matching:
     • Flow Matching for Generative Modeling
     • An Introduction to Flow Matching
     • How I Understand Flow Matching
   • VAE and VQ-VAE:
     • An Introduction to Variational Autoencoders
     • Neural Discrete Representation Learning
   • Image and video generation with Diffusion Transformers:
     • Diffusion Models for Video Generation
     • Scalable Diffusion Models with Transformers
     • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
   • Scientific Applications:
     • Molecule design:
       • Equivariant Flow Matching
       • Equivariant Flow Matching with Hybrid Probability Transport
       • Machine learning-aided generative molecular design
       • Swallowing the Bitter Pill: Simplified Scalable Conformer Generation
     • PDE emulators:
       • PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers
       • DiffFluid: Plain Diffusion Models are Effective Predictors of Flow Dynamics

3. Neural operators:

   • CViT: Continuous Vision Transformer for Operator Learning
   • Transolver: A Fast Transformer Solver for PDEs on General Geometries
   • Fourier Neural Operator for Parametric Partial Differential Equations

4. Physics-informed deep learning:

   • PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks
   • An Expert's Guide to Training Physics-informed Neural Networks
   • ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks
   • Learning the solution operator of PDEs

5. Autonomous experimentation:

   • Taking the Human Out of the Loop: A Review of Bayesian Optimization
   • Autonomous experiments using active learning and AI
   • Probabilistic machine learning and artificial intelligence
   • Composite Bayesian Optimization In Function Spaces