A Collection of Text-to-Image Generation Studies

This GitHub repository summarizes papers and resources related to the text-to-image (T2I) generation task.

Note

This document serves as the homepage of the whole GitHub repo. Papers are summarized according to different research directions, published years, and conferences.

The topics section summarizes papers that are highly related to T2I generation according to different properties, e.g., prerequisites of T2I generation, diffusion models with other techniques (e.g., Diffusion Transformer, LLMs, Mamba, etc.), and diffusion models for other tasks.

If you have any suggestions about this repository, please feel free to start a new issue or pull requests.

Recent news of this GitHub repo are listed as follows.

🔥 [Nov. 19th] We have released our latest paper titled "StableV2V: Stablizing Shape Consistency in Video-to-Video Editing", with the correponding code, model weights, and a testing benchmark DAVIS-Edit open-sourced. Feel free to check them out from the links!

Click to see more information.

• [Apr. 26th] Update a new topic: Diffusion Models Meet Federated Learning. See the topics section for more details!
• [Mar. 28th] The official AAAI 2024 paper list are released! Official version of PDFs and BibTeX references are updated accordingly.
• [Mar. 21th] The topics section has been updated. This section aims to offer paper lists that are summarized according to other properties of diffusion models, e.g., Diffusion Transformer-based methods, diffusion models for NLP, diffusion models integrated with LLMs, etc. The corresponding references of these papers are also concluded in reference.bib.
• [Mar. 7th] All available CVPR, ICLR, and AAAI 2024 papers and references are updated.
• [Mar. 1st] Websites of the off-the-shelf text-to-image generation products and toolkits are summarized.

Contents

• Products
• To-Do Lists
• Papers
  • Survey Papers
  • Text-to-Image Generation
    • Year 2024
    • Year 2023
    • Year 2022
    • Year 2021
    • Year 2020
  • Conditional Text-to-Image Generation
    • Year 2024
    • Year 2023
    • Year 2022
  • Personalized Text-to-Image Generation
    • Year 2024
    • Year 2023
  • Text-Guided Image Editing
    • Year 2024
    • Year 2023
    • Year 2022
  • Text Image Generation
    • Year 2024
• Datasets
• Toolkits
• Q&A
• References
• Star History

To-Do Lists

• Published Papers on Conferences
  • Update NeurIPS 2024 Papers
  • Update ECCV 2024 Papers
  • Update CVPR 2024 Papers
    • Update ⚠️ Papers and References
    • Update arXiv References into the Official Version
  • Update AAAI 2024 Papers
    • Update ⚠️ Papers and References
    • Update arXiv References into the Official Version
  • Update ICLR 2024 Papers
  • Update NeurIPS 2023 Papers
• Regular Maintenance of Preprint arXiv Papers and Missed Papers

<🎯Back to Top>

Products

Name	Year	Website	Specialties
Stable Diffusion 3	2024	link	Diffusion Transformer-based Stable Diffusion
Stable Video	2024	link	High-quality high-resolution images
DALL-E 3	2023	link	Collaborate with ChatGPT
Ideogram	2023	link	Text images
Playground	2023	link	Athestic images
HiDream.ai	2023	link	-
Dashtoon	2023	link	Text-to-Comic Generation
WHEE	2023	link	WHEE is an online AI generation tool, which can be applied for T2I generation, I2I generation, SR, inpainting, outpainting, image variation, virtural try-on, etc.
Vega AI	2023	link	Vega AI is an online AI generation tool, which can be applied for T2I generation, I2I generation, SR, T2V generation, I2V generation, etc.
Wujie AI	2022	link	The Chinese name is "无界AI", offering AIGC resources and online services
Midjourney	2022	link	Powerful close-sourced generation tool

<🎯Back to Top>

Papers

Survey Papers

• Text-to-Image Generation
  • Year 2024
    • ACM Computing Surveys
      • Diffusion Models: A Comprehensive Survey of Methods and Applications [Paper]
  • Year 2023
    • TPAMI
      • Diffusion Models in Vision: A Survey [Paper] [Code]
    • arXiv
      • Text-to-image Diffusion Models in Generative AI: A Survey [Paper]
      • State of the Art on Diffusion Models for Visual Computing [Paper]
  • Year 2022
    • arXiv
      • Efficient Diffusion Models for Vision: A Survey [Paper]
• Conditional Text-to-Image Generation
  • Year 2024
    • arXiv
      • Controllable Generation with Text-to-Image Diffusion Models: A Survey [Paper]
• Text-Guided Image Editing
  • Year 2024
    • arXiv
      • Diffusion Model-Based Image Editing: A Survey [Paper] [Code]

<🎯Back to Top>

Text-to-Image Generation

• Year 2024
  • CVPR
    • DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models [Paper] [Code]
    • InstanceDiffusion: Instance-level Control for Image Generation [Paper] [Code] [Project]
    • ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations [Paper] [Code] [Project] [Demo]
    • Instruct-Imagen: Image Generation with Multi-modal Instruction [Paper]
    • Learning Continuous 3D Words for Text-to-Image Generation [Paper] [Code]
    • HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances [Paper]
    • Rich Human Feedback for Text-to-Image Generation [Paper]
    • MarkovGen: Structured Prediction for Efficient Text-to-Image Generation [Paper]
    • Customization Assistant for Text-to-image Generation [Paper]
    • ADI: Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation [Paper] [Project]
    • UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs [Paper]
    • Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation [Paper]
    • Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting [Paper] [Code]
    • CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation [Paper] [Code] [Project] [Demo]
    • Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder [Paper]
    • Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation [Paper]
    • ElasticDiffusion: Training-free Arbitrary Size Image Generation [Paper] [Code] [Project] [Demo]
    • CosmicMan: A Text-to-Image Foundation Model for Humans [Paper] [Code] [Project]
    • PanFusion: Taming Stable Diffusion for Text to 360° Panorama Image Generation [Paper] [Code] [Project]
    • Intelligent Grimm: Open-ended Visual Storytelling via Latent Diffusion Models [Paper] [Code] [Project]
    • On the Scalability of Diffusion-based Text-to-Image Generation [Paper]
    • MuLAn: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation [Paper] [Project] [Dataset]
    • Learning Multi-dimensional Human Preference for Text-to-Image Generation [Paper]
    • Dynamic Prompt Optimizing for Text-to-Image Generation [Paper]
    • Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning [Paper]
    • Adversarial Text to Continuous Image Generation [Paper] [Project] [Video]
    • EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models [Paper] [Code]
  • ECCV
    • Bridging Different Language Models and Generative Vision Models for Text-to-Image Generation [Paper] [Code] [Project]
    • Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model [Paper] [Code]
    • Getting it Right: Improving Spatial Consistency in Text-to-Image Models [Paper] [Code] [Project]
    • Navigating Text-to-Image Generative Bias across Indic Languages [Paper] [Project]
    • Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion [Paper]
    • The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation [Paper] [Code] [Project] [Dataset]
    • Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models [Paper] [Code]
    • Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model [Paper] [Code]
    • StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models [Paper] [Code]
    • PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation [Paper] [Code]
    • Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [Paper] [Code]
    • Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation [Paper]
    • Bridging Different Language Models and Generative Vision Models for Text-to-Image Generation [Paper] [Code] [Project]
    • MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices [Paper]
    • PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation [Paper] [Code] [Project]
    • CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion [Paper] [Code]
  • ICLR
    • Patched Denoising Diffusion Models For High-Resolution Image Synthesis [Paper] [Code]
    • Relay Diffusion: Unifying diffusion process across resolutions for image synthesis [Paper] [Code]
    • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis [Paper] [Code]
    • Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis [Paper] [Code]
    • PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis [Paper] [Code] [Project] [Demo]
  • SIGGRAPH
    • RGB↔X: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models [Paper] [Project]
  • AAAI
    • Semantic-aware Data Augmentation for Text-to-image Synthesis [Paper]
    • Text-to-Image Generation for Abstract Concepts [Paper]
  • arXiv
    • Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [Paper]
    • RPG: Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs [Paper] [Code]
    • Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation [Paper] [Code]
    • ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models [Paper] [Code] [Project]
    • InstantID: Zero-shot Identity-Preserving Generation in Seconds [Paper] [Code] [Project] [Demo]
    • PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models [Paper] [Code]
    • ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment [Paper] [Code] [Project]
    • Text2Street: Controllable Text-to-image Generation for Street Views [Paper]
    • LayerDiffuse: Transparent Image Layer Diffusion using Latent Transparency [Paper] [Code]
    • SD3-Turbo: Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation [Paper]
    • StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based Semantic Control [Paper] [Code]
    • SVGDreamer: Text Guided SVG Generation with Diffusion Model [Paper] [Code] [Project]
    • PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement [Paper]
    • YOSO: You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs [Paper] [Code]
    • SingDiffusion: Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models [Paper] [Code]
    • CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching [Paper] [Code] [Project]
    • StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation [Paper] [Code] [Project] [Demo]
    • Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control [Paper] [Project]
    • LinFusion: 1 GPU, 1 Minute, 16K Image [Paper] [Code] [Project] [Demo]
    • OmniGen: Unified Image Generation [Paper] [Code]
  • Others
    • Stable Cascade [Blog] [Code]

<🎯Back to Top>

• Year 2023
  • CVPR
    • GigaGAN: Scaling Up GANs for Text-to-Image Synthesis [Paper] [Reproduced Code] [Project] [Video]
    • ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model With Knowledge-Enhanced Mixture-of-Denoising-Experts [Paper]
    • Shifted Diffusion for Text-to-image Generation [Paper] [Code]
    • GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis [Paper] [Code]
    • Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models to Learn Any Unseen Style [Paper] [Code]
    • Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation [Paper]
    • RIATIG: Reliable and Imperceptible Adversarial Text-to-Image Generation with Natural Prompts [Paper] [Code]
  • ICCV
    • DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning [Paper] [Code] [Demo]
  • NeurIPS
    • ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation [Paper] [Code]
    • RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths [Paper] [Project]
    • Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment [Paper] [Code]
  • ICLR
    • Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis [Paper] [Code]
  • ICML
    • StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis [Paper] [Code] [Project] [Video]
    • Muse: Text-To-Image Generation via Masked Generative Transformers [Paper] [Reproduced Code] [Project]
    • UniDiffusers: One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale [Paper] [Code]
  • ACM MM
    • SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models [Paper] [Code]
    • ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [Paper]
  • SIGGRAPH
    • Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models [Paper] [Code] [Project] [Demo]
  • arXiv
    • P+: Extended Textual Conditioning in Text-to-Image Generation [Paper]
    • SDXL-Turbo: Adversarial Diffusion Distillation [Paper] [Code]
    • Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models [Paper] [Code]
    • StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [Paper] [Project]
    • ParaDiffusion: Paragraph-to-Image Generation with Information-Enriched Diffusion Model [Paper] [Code]
  • Others
    • DALL-E 3: Improving Image Generation with Better Captions [Paper]

<🎯Back to Top>

• Year 2022
  • CVPR
    • 🔥 Stable Diffusion: High-Resolution Image Synthesis With Latent Diffusion Models [Paper] [Code] [Project]
    • Vector Quantized Diffusion Model for Text-to-Image Synthesis [Paper] [Code]
    • DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis [Paper] [Code]
    • LAFITE: Towards Language-Free Training for Text-to-Image Generation [Paper] [Code]
    • Text-to-Image Synthesis based on Object-Guided Joint-Decoding Transformer [Paper]
    • StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis [Paper] [Code]
  • ECCV
    • Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [Paper] [Code] [Demo]
    • Trace Controlled Text to Image Generation [Paper]
    • Improved Masked Image Generation with Token-Critic [Paper]
    • VQGAN-CLIP: Open Domain Image Generation and Manipulation Using Natural Language [Paper] [Code]
    • TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation [Paper] [Code]
    • StoryDALL-E: Adapting Pretrained Text-to-image Transformers for Story Continuation [Paper] [Code] [Demo]
  • NeurIPS
    • CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers [Paper] [Code]
    • Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding [Paper] [Reproduced Code] [Project] [Imagen 2]
  • ACM MM
    • Adma-GAN: Attribute-Driven Memory Augmented GANs for Text-to-Image Generation [Paper] [Code]
    • Background Layout Generation and Object Knowledge Transfer for Text-to-Image Generation [Paper]
    • DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation [Paper]
    • AtHom: Two Divergent Attentions Stimulated By Homomorphic Training in Text-to-Image Synthesis [Paper]
  • arXiv
    • DALLE-2: Hierarchical Text-Conditional Image Generation with CLIP Latents [Paper]
    • PITI: Pretraining is All You Need for Image-to-Image Translation [Paper] [Code]

<🎯Back to Top>

• Year 2021
  • ICCV
    • DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis [Paper] [Code]
  • NeurIPS
    • CogView: Mastering Text-to-Image Generation via Transformers [Paper] [Code] [Demo]
    • UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis [Paper]
  • ICML
    • DALLE-1: Zero-Shot Text-to-Image Generation [Paper] [Reproduced Code]
  • ACM MM
    • Cycle-Consistent Inverse GAN for Text-to-Image Synthesis [Paper]
    • R-GAN: Exploring Human-like Way for Reasonable Text-to-Image Synthesis via Generative Adversarial Networks [Paper]

<🎯Back to Top>

• Year 2020
  • ACM MM
    • Text-to-Image Synthesis via Aesthetic Layout [Paper]

<🎯Back to Top>

Conditional Text-to-Image Generation

• Year 2024
  • CVPR
    • PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis [Paper]
    • One-Shot Structure-Aware Stylized Image Synthesis [Paper]
    • Grounded Text-to-Image Synthesis with Attention Refocusing [Paper] [Code] [Project] [Demo]
    • Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [Paper] [Code]
    • DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception [Paper]
    • CAN: Condition-Aware Neural Network for Controlled Image Generation [Paper]
    • SceneDiffusion: Move Anything with Layered Scene Diffusion [Paper]
    • Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis [Paper] [Code]
    • MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis [Paper] [Code] [Project]
    • FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition [Paper] [Code] [Project]
  • ECCV
    • PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control [Paper] [Code] [Project]
    • AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation [Paper] [Code]
  • ICLR
    • Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models [Paper] [Code]
  • WACV
    • Training-Free Layout Control with Cross-Attention Guidance [Paper] [Code] [Project] [Demo]
  • AAAI
    • SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-image Generation [Paper]
    • Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models [Paper] [Code]
  • arXiv
    • DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations [Paper]
    • InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation [Paper] [Code] [Project]
    • ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback [Paper] [Project]
    • Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding [Paper] [Code] [Project]
    • DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation [Paper] [Code] [Project]
    • ControlNeXt: Powerful and Efficient Control for Image and Video Generation [Paper] [Code] [Project]
    • UniPortrait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization [Paper] [Code] [Project] [Demo]
    • OmniControl: Minimal and Universal Control for Diffusion Transformer [Paper] [Code] [Demo]

<🎯Back to Top>

• Year 2023
  • CVPR
    • GLIGEN: Open-Set Grounded Text-to-Image Generation [Paper] [Code] [Project] [Demo] [Video]
    • Autoregressive Image Generation using Residual Quantization [Paper] [Code]
    • SpaText: Spatio-Textual Representation for Controllable Image Generation [Paper] [Project] [Video]
    • Text to Image Generation with Semantic-Spatial Aware GAN [Paper]
    • ReCo: Region-Controlled Text-to-Image Generation [Paper] [Code]
    • LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation [Paper] [Code]
  • ICCV
    • ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models [Paper] [Code]
    • SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis [Paper] [Code]
    • ZestGuide: Zero-Shot Spatial Layout Conditioning for Text-to-Image Diffusion Models [Paper]
  • ICML
    • Composer: Creative and Controllable Image Synthesis with Composable Conditions [Paper] [Code] [Project]
    • MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation [Paper] [Code] [Video] [Project] [Demo]
  • SIGGRAPH
    • Sketch-Guided Text-to-Image Diffusion Models [Paper] [Reproduced Code] [Project]
  • NeurIPS
    • Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models [Paper] [Code] [Project]
    • Prompt Diffusion: In-Context Learning Unlocked for Diffusion Models [Paper] [Code] [Project]
  • WACV
    • More Control for Free! Image Synthesis with Semantic Diffusion Guidance [Paper]
  • ACM MM
    • LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [Paper]
  • arXiv
    • T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models [Paper] [Code] [Demo]
    • BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing [Paper] [Code]
    • Late-Constraint Diffusion Guidance for Controllable Image Synthesis [Paper] [Code]
• Year 2022
  • ICLR
    • SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations [Paper] [Code] [Project]

<🎯Back to Top>

Personalized Text-to-Image Generation

• Year 2024
  • CVPR
    • Cross Initialization for Personalized Text-to-Image Generation [Paper]
    • When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation [Paper] [Code] [Project]
    • Style Aligned Image Generation via Shared Attention [Paper] [Code] [Project]
    • InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning [Paper] [Project]
    • High Fidelity Person-centric Subject-to-Image Synthesis [Paper]
    • RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization [Paper] [Project]
    • DisenDiff: Attention Calibration for Disentangled Text-to-Image Personalization [Paper] [Code]
    • FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [Paper] [Code] [Project]
    • Personalized Residuals for Concept-Driven Text-to-Image Generation [Paper]
    • Improving Subject-Driven Image Synthesis with Subject-Agnostic Guidance [Paper]
    • JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [Paper]
    • Countering Personalized Text-to-Image Generation with Influence Watermarks [Paper]
  • ECCV
    • Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation [Paper] [Project]
    • Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning [Paper] [Code]
    • TIGC: Tuning-Free Image Customization with Image and Text Guidance [Paper] [Code] [Project]
    • MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation [Paper] [Code] [Project]
  • AAAI
    • Decoupled Textual Embeddings for Customized Image Generation [Paper]
  • arXiv
    • FlashFace: Human Image Personalization with High-fidelity Identity Preservation [Paper] [Code] [Project]
    • MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation [Paper]
    • IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models [Paper]
    • CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization [Paper]
    • Imagine yourself: Tuning-Free Personalized Image Generation [Paper] [Project]
• Year 2023
  • CVPR
    • Custom Diffusion: Multi-Concept Customization of Text-to-Image Diffusion [Paper] [Code] [Project]
    • DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation [Paper] [Code] [Project]
  • ICCV
    • ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation [Paper] [Code]
  • ICLR
    • Textual Inversion: An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion [Paper] [Code] [Project]
  • SIGGRAPH
    • Break-A-Scene: Extracting Multiple Concepts from a Single Image [Paper] [Code]
    • Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models [Paper] [Project]
    • LayerDiffusion: Layered Controlled Image Editing with Diffusion Models [Paper]
  • arXiv
    • DreamTuner: Single Image is Enough for Subject-Driven Generation [Paper] [Project]
    • PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding [Paper] [Code]
    • IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models [Paper] [Code] [Project]
    • FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention [Paper] [Code]

<🎯Back to Top>

Text-Guided Image Editing

• Year 2024
  • CVPR
    • InfEdit: Inversion-Free Image Editing with Natural Language [Paper] [Code] [Project]
    • Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing [Paper]
    • Doubly Abductive Counterfactual Inference for Text-based Image Editing [Paper] [Code]
    • Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation [Paper] [Code]
    • Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing [Paper]
    • DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [Paper] [Code]
    • DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [Paper]
    • FreeDrag: Feature Dragging for Reliable Point-based Image Editing [Paper] [Code]
    • Text-Driven Image Editing via Learnable Regions [Paper] [Code] [Project] [Video]
    • LEDITS++: Limitless Image Editing using Text-to-Image Models [Paper] [Code] [Project] [Demo]
    • SmartEdit: Exploring Complex Instruction-based Image Editing with Large Language Models [Paper] [Code] [Project]
    • Edit One for All: Interactive Batch Image Editing [Paper] [Code] [Project]
    • DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing [Paper] [Code] [Project] [Demo]
    • TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing [Paper] [Code]
    • Person in Place: Generating Associative Skeleton-Guidance Maps for Human-Object Interaction Image Editing [Paper] [Project] [Code]
    • Referring Image Editing: Object-level Image Editing via Referring Expressions [Paper]
    • Prompt Augmentation for Self-supervised Text-guided Image Manipulation [Paper]
    • The Devil is in the Details: StyleFeatureEditor for Detail-Rich StyleGAN Inversion and High Quality Image Editing [Paper] [Code]
  • ECCV
    • RegionDrag: Fast Region-Based Image Editing with Diffusion Models [Paper] [Code] [Project] [Demo]
    • TurboEdit: Instant text-based image editing [Paper] [Project]
    • InstructGIE: Towards Generalizable Image Editing [Paper]
    • StableDrag: Stable Dragging for Point-based Image Editing [Paper]
    • Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editing [Paper] [Code] [Project]
    • SwapAnything: Enabling Arbitrary Object Swapping in Personalized Image Editing [Paper] [Code] [Project]
    • Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing [Paper]
    • FreeDiff: Progressive Frequency Truncation for Image Editing with Diffusion Models [Paper] [Code]
    • Lazy Diffusion Transformer for Interactive Image Editing [Paper] [Project]
    • ByteEdit: Boost, Comply and Accelerate Generative Image Editing [Paper] [Project]
  • ICLR
    • Guiding Instruction-based Image Editing via Multimodal Large Language Models [Paper] [Code] [Project]
    • The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing [Paper] [Code] [Project]
    • Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators [Paper] [Code] [Project]
    • Object-Aware Inversion and Reassembly for Image Editing [Paper] [Code] [Project]
    • Noise Map Guidance: Inversion with Spatial Context for Real Image Editing [Paper]
  • AAAI
    • Tuning-Free Inversion-Enhanced Control for Consistent Image Editing [Paper]
    • BARET: Balanced Attention based Real image Editing driven by Target-text Inversion [Paper]
    • Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference [Paper]
    • High-Fidelity Diffusion-based Image Editing [Paper]
    • AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing [Paper]
    • TexFit: Text-Driven Fashion Image Editing with Diffusion Models [Paper]
  • arXiv
    • An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control [Paper] [Code]
    • One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications [Paper] [Code] [Project]
    • EditWorld: Simulating World Dynamics for Instruction-Following Image Editing [Paper] [Code] [Project]
    • ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing [Paper]
• Year 2023
  • CVPR
    • Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [Paper] [Code]
    • SINE: SINgle Image Editing with Text-to-Image Diffusion Models [Paper] [Code]
    • Imagic: Text-Based Real Image Editing with Diffusion Models [Paper]
    • InstructPix2Pix: Learning to Follow Image Editing Instructions [Paper] [Code] [Dataset] [Project] [Demo]
    • Null-text Inversion for Editing Real Images using Guided Diffusion Models [Paper] [Code]
  • ICCV
    • MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing [Paper] [Code] [Project] [Demo]
    • Localizing Object-level Shape Variations with Text-to-Image Diffusion Models [Paper] [Code] [Project] [Demo]
  • ICLR
    • SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations [Paper] [Code] [Project]
• Year 2022
  • CVPR
    • DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation [Paper] [Code]

<🎯Back to Top>

Text Image Generation

• Year 2024
  • arXiv
    • AnyText: Multilingual Visual Text Generation And Editing [Paper] [Code] [Project]
  • CVPR
    • SceneTextGen: Layout-Agnostic Scene Text Image Synthesis with Integrated Character-Level Diffusion and Contextual Consistency [Paper]

<🎯Back to Top>

Datasets

• Microsoft COCO: Common Objects in Context [Paper] [Dataset]
• Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning [Paper] [Dataset]
• LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models [Paper] [Dataset]
• PartiPrompts: Scaling Autoregressive Models for Content-Rich Text-to-Image Generation [Paper] [Dataset] [Project]

<🎯Back to Top>

Toolkits

Name	Website	Description
Stable Diffusion WebUI	link	Built based on Gradio, deployed locally to run Stable Diffusion checkpoints, LoRA weights, ControlNet weights, etc.
Stable Diffusion WebUI-forge	link	Built based on Gradio, deployed locally to run Stable Diffusion checkpoints, LoRA weights, ControlNet weights, etc.
Fooocus	link	Built based on Gradio, offline, open source, and free. The manual tweaking is not needed, and users only need to focus on the prompts and images.
ComfyUI	link	Deployed locally to enable customized workflows with Stable Diffusion
Civitai	link	Websites for community Stable Diffusion and LoRA checkpoints

<🎯Back to Top>

Q&A

• Q: The conference sequence of this paper list?
  • This paper list is organized according to the following sequence:
    • CVPR
    • ICCV
    • ECCV
    • WACV
    • NeurIPS
    • ICLR
    • ICML
    • ACM MM
    • SIGGRAPH
    • AAAI
    • arXiv
    • Others
• Q: What does Others refers to?
  • Some of the following studies (e.g., Stable Casacade) does not publish their technical report on arXiv. Instead, they tend to write a blog in their official websites. The Others category refers to such kind of studies.

<🎯Back to Top>

References

The reference.bib file summarizes bibtex references of up-to-date image inpainting papers, widely used datasets, and toolkits. Based on the original references, I have made the following modifications to make their results look nice in the LaTeX manuscripts:

• Refereces are normally constructed in the form of author-etal-year-nickname. Particularly, references of datasets and toolkits are directly constructed as nickname, e.g., imagenet.
• In each reference, all names of conferences/journals are converted into abbreviations, e.g., Computer Vision and Pattern Recognition -> CVPR.
• The url, doi, publisher, organization, editor, series in all references are removed.
• The pages of all references are added if they are missing.
• All paper names are in title case. Besides, I have added an additional {} to make sure that the title case would also work well in some particular templates.

If you have other demands of reference formats, you may refer to the original references of papers by searching their names in DBLP or Google Scholar.

Note

Note that references in the homepage and the topic section can be repeated in reference.bib. Personally, I recommend using "Ctrl+F" / "Command+F" to search your desired BibTeX reference.

<🎯Back to Top>

Star History

<🎯Back to Top>